Original: A TTENTION , L EARN TO S OLVE R OUTING P ROBLEMS !
Revised: A TTENTION , L EARN TO S OLVE R OUTING P ROBLEMS !

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development.
Revised: The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development.

Original: However, in order to push this idea towards practical implementation, we need better models and better ways of training.
Revised: However, in order to push this idea towards practical implementation, we need better models and better ways of training.

Original: We contribute in both directions: we propose a model based entirely on attention layers with beneﬁts over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we ﬁnd is much more efﬁcient than using a value function.
Revised: We contribute in both directions: we propose a model based entirely on attention layers with beneﬁts over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we ﬁnd is much more efﬁcient than using a value function.

Original: We signiﬁcantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes.
Revised: We signiﬁcantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes.

Original: With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.
Revised: With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.

Original: The ability to construct a tour in order is beneﬁcial in the online (stochastic) setting.
Revised: The ability to construct a tour in order is beneﬁcial in the online (stochastic) setting.

Original: We make code publicly available.
Revised: We make code publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imagine yourself travelling to a scientiﬁc conference.
Revised: Imagine yourself travelling to a scientiﬁc conference.

Original: The ﬁeld is popular, and surely you do not want to miss out on anything.
Revised: The ﬁeld is popular, and surely you do not want to miss out on anything.

Original: You have selected several posters you want to visit, and naturally you must return to the place where you are now: the coffee corner.
Revised: You have selected several posters you want to visit, and naturally you must return to the place where you are now: the coffee corner.

Original: In which order should you visit the posters, to minimize your time walking around?
Revised: In which order should you visit the posters, to minimize your time walking around?

Original: This is the Travelling Scientist Problem (TSP).
Revised: This is the Travelling Scientist Problem (TSP).

Original: You realize that your problem is equivalent to the Travelling Salesman Problem (conveniently also
Revised: You realize that your problem is equivalent to the Travelling Salesman Problem (conveniently also

Original: TSP).
Revised: TSP).

Original: This seems discouraging as you know the problem is (NP-)hard (Garey & Johnson, 1979).
Revised: This seems discouraging as you know the problem is (NP-)hard (Garey & Johnson, 1979).

Original: Fortunately, complexity theory analyzes the worst case, and your Bayesian view considers this unlikely.
Revised: Fortunately, complexity theory analyzes the worst case, and your Bayesian view considers this unlikely.

Original: In particular, you have a strong prior: the posters will probably be laid out regularly.
Revised: In particular, you have a strong prior: the posters will probably be laid out regularly.

Original: You want a special algorithm that solves not any, but this type of problem instance.
Revised: You want a special algorithm that solves not any, but this type of problem instance.

Original: You have some months left to prepare.
Revised: You have some months left to prepare.

Original: As a machine learner, you wonder whether your algorithm can be learned?
Revised: As a machine learner, you wonder whether your algorithm can be learned?

Original: Motivation Machine learning algorithms have replaced humans as the engineers of algorithms to solve various tasks.
Revised: Motivation Machine learning algorithms have replaced humans as the engineers of algorithms to solve various tasks.

Original: A decade ago, computer vision algorithms used hand-crafted features but today they are learned end-to-end by Deep Neural Networks (DNNs).
Revised: A decade ago, computer vision algorithms used hand-crafted features but today they are learned end-to-end by Deep Neural Networks (DNNs).

Original: DNNs have outperformed classic approaches in speech recognition, machine translation, image captioning and other problems, by learning from data (LeCun et al., 2015).
Revised: DNNs have outperformed classic approaches in speech recognition, machine translation, image captioning and other problems, by learning from data (LeCun et al., 2015).

Original: While DNNs are mainly used to make predictions , Reinforcement Learning (RL) has enabled algorithms to learn to make decisions , either by interacting with an environment, e.g. to learn to play Atari games (Mnih et al., 2015), or by inducing knowledge through look-ahead search: this was used to master the game of Go (Silver et al., 2017).
Revised: While DNNs are mainly used to make predictions , Reinforcement Learning (RL) has enabled algorithms to learn to make decisions , either by interacting with an environment, e.g. to learn to play Atari games (Mnih et al., 2015), or by inducing knowledge through look-ahead search: this was used to master the game of Go (Silver et al., 2017).

Original: The world is not a game, and we desire to train models that make decisions to solve real problems.
Revised: The world is not a game, and we desire to train models that make decisions to solve real problems.

Original: These models must learn to select good solutions for a problem from a combinatorially large set of potential solutions.
Revised: These models must learn to select good solutions for a problem from a combinatorially large set of potential solutions.

Original: Classically, approaches to this problem of combinatorial optimization can be divided into exact methods , that guarantee ﬁnding optimal solutions, and heuristics , that trade off optimality for computational cost, although exact methods can use heuristics internally and vice versa.
Revised: Classically, approaches to this problem of combinatorial optimization can be divided into exact methods , that guarantee ﬁnding optimal solutions, and heuristics , that trade off optimality for computational cost, although exact methods can use heuristics internally and vice versa.

Original: Heuristics are typically expressed in the form of rules, which can be interpreted as policies to make decisions.
Revised: Heuristics are typically expressed in the form of rules, which can be interpreted as policies to make decisions.

Original: We believe that these policies can be parameterized using DNNs, and be trained to obtain new and stronger algorithms for many different combinatorial optimization problems, similar to the way DNNs have boosted performance in the applications mentioned before.
Revised: We believe that these policies can be parameterized using DNNs, and be trained to obtain new and stronger algorithms for many different combinatorial optimization problems, similar to the way DNNs have boosted performance in the applications mentioned before.

Original: In this paper, we focus on routing problems: an important class of practical combinatorial optimization problems.
Revised: In this paper, we focus on routing problems: an important class of practical combinatorial optimization problems.

Original: The promising idea to learn heuristics has been tested on TSP (Bello et al., 2016).
Revised: The promising idea to learn heuristics has been tested on TSP (Bello et al., 2016).

Original: In order to push this idea, we need better models and better ways of training.
Revised: In order to push this idea, we need better models and better ways of training.

Original: Therefore, we propose to use a powerful model based on attention and we propose to train this model using REINFORCE with a simple but effective greedy rollout baseline.
Revised: Therefore, we propose to use a powerful model based on attention and we propose to train this model using REINFORCE with a simple but effective greedy rollout baseline.

Original: The goal of our method is not to outperform a nonlearned, specialized TSP algorithm such as Concorde (Applegate et al., 2006).
Revised: The goal of our method is not to outperform a nonlearned, specialized TSP algorithm such as Concorde (Applegate et al., 2006).

Original: Rather, we show the ﬂexibility of our approach on multiple (routing) problems of reasonable size, with a single set of hyperparameters .
Revised: Rather, we show the ﬂexibility of our approach on multiple (routing) problems of reasonable size, with a single set of hyperparameters .

Original: This is important progress towards the situation where we can learn strong heuristics to solve a wide range of different practical problems for which no good heuristics exist.
Revised: This is important progress towards the situation where we can learn strong heuristics to solve a wide range of different practical problems for which no good heuristics exist.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The application of Neural Networks (NNs) for optimizing decisions in combinatorial optimization problems dates back to Hopﬁeld & Tank (1985), who applied a Hopﬁeld-network for solving small
Revised: The application of Neural Networks (NNs) for optimizing decisions in combinatorial optimization problems dates back to Hopﬁeld & Tank (1985), who applied a Hopﬁeld-network for solving small

Original: TSP instances.
Revised: TSP instances.

Original: NNs have been applied to many related problems (Smith, 1999), although in most cases in an online manner, starting ‘from scratch’ and ‘learning’ a solution for every instance.
Revised: NNs have been applied to many related problems (Smith, 1999), although in most cases in an online manner, starting ‘from scratch’ and ‘learning’ a solution for every instance.

Original: More recently, (D)NNs have also been used ofﬂine to learn about an entire class of problem instances.
Revised: More recently, (D)NNs have also been used ofﬂine to learn about an entire class of problem instances.

Original: Vinyals et al.
Revised: Vinyals et al.

Original: (2015) introduce the Pointer Network (PN) as a model that uses attention to output a permutation of the input, and train this model ofﬂine to solve the (Euclidean) TSP, supervised by example solutions.
Revised: (2015) introduce the Pointer Network (PN) as a model that uses attention to output a permutation of the input, and train this model ofﬂine to solve the (Euclidean) TSP, supervised by example solutions.

Original: Upon test time, their beam search procedure ﬁlters invalid tours.
Revised: Upon test time, their beam search procedure ﬁlters invalid tours.

Original: Bello et al.
Revised: Bello et al.

Original: (2016) introduce an Actor-Critic algorithm to train the PN without supervised solutions.
Revised: (2016) introduce an Actor-Critic algorithm to train the PN without supervised solutions.

Original: They consider each instance as a training sample and use the cost (tour length) of a sampled solution for an unbiased Monte-Carlo estimate of the policy gradient.
Revised: They consider each instance as a training sample and use the cost (tour length) of a sampled solution for an unbiased Monte-Carlo estimate of the policy gradient.

Original: They introduce extra model depth in the decoder by an additional glimpse (Vinyals et al., 2016) at the embeddings, masking nodes already visited.
Revised: They introduce extra model depth in the decoder by an additional glimpse (Vinyals et al., 2016) at the embeddings, masking nodes already visited.

Original: For small instances ( n = 20 ), they get close to the results by Vinyals et al.
Revised: For small instances ( n = 20 ), they get close to the results by Vinyals et al.

Original: (2015), they improve for n = 50 and additionally include results for n = 100 .
Revised: (2015), they improve for n = 50 and additionally include results for n = 100 .

Original: Nazari et al.
Revised: Nazari et al.

Original: (2018) replace the LSTM encoder of the PN by element-wise projections, such that the updated embeddings after state-changes can be effectively computed.
Revised: (2018) replace the LSTM encoder of the PN by element-wise projections, such that the updated embeddings after state-changes can be effectively computed.

Original: They apply this model on the Vehicle Routing Problem (VRP) with split deliveries and a stochastic variant.
Revised: They apply this model on the Vehicle Routing Problem (VRP) with split deliveries and a stochastic variant.

Original: Dai et al.
Revised: Dai et al.

Original: (2017) do not use a separate encoder and decoder, but a single model based on graph embeddings.
Revised: (2017) do not use a separate encoder and decoder, but a single model based on graph embeddings.

Original: They train the model to output the order in which nodes are inserted into a partial tour, using a helper function to insert at the best possible location.
Revised: They train the model to output the order in which nodes are inserted into a partial tour, using a helper function to insert at the best possible location.

Original: Their 1-step DQN (Mnih et al., 2015) training method trains the algorithm per step and incremental rewards provided to the agent at every step effectively encourage greedy behavior.
Revised: Their 1-step DQN (Mnih et al., 2015) training method trains the algorithm per step and incremental rewards provided to the agent at every step effectively encourage greedy behavior.

Original: As mentioned in their appendix, they use the negative of the reward, which combined with discounting encourages the agent to insert the farthest nodes ﬁrst, which is known to be an effective heuristic (Rosenkrantz et al., 2009).
Revised: As mentioned in their appendix, they use the negative of the reward, which combined with discounting encourages the agent to insert the farthest nodes ﬁrst, which is known to be an effective heuristic (Rosenkrantz et al., 2009).

Original: Nowak et al.
Revised: Nowak et al.

Original: (2017) train a Graph Neural Network in a supervised manner to directly output a tour as an adjacency matrix, which is converted into a feasible solution by a beam search.
Revised: (2017) train a Graph Neural Network in a supervised manner to directly output a tour as an adjacency matrix, which is converted into a feasible solution by a beam search.

Original: The model is non-autoregressive, so cannot condition its output on the partial tour and the authors report an optimality gap of 2 .
Revised: The model is non-autoregressive, so cannot condition its output on the partial tour and the authors report an optimality gap of 2 .

Original: 7% for n = 20 , worse than autoregressive approaches mentioned in this section.
Revised: 7% for n = 20 , worse than autoregressive approaches mentioned in this section.

Original: Kaempfer & Wolf (2018) train a model based on the Transformer architecture (Vaswani et al., 2017) that outputs a fractional solution to the multiple TSP (mTSP).
Revised: Kaempfer & Wolf (2018) train a model based on the Transformer architecture (Vaswani et al., 2017) that outputs a fractional solution to the multiple TSP (mTSP).

Original: The result can be seen as a solution to the linear relaxation of the problem and they use a beam search to obtain a feasible integer solution.
Revised: The result can be seen as a solution to the linear relaxation of the problem and they use a beam search to obtain a feasible integer solution.

Original: Independently of our work, Deudon et al.
Revised: Independently of our work, Deudon et al.

Original: (2018) presented a model for TSP using attention in the OR community.
Revised: (2018) presented a model for TSP using attention in the OR community.

Original: They show performance can improve using 2OPT local search, but do not show beneﬁt of their model in direct comparison to the PN.
Revised: They show performance can improve using 2OPT local search, but do not show beneﬁt of their model in direct comparison to the PN.

Original: We use a different decoder and improved training algorithm, both contributing to signiﬁcantly improved results, without 2OPT and additionally show application to  the VRP, the Orienteering Problem (OP) and (Stochastic) Prize Collecting TSP.
Revised: We use a different decoder and improved training algorithm, both contributing to signiﬁcantly improved results, without 2OPT and additionally show application to different problems. For a full discussion of the differences, we refer to Appendix B.4.

Original:  A TTENTION MODEL
Revised:  A TTENTION MODEL

Original: We deﬁne the Attention Model in terms of the TSP.
Revised: We deﬁne the Attention Model in terms of the TSP.

Original: For other problems, the model is the same but the input, mask and decoder context need to be deﬁned accordingly, which is discussed in the Appendix.
Revised: For other problems, the model is the same but the input, mask and decoder context need to be deﬁned accordingly, which is discussed in the Appendix.

Original: We deﬁne a problem instance s as a graph with n nodes, where node i ∈ { 1 , . . . , n } is represented by features x i .
Revised: We deﬁne a problem instance s as a graph with n nodes, where node i ∈ { 1 , . . . , n } is represented by features x i .

Original: For TSP, x i is the coordinate of node i and the graph is fully connected (with selfconnections) but in general, the model can be considered a Graph Attention Network (Velickovic et al., 2018) and take graph structure into account by a masking procedure (see Appendix A).
Revised: For TSP, x i is the coordinate of node i and the graph is fully connected (with selfconnections) but in general, the model can be considered a Graph Attention Network (Velickovic et al., 2018) and take graph structure into account by a masking procedure (see Appendix A).

Original: We deﬁne a solution (tour) π
Revised: We deﬁne a solution (tour) π

Original: = ( π 1 , . . . , π n ) as a permutation of the nodes, so π t ∈ { 1 , . . .
Revised: = ( π 1 , . . . , π n ) as a permutation of the nodes, so π t ∈ { 1 , . . .

Original: n } and π t (cid:54) =
Revised: n } and π t (cid:54) =

Original: π t (cid:48) ∀ t (cid:54) =
Revised: π t (cid:48) ∀ t (cid:54) =

Original: t (cid:48) .
Revised: t (cid:48) .

Original: Our attention based encoder-decoder model deﬁnes a stochastic policy p ( π | s ) for selecting a solution π given a problem instance s .
Revised: Our attention based encoder-decoder model deﬁnes a stochastic policy p ( π | s ) for selecting a solution π given a problem instance s .

Original: It is factorized and parameterized by θ as
Revised: It is factorized and parameterized by θ as

Original: [Equation]
Revised: [Equation]

Original: The encoder produces embeddings of all input nodes.
Revised: The encoder produces embeddings of all input nodes.

Original: The decoder produces the sequence π of input nodes, one node at a time.
Revised: The decoder produces the sequence π of input nodes, one node at a time.

Original: It takes as input the encoder embeddings and a problem speciﬁc mask and context.
Revised: It takes as input the encoder embeddings and a problem speciﬁc mask and context.

Original: For TSP, when a partial tour has been constructed, it cannot be changed and the remaining problem is to ﬁnd a path from the last node, through all unvisited nodes, to the ﬁrst node.
Revised: For TSP, when a partial tour has been constructed, it cannot be changed and the remaining problem is to ﬁnd a path from the last node, through all unvisited nodes, to the ﬁrst node.

Original: The order and coordinates of other nodes already visited are irrelevant.
Revised: The order and coordinates of other nodes already visited are irrelevant.

Original: To know the ﬁrst and last node, the decoder context consists (next to the graph embedding) of embeddings of the ﬁrst and last node.
Revised: To know the ﬁrst and last node, the decoder context consists (next to the graph embedding) of embeddings of the ﬁrst and last node.

Original: Similar to Bello et al.
Revised: Similar to Bello et al.

Original: , the decoder observes a mask to know which nodes have been visited.
Revised: , the decoder observes a mask to know which nodes have been visited.

Original: 3. E NCODER
Revised: 3. E NCODER

Original: The encoder that we use (Figure 1) is similar to the encoder used in the Transformer architecture by Vaswani et al.
Revised: The encoder that we use (Figure 1) is similar to the encoder used in the Transformer architecture by Vaswani et al.

Original: but we do not use positional encoding such that the resulting node embeddings are invariant to the input order.
Revised: but we do not use positional encoding such that the resulting node embeddings are invariant to the input order.

Original: From the d x -dimensional input features x i (for TSP d x = 2), the encoder computes initial d h -dimensional node embeddings h (0)
Revised: From the d x -dimensional input features x i (for TSP d x = 2), the encoder computes initial d h -dimensional node embeddings h (0)

Original: i (we use d h = 128 ) through a learned linear projection with parameters W x and b x : h (0) i
Revised: i (we use d h = 128 ) through a learned linear projection with parameters W x and b x : h (0) i

Original: =
Revised: =

Original: W x x i + b x .
Revised: W x x i + b x .

Original: The embeddings are updated using N attention layers, each consisting of two sublayers.
Revised: The embeddings are updated using N attention layers, each consisting of two sublayers.

Original: We denote with h ( (cid:96) )
Revised: We denote with h ( (cid:96) )

Original: i the node embeddings produced by layer (cid:96) ∈ { 1 , .., N } .
Revised: i the node embeddings produced by layer (cid:96) ∈ { 1 , .., N } .

Original: The encoder computes an aggregated embedding ¯ h ( N ) of the input graph as the mean of the ﬁnal node embeddings h ( N )
Revised: The encoder computes an aggregated embedding ¯ h ( N ) of the input graph as the mean of the ﬁnal node embeddings h ( N )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Both the node embeddings h ( N ) i and the graph embedding ¯ h ( N ) are used as input to the decoder.
Revised: Both the node embeddings h ( N ) i and the graph embedding ¯ h ( N ) are used as input to the decoder.

Original: Attention layer Following the Transformer architecture (Vaswani et al., 2017), each attention layer consist of two sublayers: a multi-head attention (MHA) layer that executes message passing between the nodes and a node-wise fully connected feed-forward (FF) layer.
Revised: Attention layer Following the Transformer architecture (Vaswani et al., 2017), each attention layer consist of two sublayers: a multi-head attention (MHA) layer that executes message passing between the nodes and a node-wise fully connected feed-forward (FF) layer.

Original: Each sublayer adds a skip-connection (He et al., 2016) and batch normalization (BN) (Ioffe & Szegedy, 2015) (which we found to work better than layer normalization (Ba et al., 2016)):
Revised: Each sublayer adds a skip-connection (He et al., 2016) and batch normalization (BN) (Ioffe & Szegedy, 2015) (which we found to work better than layer normalization (Ba et al., 2016)):

Original: [Equation]
Revised: [Equation]

Original: The layer index (cid:96) indicates that the layers do not share parameters.
Revised: The layer index (cid:96) indicates that the layers do not share parameters.

Original: The MHA sublayer uses M = 8 heads with dimensionality d h M = 16 , and the FF sublayer has one hidden (sub)sublayer with dimension 512 and ReLu activation.
Revised: The MHA sublayer uses M = 8 heads with dimensionality d h M = 16 , and the FF sublayer has one hidden (sub)sublayer with dimension 512 and ReLu activation.

Original: See Appendix A for details.
Revised: See Appendix A for details.

Original: 3. D ECODER
Revised: 3. D ECODER

Original: Decoding happens sequentially, and at timestep t ∈ { 1 , . . .
Revised: Decoding happens sequentially, and at timestep t ∈ { 1 , . . .

Original: n } , the decoder outputs the node π t based on the embeddings from the encoder and the outputs π t (cid:48) generated at time t (cid:48) < t .
Revised: n } , the decoder outputs the node π t based on the embeddings from the encoder and the outputs π t (cid:48) generated at time t (cid:48) < t .

Original: During decoding, we augment the graph with a special context node ( c ) to represent the decoding context.
Revised: During decoding, we augment the graph with a special context node ( c ) to represent the decoding context.

Original: The decoder computes an attention (sub)layer on top of the encoder, but with messages only to the context node for efﬁciency.
Revised: The decoder computes an attention (sub)layer on top of the encoder, but with messages only to the context node for efﬁciency.

Original: The ﬁnal probabilities are computed using a single-head attention mechanism.
Revised: The ﬁnal probabilities are computed using a single-head attention mechanism.

Original: See
Revised: See

Original: Figure 2 for an illustration of the decoding process.
Revised: Figure 2 for an illustration of the decoding process.

Original: Context embedding The context of the decoder at time t comes from the encoder and the output up to time t .
Revised: Context embedding The context of the decoder at time t comes from the encoder and the output up to time t .

Original: As mentioned, for the TSP it consists of the embedding of the graph, the previous (last) node π t − 1 and the ﬁrst node π 1 .
Revised: As mentioned, for the TSP it consists of the embedding of the graph, the previous (last) node π t − 1 and the ﬁrst node π 1 .

Original: For t = 1 we use learned d h -dimensional parameters v l and v f as input placeholders:
Revised: For t = 1 we use learned d h -dimensional parameters v l and v f as input placeholders:

Original: [Equation]
Revised: [Equation]

Original: Here [ · , · , · ] is the horizontal concatenation operator and we write the (3 · d h ) -dimensional result vector as h ( N ) ( c ) to indicate we interpret it as the embedding of the special context node ( c ) and use the superscript ( N ) to align with the node embeddings h ( N ) i .
Revised: Here [ · , · , · ] is the horizontal concatenation operator and we write the (3 · d h ) -dimensional result vector as h ( N ) ( c ) to indicate we interpret it as the embedding of the special context node ( c ) and use the superscript ( N ) to align with the node embeddings h ( N ) i .

Original: We could project the embedding back to d h dimensions, but we absorb this transformation in the parameter W Q in equation 5.
Revised: We could project the embedding back to d h dimensions, but we absorb this transformation in the parameter W Q in equation 5.

Original: Now we compute a new context node embedding h ( N +1)
Revised: Now we compute a new context node embedding h ( N +1)

Original: ( c ) using the ( M -head) attention mechanism described in Appendix A.
Revised: ( c ) using the ( M -head) attention mechanism described in Appendix A.

Original: The keys and values come from the node embeddings h ( N ) i , but we only compute a single query q ( c ) (per head) from the context node (we omit the ( N ) for readability):
Revised: The keys and values come from the node embeddings h ( N ) i , but we only compute a single query q ( c ) (per head) from the context node (we omit the ( N ) for readability):

Original: [Equation]
Revised: [Equation]

Original: We compute the compatibility of the query with all nodes, and mask (set u ( c ) j = −∞ ) nodes which cannot be visited at time t .
Revised: We compute the compatibility of the query with all nodes, and mask (set u ( c ) j = −∞ ) nodes which cannot be visited at time t .

Original: For TSP, this simply means we mask the nodes already visited:
Revised: For TSP, this simply means we mask the nodes already visited:

Original: [Equation]
Revised: [Equation]

Original: Here d k = d h M is the query/key dimensionality (see Appendix A).
Revised: Here d k = d h M is the query/key dimensionality (see Appendix A).

Original: Again, we compute u ( c ) j and v i for M = 8 heads and compute the ﬁnal multi-head attention value for the context node using equations 12–14 from Appendix A, but with ( c ) instead of i .
Revised: Again, we compute u ( c ) j and v i for M = 8 heads and compute the ﬁnal multi-head attention value for the context node using equations 12–14 from Appendix A, but with ( c ) instead of i .

Original: This mechanism is similar to our encoder, but does not use skip-connections, batch normalization or the feed-forward sublayer for maximal efﬁciency.
Revised: This mechanism is similar to our encoder, but does not use skip-connections, batch normalization or the feed-forward sublayer for maximal efﬁciency.

Original: The result h ( N +1)
Revised: The result h ( N +1)

Original: ( c ) is similar to the glimpse described by Bello et al.
Revised: ( c ) is similar to the glimpse described by Bello et al.

Original: Calculation of log-probabilities To compute output probabilities p θ ( π t | s, π 1: t − 1 ) in equation 1, we add one ﬁnal decoder layer with a single attention head ( M = 1 so d k = d h ).
Revised: Calculation of log-probabilities To compute output probabilities p θ ( π t | s, π 1: t − 1 ) in equation 1, we add one ﬁnal decoder layer with a single attention head ( M = 1 so d k = d h ).

Original: For this layer, we only compute the compatibilities u ( c )
Revised: For this layer, we only compute the compatibilities u ( c )

Original: j using equation 6, but following Bello et al.
Revised: j using equation 6, but following Bello et al.

Original: we clip the result (before masking!)
Revised: we clip the result (before masking!)

Original: within [ − C, C ] (C = 10) using tanh :
Revised: within [ − C, C ] (C = 10) using tanh :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We interpret these compatibilities as unnormalized log-probabilities (logits) and compute the ﬁnal output probability vector p using a softmax (similar to equation 12 in Appendix A):
Revised: We interpret these compatibilities as unnormalized log-probabilities (logits) and compute the ﬁnal output probability vector p using a softmax (similar to equation 12 in Appendix A):

Original: [Equation]
Revised: [Equation]

Original:  REINFORCE WITH GREEDY ROLLOUT BASELINE
Revised:  REINFORCE WITH GREEDY ROLLOUT BASELINE

Original: Section 3 deﬁned our model that given an instance s deﬁnes a probability distribution p θ ( π | s ) , from which we can sample to obtain a solution (tour) π | s .
Revised: Section 3 deﬁned our model that given an instance s deﬁnes a probability distribution p θ ( π | s ) , from which we can sample to obtain a solution (tour) π | s .

Original: In order to train our model, we deﬁne the loss
Revised: In order to train our model, we deﬁne the loss

Original: L ( θ | s ) =
Revised: L ( θ | s ) =

Original: E p θ ( π | s )
Revised: E p θ ( π | s )

Original: [ L ( π )] : the expectation of the cost L ( π ) (tour length for TSP).
Revised: [ L ( π )] : the expectation of the cost L ( π ) (tour length for TSP).

Original: We optimize L by gradient descent, using the REINFORCE (Williams, 1992) gradient estimator with baseline b ( s ) :
Revised: We optimize L by gradient descent, using the REINFORCE (Williams, 1992) gradient estimator with baseline b ( s ) :

Original: [Equation]
Revised: [Equation]

Original: A good baseline b ( s ) reduces gradient variance and therefore increases speed of learning.
Revised: A good baseline b ( s ) reduces gradient variance and therefore increases speed of learning.

Original: A simple example is an exponential moving average b ( s ) =
Revised: A simple example is an exponential moving average b ( s ) =

Original: M with decay β .
Revised: M with decay β .

Original: Here M = L ( π ) in the ﬁrst iteration and gets updated as M ← βM +(1 − β ) L ( π ) in subsequent iterations.
Revised: Here M = L ( π ) in the ﬁrst iteration and gets updated as M ← βM +(1 − β ) L ( π ) in subsequent iterations.

Original: A popular alternative is the use of a learned value function (critic) ˆ v ( s, w ) , where the parameters w are learned from the observations ( s, L ( π )) .
Revised: A popular alternative is the use of a learned value function (critic) ˆ v ( s, w ) , where the parameters w are learned from the observations ( s, L ( π )) .

Original: However, getting such actor-critic algorithms to work is non-trivial.
Revised: However, getting such actor-critic algorithms to work is non-trivial.

Original: We propose to use a rollout baseline in a way that is similar to self-critical training by Rennie et al.
Revised: We propose to use a rollout baseline in a way that is similar to self-critical training by Rennie et al.

Original: (2017), but with periodic updates of the baseline policy.
Revised: (2017), but with periodic updates of the baseline policy.

Original: It is deﬁned as follows:
Revised: It is deﬁned as follows:

Original: b ( s ) is the cost of a solution from a deterministic greedy rollout of the policy deﬁned by the best model so far.
Revised: b ( s ) is the cost of a solution from a deterministic greedy rollout of the policy deﬁned by the best model so far.

Original: Motivation The goal of a baseline is to estimate the difﬁculty of the instance s , such that it can relate to the cost L ( π ) to estimate the advantage of the solution π selected by the model.
Revised: Motivation The goal of a baseline is to estimate the difﬁculty of the instance s , such that it can relate to the cost L ( π ) to estimate the advantage of the solution π selected by the model.

Original: We make the following key observation: The difﬁculty of an instance can (on average) be estimated by the performance of an algorithm applied to it.
Revised: We make the following key observation: The difﬁculty of an instance can (on average) be estimated by the performance of an algorithm applied to it.

Original: This follows from the assumption that (on average) an algorithm will have a higher cost on instances that are more difﬁcult.
Revised: This follows from the assumption that (on average) an algorithm will have a higher cost on instances that are more difﬁcult.

Original: Therefore we form a baseline by applying (rolling out) the algorithm deﬁned by our model during training.
Revised: Therefore we form a baseline by applying (rolling out) the algorithm deﬁned by our model during training.

Original: To eliminate variance we force the result to be deterministic by selecting greedily the action with maximum probability.
Revised: To eliminate variance we force the result to be deterministic by selecting greedily the action with maximum probability.

Original: Determining the baseline policy As the model changes during training, we stabilize the baseline by freezing the greedy rollout policy p θ BL for a ﬁxed number of steps (every epoch), similar to freezing of the target Q-network in DQN (Mnih et al., 2015).
Revised: Determining the baseline policy As the model changes during training, we stabilize the baseline by freezing the greedy rollout policy p θ BL for a ﬁxed number of steps (every epoch), similar to freezing of the target Q-network in DQN (Mnih et al., 2015).

Original: A stronger algorithm deﬁnes a stronger baseline, so we compare (with greedy decoding) the current training policy with the baseline policy at the end of every epoch, and replace the parameters θ BL of the baseline policy only if the improvement is signiﬁcant according to a paired t-test ( α = 5% ), on 10000 separate (evaluation) instances.
Revised: A stronger algorithm deﬁnes a stronger baseline, so we compare (with greedy decoding) the current training policy with the baseline policy at the end of every epoch, and replace the parameters θ BL of the baseline policy only if the improvement is signiﬁcant according to a paired t-test ( α = 5% ), on 10000 separate (evaluation) instances.

Original: If the baseline policy is updated, we sample new evaluation instances to prevent overﬁtting.
Revised: If the baseline policy is updated, we sample new evaluation instances to prevent overﬁtting.

Original: Analysis Using the greedy rolloutpolicy as baseline b ( s ) , the function L ( π ) − b ( s ) is negative if the sampled solution π is better than the greedy rollout solution, causing actions to be reinforced, and vice versa.
Revised: Analysis With the greedy rollout as baseline b ( s ) , the function L ( π ) − b ( s ) is negative if the sampled solution π is better than the greedy rollout, causing actions to be reinforced, and vice versa.

Original: Combined with the evaluation at the end of each epoch, this challenges the model  to improve over itself, and we seestrong similarities with  the self-play improvement scheme that was successfully applied tomaster the game of Go (Silver et al., 2017).
Revised: This way the model is trained to improve over its (greedy) self. We see similarities with selfplay improvement (Silver et al., 2017): sampling replaces tree search for exploration and the model is rewarded if it yields improvement (‘wins’) compared to the best model. Similar to AlphaGo, the evaluation at the end of each epoch ensures that we are always challenged by the best model.

Original: Algorithm We use Adam (Kingma & Ba, 2015) as optimizer resulting in Algorithm 1.
Revised: Algorithm We use Adam (Kingma & Ba, 2015) as optimizer resulting in Algorithm 1.

Original: [Table]
Revised: [Table]

Original: Efﬁciency Each rollout constitutes an additional forward pass, increasing computation by 50% .
Revised: Efﬁciency Each rollout constitutes an additional forward pass, increasing computation by 50% .

Original: However, as the baseline policy is ﬁxed for an epoch, we can sample the data and compute baselines per epoch using larger batch sizes, allowed by the reduced memory requirement as the computations can run in pure inference mode.
Revised: However, as the baseline policy is ﬁxed for an epoch, we can sample the data and compute baselines per epoch using larger batch sizes, allowed by the reduced memory requirement as the computations can run in pure inference mode.

Original: Empirically we ﬁnd that it adds only 25% (see Appendix B.4), taking up 20% of total time.
Revised: Empirically we ﬁnd that it adds only 25% (see Appendix B.5), taking up 20% of total time.

Original: If desired, the baseline rollout can be computed in parallel such that there is no increase in time per iteration, as an easy way to beneﬁt from an additional GPU.
Revised: If desired, the baseline rollout can be computed in parallel such that there is no increase in time per iteration, as an easy way to beneﬁt from an additional GPU.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We focus on routing problems: we consider the TSP, two variants of the VRP, the Orienteering
Revised: We focus on routing problems: we consider the TSP, two variants of the VRP, the Orienteering

Original: Problem and the (Stochastic) Prize Collecting TSP.
Revised: Problem and the (Stochastic) Prize Collecting TSP.

Original: These provide a range of different challenges, constraints and objectives and are traditionally solved by different algorithms .
Revised: These provide a range of different challenges, constraints and objectives and are traditionally solved by different algorithms .

Original: For the Attention
Revised: For the Attention

Original: Model (AM), we adjust the input, mask, decoder context and objective function for each problem (see Appendix for details and data generation) and train on problem instances of n = 20 , 50 andnodes.
Revised: Model (AM), we adjust the input, mask, decoder context and objective function for each problem (see Appendix for details and data generation) and train on problem instances of n = 20 , 50 andnodes.

Original: For all problems, we use the same hyperparameters : those we found to work well on TSP.
Revised: For all problems, we use the same hyperparameters : those we found to work well on TSP.

Original: Hyperparameters We initialize parameters Uniform ( − 1 / √ d, 1 / √ d ) , with d the input dimension.
Revised: Hyperparameters We initialize parameters Uniform ( − 1 / √ d, 1 / √ d ) , with d the input dimension.

Original: Every epoch we process 2500 batches of 512 instances (except for VRP with n = 100 , where we use 2500 × 256 for memory constraints).
Revised: Every epoch we process 2500 batches of 512 instances (except for VRP with n = 100 , where we use 2500 × 256 for memory constraints).

Original: For TSP, an epoch takes 5:30 minutes for n = 20 , 16: for n = 50 (single GPU 1080Ti) and 27:30 for n = 100 (on 2 1080Ti’s).
Revised: For TSP, an epoch takes 5:30 minutes for n = 20 , 16: for n = 50 (single GPU 1080Ti) and 27:30 for n = 100 (on 2 1080Ti’s).

Original: We train for 100 epochs using training data generated on the ﬂy.
Revised: We train for 100 epochs using training data generated on the ﬂy.

Original: We found training to be stable and results to be robust against different seeds, where only in one case (PCTSP with n = 20 ) we had to restart training with a different seed because the run diverged.
Revised: We found training to be stable and results to be robust against different seeds, where only in one case (PCTSP with n = 20 ) we had to restart training with a different seed because the run diverged.

Original: We use N = 3 layers in the encoder, which we found is a good trade-off between quality of the results and computational complexity.
Revised: We use N = 3 layers in the encoder, which we found is a good trade-off between quality of the results and computational complexity.

Original: We use a constant learning rate η = 10 − 4 .
Revised: We use a constant learning rate η = 10 − 4 .

Original: Training with a higher learning rate η = 10 − 3 is possible and speeds up initial learning, but requires decay ( 0 .
Revised: Training with a higher learning rate η = 10 − 3 is possible and speeds up initial learning, but requires decay ( 0 .

Original: 96 per epoch) to converge and may be a bit more unstable.
Revised: 96 per epoch) to converge and may be a bit more unstable.

Original: See Appendix B.4.
Revised: See Appendix B.5.

Original: With the rollout baseline, we use an exponential baseline ( β = 0 . 8 ) during the ﬁrst epoch, to stabilize initial learning, although in many cases learning also succeeds without this ‘warmup’.
Revised: With the rollout baseline, we use an exponential baseline ( β = 0 . 8 ) during the ﬁrst epoch, to stabilize initial learning, although in many cases learning also succeeds without this ‘warmup’.

Original: Our code in PyTorch (Paszke et al., 2017) will be publicly available.
Revised: Our code in PyTorch (Paszke et al., 2017) will be publicly available.

Original: Decoding strategy and baselines For each problem
Revised: Decoding strategy and baselines For each problem

Original: , we report performance on 10000 test instances.
Revised: , we report performance on 10000 test instances.

Original: At test time we use greedy decoding, where we select the best action (according to the model) at each step, or sampling , where we sample 1280 solutions (in < 1 s on a single GPU) and report the best.
Revised: At test time we use greedy decoding, where we select the best action (according to the model) at each step, or sampling , where we sample 1280 solutions (in < 1 s on a single GPU) and report the best.

Original: More sampling improves solution quality at increased computation.
Revised: More sampling improves solution quality at increased computation.

Original: In Table 1 we compare greedy decoding against baselines that also construct a single solution, and compare sampling against baselines that also consider multiple solutions, either via sampling or (local) search.
Revised: In Table 1 we compare greedy decoding against baselines that also construct a single solution, and compare sampling against baselines that also consider multiple solutions, either via sampling or (local) search.

Original: For each problem, we also report the ‘best possible solution’: either optimal via Gurobi Optimization (2016) (intractable for n > 20 except for TSP) or a problem speciﬁc state-of-the-art algorithm.
Revised: For each problem, we also report the ‘best possible solution’: either optimal via Gurobi Optimization (2016) (intractable for n > 20 except for TSP) or a problem speciﬁc state-of-the-art algorithm.

Original: Run times Run times are important but hard to compare: they can vary by two orders of magnitude as a result of implementation (Python vs C++) and hardware (GPU vs CPU).
Revised: Run times Run times are important but hard to compare: they can vary by two orders of magnitude as a result of implementation (Python vs C++) and hardware (GPU vs CPU).

Original: We take a practical view and report the time it takes to solve the test set of 10000 instances, either on a single GPU (1080Ti) or 32 instances in parallel on a 32 virtual CPU system (2 × Xeon E5-2630).
Revised: We take a practical view and report the time it takes to solve the test set of 10000 instances, either on a single GPU (1080Ti) or 32 instances in parallel on a 32 virtual CPU system (2 × Xeon E5-2630).

Original: This is conservative: our model is parallelizable while most of the baselines are single thread CPU implementations which cannot parallelize when running individually.
Revised: This is conservative: our model is parallelizable while most of the baselines are single thread CPU implementations which cannot parallelize when running individually.

Original: Also we note that after training our run time can likely be reduced by model compression (Hinton et al., 2015).
Revised: Also we note that after training our run time can likely be reduced by model compression (Hinton et al., 2015).

Original: In Table 1 we do not report running times for the results which were reported by others as they are not directly comparable but we note that in general our model and implementation is fast: for instance Bello et al.
Revised: In Table 1 we do not report running times for the results which were reported by others as they are not directly comparable but we note that in general our model and implementation is fast: for instance Bello et al.

Original: (2016) report 10.3s for sampling 1280 TSP solutions (K80 GPU) which we do in less than one second (on a 1080Ti).
Revised: (2016) report 10.3s for sampling 1280 TSP solutions (K80 GPU) which we do in less than one second (on a 1080Ti).

Revised: For most algorithms it is possible to trade off runtime for performance.

Revised: As reporting full trade-off curves is impractical we tried to pick reasonable spots, reporting the fastest if results were similar or reporting results with different time limits (for example we use Gurobi with time limits as heuristic).

Original: 5. P ROBLEMS
Revised: 5. P ROBLEMS

Revised: Travelling Salesman Problem (TSP) For the TSP, we report optimal results by Gurobi, as well as by Concorde (Applegate et al., 2006) (faster than Gurobi as it is specialized for TSP) and LKH (Helsgaun, 2017), a state-of-the-art heuristic solver that empirically also ﬁnds optimal solutions in time comparable to Gurobi.

Original: Travelling Salesman Problem (TSP) For the TSP, we compare against Nearest, Random and Farthest Insertion, as well as Nearest Neighbor, which is the only non-learned baseline algorithm that also constructs a tour directly in order (i.e. is structurally similar to our model).
Revised: We compare against Nearest, Random and Farthest Insertion, as well as Nearest Neighbor, which is the only non-learned baseline algorithm that also constructs a tour directly in order (i.e. is structurally similar to our model).

Original: For details, see Appendix B.3.
Revised: For details, see Appendix B.3.

Original: Additionally we compare against the learned heuristics in Section 2, most importantly Bello et al.
Revised: Additionally we compare against the learned heuristics in Section 2, most importantly Bello et al.

Original: (2016), as well as OR Tools reported by Bello et al.
Revised: (2016), as well as OR Tools reported by Bello et al.

Original: and Christoﬁdes + 2OPT local search reported by Vinyals et al. (2015).
Revised: and Christoﬁdes + 2OPT local search reported by Vinyals et al. (2015).

Original: Results for Dai et al.
Revised: Results for Dai et al.

Original: (2017) are (optimistically) computed from the optimality gaps they report on 15-20, 40-50 and 50-100 node graphs, respectively. 
Revised: (2017) are (optimistically) computed from the optimality gaps they report on 15-20, 40-50 and 50-100 node graphs, respectively. Using a single greedy construc- [Table] C

Original: Using a single greedy construction we outperform traditional baselines and we are able to achieve signiﬁcantly closer to optimal results than previous learned heuristics (from around 1.5% to 0.3% above optimal for n = 20 ).
Revised: [Table] ) OR Tools [Table] 10s)

Revised: [Table]

Revised: 52m)

Revised: [Table]

Revised: [Table]

Revised: Tools

Revised: [Table]

Revised: 60s)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: tion we outperform traditional baselines and we are able to achieve signiﬁcantly closer to optimal results than previous learned heuristics (from around 1.5% to 0.3% above optimal for n = 20 ).

Original: Naturally, the difference with Bello et al.
Revised: Naturally, the difference with Bello et al.

Original: (2016) gets diluted when sampling many solutions (as with many samples even a random policy performs well), but we still obtain signiﬁcantly better results, without tuning the softmax temperature. 
Revised: (2016) gets diluted when sampling many solutions (as with many samples even a random policy performs well), but we still obtain signiﬁcantly better results, without tuning the softmax temperature. For completeness, we also report results from running the

Revised: Encode-Attend-Navigate (EAN) code 2 which is concurrent work by Deudon et al.

Revised: (2018) (for details see Appendix B.4).

Revised: Our model outperforms EAN, even if EAN is improved with 2OPT local search.

Original: Appendix B.4 presents the generalization to different n and visually compares results in terms of the optimality gap.[Table]
Revised: Appendix B.5 presents the generalization to different n and visually compares results in terms of the optimality gap.

Original: Vehicle Routing Problem (VRP)
Revised: Vehicle Routing Problem (VRP)

Original: In the standard Capacitated VRP (CVRP) (Toth & Vigo, 2014), each node has a demand and multiple routes should be constructed (starting and ending at the depot), such that the total demand of the nodes in each route does not exceed the vehicle capacity.
Revised: In the standard Capacitated VRP (CVRP) (Toth & Vigo, 2014), each node has a demand and multiple routes should be constructed (starting and ending at the depot), such that the total demand of the nodes in each route does not exceed the vehicle capacity.

Original: We also consider the Split Delivery VRP (SDVRP), which allows to split customer demands over multiple routes.
Revised: We also consider the Split Delivery VRP (SDVRP), which allows to split customer demands over multiple routes.

Original: We implement datasets described by Nazari et al.
Revised: We implement datasets described by Nazari et al.

Original: and compare against their Reinforcement Learning (RL) framework and the strongest baselines they report.
Revised: and compare against their Reinforcement Learning (RL) framework and the strongest baselines they report.

Original: Comparing greedy decoding, we obtain signiﬁcantly better results.
Revised: Comparing greedy decoding, we obtain signiﬁcantly better results.

Original: We cannot directly compare our sampling (1280 samples) to their beam search with size 10 (they do not report sampling or larger beam sizes), but note that our greedy method also outperforms their beam search in most (larger) cases, getting (in < 1 second/instance) much closer to LKH (Helsgaun, 2017), a state-of-the-art algorithm which found best known solutions to CVRP benchmarks.
Revised: We cannot directly compare our sampling (1280 samples) to their beam search with size 10 (they do not report sampling or larger beam sizes), but note that our greedy method also outperforms their beam search in most (larger) cases, getting (in < 1 second/instance) much closer to LKH3 (Helsgaun, 2017), a state-of-the-art algorithm which found best known solutions to CVRP benchmarks.

Original: See Appendix C.4 for greedy example solution plots.
Revised: See Appendix C.4 for greedy example solution plots.

Original: Orienteering Problem (OP)
Revised: Orienteering Problem (OP)

Original: The OP (Golden et al., 1987) is an important problem used to model many real world problems.
Revised: The OP (Golden et al., 1987) is an important problem used to model many real world problems.

Original: Each node has an associated prize , and the goal is to construct a single tour (starting and ending at the depot) that maximizes the sum of prizes of nodes visited while being shorter than a maximum (given) length.
Revised: Each node has an associated prize , and the goal is to construct a single tour (starting and ending at the depot) that maximizes the sum of prizes of nodes visited while being shorter than a maximum (given) length.

Original: We consider the prize distributions proposed in Fischetti et al.
Revised: We consider the prize distributions proposed in Fischetti et al.

Original: (1998): constant , uniform (in Appendix D.4), and increasing with the distance to the depot, which we report here as this is the hardest problem.
Revised: (1998): constant , uniform (in Appendix D.4), and increasing with the distance to the depot, which we report here as this is the hardest problem.

Original: As ‘best possible solution’ we report Gurobi (intractable for n > 20 ) and Compass , the recent state-of-the-art Genetic Algorithm (GA) by Kobeaga et al.
Revised: As ‘best possible solution’ we report Gurobi (intractable for n > 20 ) and Compass , the recent state-of-the-art Genetic Algorithm (GA) by Kobeaga et al.

Original: (2018), which is only 2% better than sampling 1280 solutions with our method (objective is maximization).
Revised: (2018), which is only 2% better than sampling 1280 solutions with our method (objective is maximization).

Original: We outperform a Python GA 2 (which seems not to scale), as well the construction phase of the heuristic by Tsiligirides (1984) (comparing greedy or 1280 samples) which is structurally similar to the one learned by our model.
Revised: We outperform a Python GA 3 (which seems not to scale), as well the construction phase of the heuristic by Tsiligirides (1984) (comparing greedy or 1280 samples) which is structurally similar to the one learned by our model.

Original: OR Tools fails to ﬁnd feasible solutions in a few percent of the cases for n > 20 .
Revised: OR Tools fails to ﬁnd feasible solutions in a few percent of the cases for n > 20 .

Original: Prize Collecting TSP (PCTSP)
Revised: Prize Collecting TSP (PCTSP)

Original: In the PCTSP (Balas, 1989), each node has not only an associated prize, but also an associated penalty.
Revised: In the PCTSP (Balas, 1989), each node has not only an associated prize, but also an associated penalty.

Original: The goal is to collect at least a minimum total prize, while minimizing the total tour length plus the sum of penalties of unvisited nodes.
Revised: The goal is to collect at least a minimum total prize, while minimizing the total tour length plus the sum of penalties of unvisited nodes.

Original: This problem is difﬁcult as an algorithm has to trade off the penalty for not visiting a node with the marginal cost/tour length of visiting (which depends on the other nodes visited), while also satisfying the minimum total prize constraint.
Revised: This problem is difﬁcult as an algorithm has to trade off the penalty for not visiting a node with the marginal cost/tour length of visiting (which depends on the other nodes visited), while also satisfying the minimum total prize constraint.

Original: We compare against OR Tools with 10 or 60 seconds of local search, as well as open source C++ 3 and Python 4 implementations of Iterated Local Search (ILS).
Revised: We compare against OR Tools with 10 or 60 seconds of local search, as well as open source C++ 4 and Python 5 implementations of Iterated Local Search (ILS).

Original: Although the Attention
Revised: Although the Attention

Original: Model does not ﬁnd better solutions than OR Tools with 60s of local search, it ﬁnds almost equally good results in signiﬁcantly less time.
Revised: Model does not ﬁnd better solutions than OR Tools with 60s of local search, it ﬁnds almost equally good results in signiﬁcantly less time.

Original: The results are also within 2% of the C++
Revised: The results are also within 2% of the C++

Original: ILS algorithm
Revised: ILS algorithm

Original: (but obtained much faster), which was the best open-source algorithm for PCTSP we could ﬁnd.
Revised: (but obtained much faster), which was the best open-source algorithm for PCTSP we could ﬁnd.

Original: Stochastic PCTSP (SPCTSP)
Revised: Stochastic PCTSP (SPCTSP)

Original: The Stochastic variant of the PCTSP (SPCTSP)
Revised: The Stochastic variant of the PCTSP (SPCTSP)

Original: we consider shows how our model can deal with uncertainty naturally.
Revised: we consider shows how our model can deal with uncertainty naturally.

Original: In the SPCTSP, the expected node prize is known upfront, but the real collected prize only becomes known upon visitation.
Revised: In the SPCTSP, the expected node prize is known upfront, but the real collected prize only becomes known upon visitation.

Original: With penalties, this problem is a generalization of the stochastic k-TSP (Ene et al., 2018).
Revised: With penalties, this problem is a generalization of the stochastic k-TSP (Ene et al., 2018).

Original: Since our model constructs a tour one node at the time, we only need to use the real prizes to compute the remaining prize constraint.
Revised: Since our model constructs a tour one node at the time, we only need to use the real prizes to compute the remaining prize constraint.

Original: By contrast, any algorithm that selects a ﬁxed tour may fail to satisfy the prize constraint so an algorithm must be adaptive.
Revised: By contrast, any algorithm that selects a ﬁxed tour may fail to satisfy the prize constraint so an algorithm must be adaptive.

Original: As a baseline, we implement an algorithm that plans a tour, executes part of it and then re-optimizes using the C++ ILS algorithm.
Revised: As a baseline, we implement an algorithm that plans a tour, executes part of it and then re-optimizes using the C++ ILS algorithm.

Original: We either execute all node visits (so planning additional nodes if the result does not satisfy the prize constraint), half of the planned node visits (for O (log n ) replanning iterations) or only the ﬁrst node visit, for maximum adaptivity.
Revised: We either execute all node visits (so planning additional nodes if the result does not satisfy the prize constraint), half of the planned node visits (for O (log n ) replanning iterations) or only the ﬁrst node visit, for maximum adaptivity.

Original: We observe that our model outperforms all baselines for n = 20 .
Revised: We observe that our model outperforms all baselines for n = 20 .

Original: We think that failure to account for uncertainty (by the baselines) in the prize might result in the need to visit one or two additional nodes, which is relatively costly for small instances but relatively cheap for larger n .
Revised: We think that failure to account for uncertainty (by the baselines) in the prize might result in the need to visit one or two additional nodes, which is relatively costly for small instances but relatively cheap for larger n .

Original: Still, our method is beneﬁcial as it provides competitive solutions at a fraction of the computational cost, which is important in online settings.
Revised: Still, our method is beneﬁcial as it provides competitive solutions at a fraction of the computational cost, which is important in online settings.

Original: 5. A TTENTION M ODEL VS . P OINTER N ETWORK AND DIFFERENT BASELINES
Revised: 5. A TTENTION M ODEL VS . P OINTER N ETWORK AND DIFFERENT BASELINES

Original: Figure 3 compares the performance of the TSP
Revised: Figure 3 compares the performance of the TSP

Original: Attention Model (AM) and our implementation of the Pointer Network (PN) during training.
Revised: Attention Model (AM) and our implementation of the Pointer Network (PN) during training.

Original: We use a validation set of size 10000 with greedy decoding, and compare to using an exponential ( β = 0 . 8 ) and a critic (see Appendix B.1) baseline.
Revised: We use a validation set of size 10000 with greedy decoding, and compare to using an exponential ( β = 0 . 8 ) and a critic (see Appendix B.1) baseline.

Original: We used two random seeds and a decaying learning rate of η = 10 − 3 × 0 .
Revised: We used two random seeds and a decaying learning rate of η = 10 − 3 × 0 .

Original: 96 epoch .
Revised: 96 epoch .

Original: This performs best for the
Revised: This performs best for the

Original: PN, while for the AM results are similar to using η = 10 − 4 (see Appendix B.4).
Revised: PN, while for the AM results are similar to using η = 10 − 4 (see Appendix B.5).

Original: This clearly illustrates how the improvement we obtain is the result of both the AM and the rollout baseline: the AM outperforms the PN using any baseline and the rollout baseline improves the quality and convergence speed for both AM and PN.
Revised: This clearly illustrates how the improvement we obtain is the result of both the AM and the rollout baseline: the AM outperforms the PN using any baseline and the rollout baseline improves the quality and convergence speed for both AM and PN.

Original: For the PN with critic baseline, we are unable to reproduce the 1 . 5% reported by Bello et al.
Revised: For the PN with critic baseline, we are unable to reproduce the 1 . 5% reported by Bello et al.

Original: (also when using an
Revised: (also when using an

Original: LSTM based critic), but our reproduction is closer than others have reported (Dai et al., 2017; Nazari et al., 2018).
Revised: LSTM based critic), but our reproduction is closer than others have reported (Dai et al., 2017; Nazari et al., 2018).

Original: In Table 1 we compare against the original results.
Revised: In Table 1 we compare against the original results.

Original: Compared to the rollout baseline, the exponential baseline is around 20% faster per epoch, whereas the critic baseline is around 13% slower (see Appendix B.4), so the picture does not change signiﬁcantly if time is used as x-axis.
Revised: Compared to the rollout baseline, the exponential baseline is around 20% faster per epoch, whereas the critic baseline is around 13% slower (see Appendix B.5), so the picture does not change signiﬁcantly if time is used as x-axis.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we have introduced a model and training method which both contribute to signiﬁcantly improved results on learned heuristics for TSP and additionally learned strong (single construction) heuristics for multiple routing problems, which are traditionally solved by problem-speciﬁc approaches.
Revised: In this work we have introduced a model and training method which both contribute to signiﬁcantly improved results on learned heuristics for TSP and additionally learned strong (single construction) heuristics for multiple routing problems, which are traditionally solved by problem-speciﬁc approaches.

Original: We believe that our method is a powerful starting point for learning heuristics for other combinatorial optimization problems deﬁned on graphs, if their solutions can be described as sequential decisions.
Revised: We believe that our method is a powerful starting point for learning heuristics for other combinatorial optimization problems deﬁned on graphs, if their solutions can be described as sequential decisions.

Original: In practice, operational constraints often lead to many variants of problems for which no good (human-designed) heuristics are available such that the ability to learn heuristics could be of great practical value.
Revised: In practice, operational constraints often lead to many variants of problems for which no good (human-designed) heuristics are available such that the ability to learn heuristics could be of great practical value.

Revised: Compared to previous works, by using attention instead of recurrence (LSTMs) we introduce invariance to the input order of the nodes, increasing learning efﬁciency.

Revised: Also this enables parallelization, for increased computational efﬁciency.

Revised: The multi-head attention mechanism can be seen as a message passing algorithm that allows nodes to communicate relevant information over different channels, such that the node embeddings from the encoder can learn to include valuable information about the node in the context of the graph .

Revised: This information is important in our setting where decisions relate directly to the nodes in a graph.

Revised: Being a graph based method, our model has increased scaling potential (compared to LSTMs) as it can be applied on a sparse graph and operate locally.

Original: Scaling to larger problem instances is an important direction for future research, where we think we have made an important ﬁrst step by using a graph based method, as a large graph could be sparsiﬁed for improved computational efﬁciency.
Revised: Scaling to larger problem instances is an important direction for future research, where we think we have made an important ﬁrst step by using a graph based method, which can be sparsiﬁed for improved computational efﬁciency.

Original: Another challenge is that many problems of practical importance have feasibility constraints that cannot be satisﬁed by a simple masking procedure, and we think it is promising to investigate if these problems can be addressed by a combination of heuristic learning and backtracking.
Revised: Another challenge is that many problems of practical importance have feasibility constraints that cannot be satisﬁed by a simple masking procedure, and we think it is promising to investigate if these problems can be addressed by a combination of heuristic learning and backtracking.

Original: This would unleash the potential of our method, already highly competitive to the popular Google OR Tools project, to an even larger class of difﬁcult practical problems.
Revised: This would unleash the potential of our method, already highly competitive to the popular Google OR Tools project, to an even larger class of difﬁcult practical problems.

Original: A A TTENTION MODEL DETAILS
Revised: A A TTENTION MODEL DETAILS

Original: Attention mechanism We interpret the attention mechanism by Vaswani et al.
Revised: Attention mechanism We interpret the attention mechanism by Vaswani et al.

Original: (2017) as a weighted message passing algorithm between nodes in a graph.
Revised: (2017) as a weighted message passing algorithm between nodes in a graph.

Original: The weight of the message value that a node receives from a neighbor depends on the compatibility of its query with the key of the neighbor, as illustrated in Figure 4.
Revised: The weight of the message value that a node receives from a neighbor depends on the compatibility of its query with the key of the neighbor, as illustrated in Figure 4.

Original: Formally, we deﬁne dimensions d k and d v and compute the key k i ∈ R d k , value v i ∈ R d v and query q i ∈ R
Revised: Formally, we deﬁne dimensions d k and d v and compute the key k i ∈ R d k , value v i ∈ R d v and query q i ∈ R

Original: d k for each node by projecting the embedding h i :
Revised: d k for each node by projecting the embedding h i :

Original: [Equation]
Revised: [Equation]

Original: Here parameters W Q and W K are ( d k × d h ) matrices and W V has size ( d v × d h ) .
Revised: Here parameters W Q and W K are ( d k × d h ) matrices and W V has size ( d v × d h ) .

Original: From the queries and keys, we compute the compatibility u ij ∈ R of the query q i of node i with the key k j of node j as the (scaled, see Vaswani et al.
Revised: From the queries and keys, we compute the compatibility u ij ∈ R of the query q i of node i with the key k j of node j as the (scaled, see Vaswani et al.

Original: dot-product:
Revised: dot-product:

Original: [Equation]
Revised: [Equation]

Original: In a general graph, deﬁning the compatibility of non-adjacent nodes as −∞ prevents message passing between these nodes.
Revised: In a general graph, deﬁning the compatibility of non-adjacent nodes as −∞ prevents message passing between these nodes.

Original: From the compatibilities u ij , we compute the attention weights a ij ∈ [0 , 1] using a softmax:
Revised: From the compatibilities u ij , we compute the attention weights a ij ∈ [0 , 1] using a softmax:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, the vector h (cid:48) i that is received by node i is the convex combination of messages v j :
Revised: Finally, the vector h (cid:48) i that is received by node i is the convex combination of messages v j :

Original: [Equation]
Revised: [Equation]

Original: Multi-head attention As was noted by Vaswani et al.
Revised: Multi-head attention As was noted by Vaswani et al.

Original: (2017) and Velickovic et al.
Revised: (2017) and Velickovic et al.

Original: (2018), it is beneﬁcial to have multiple attention heads.
Revised: (2018), it is beneﬁcial to have multiple attention heads.

Original: This allows nodes to receive different types of messages from different neighbors.
Revised: This allows nodes to receive different types of messages from different neighbors.

Original: Especially, we compute the value in equation 13 M = 8 times with different parameters, using d k = d v = d h M = 16 .
Revised: Especially, we compute the value in equation 13 M = 8 times with different parameters, using d k = d v = d h M = 16 .

Original: We denote the result vectors by h (cid:48)
Revised: We denote the result vectors by h (cid:48)

Original: im for m ∈ 1 , . . .
Revised: im for m ∈ 1 , . . .

Original: , M .
Revised: , M .

Original: These are projected back to a single d h -dimensional vector using ( d h × d v ) parameter matrices W Om .
Revised: These are projected back to a single d h -dimensional vector using ( d h × d v ) parameter matrices W Om .

Original: The ﬁnal multi-head attention value for node i is a function of h 1 , . .
Revised: The ﬁnal multi-head attention value for node i is a function of h 1 , . .

Original: , h n through h (cid:48)
Revised: , h n through h (cid:48)

Original: im :
Revised: im :

Original: [Equation]
Revised: [Equation]

Original: Feed-forward sublayer
Revised: Feed-forward sublayer

Original: The feed-forward sublayer computes node-wise projections using a hidden (sub)sublayer with dimension d ff = 512 and a ReLu activation:
Revised: The feed-forward sublayer computes node-wise projections using a hidden (sub)sublayer with dimension d ff = 512 and a ReLu activation:

Original: [Equation]
Revised: [Equation]

Original: Batch normalization We use batch normalization with learnable d h -dimensional afﬁne parameters w bn and b bn :
Revised: Batch normalization We use batch normalization with learnable d h -dimensional afﬁne parameters w bn and b bn :

Original: [Equation]
Revised: [Equation]

Original: Here (cid:12) denotes the element-wise product and BN refers to batch normalization without afﬁne transformation.
Revised: Here (cid:12) denotes the element-wise product and BN refers to batch normalization without afﬁne transformation.

Original: BT
Revised: B

Original:  RAVELLING S ALESMAN P ROBLEM
Revised: T RAVELLING S ALESMAN P ROBLEM

Original: B. C RITIC ARCHITECTURE
Revised: B. C RITIC ARCHITECTURE

Original: The critic network architecture uses 3 attention layers similar to our encoder, after which the node embeddings are averaged and processed by an MLP with one hidden layer with 128 neurons and
Revised: The critic network architecture uses 3 attention layers similar to our encoder, after which the node embeddings are averaged and processed by an MLP with one hidden layer with 128 neurons and

Original: ReLu activation and a single output.
Revised: ReLu activation and a single output.

Original: We used the same learning rate as for the AM/PN in all experiments.
Revised: We used the same learning rate as for the AM/PN in all experiments.

Original: B. I NSTANCE GENERATION
Revised: B. I NSTANCE GENERATION

Original: For all TSP instances, the n node locations are sampled uniformly at random in the unit square.
Revised: For all TSP instances, the n node locations are sampled uniformly at random in the unit square.

Original: This distribution is chosen to be neither easy nor artiﬁcially hard and to be able to compare to other learned heuristics.
Revised: This distribution is chosen to be neither easy nor artiﬁcially hard and to be able to compare to other learned heuristics.

Original: B. D ETAILS OF BASELINES
Revised: B. D ETAILS OF BASELINES

Original: This section describes details of the heuristics implemented for the TSP.
Revised: This section describes details of the heuristics implemented for the TSP.

Original: All of the heuristics construct a single tour in a single pass, by extending a partial solution one node at the time.
Revised: All of the heuristics construct a single tour in a single pass, by extending a partial solution one node at the time.

Original: Nearest neighbor The nearest neighbor heuristic represents the partial solution as a path with a start and end node.
Revised: Nearest neighbor The nearest neighbor heuristic represents the partial solution as a path with a start and end node.

Original: The initial path is formed by a single node, selected randomly, which becomes the start node but also the end node of the initial path.
Revised: The initial path is formed by a single node, selected randomly, which becomes the start node but also the end node of the initial path.

Original: In each iteration, the next node is selected as the node nearest to the end node of the partial path.
Revised: In each iteration, the next node is selected as the node nearest to the end node of the partial path.

Original: This node is added to the path and becomes the new end node.
Revised: This node is added to the path and becomes the new end node.

Original: Finally, after all nodes are added this way, the end node is connected with the start node to form a tour.
Revised: Finally, after all nodes are added this way, the end node is connected with the start node to form a tour.

Original: In our implementation, for deterministic results we always start with the ﬁrst node in the input, which can be considered random as the instances are generated randomly.
Revised: In our implementation, for deterministic results we always start with the ﬁrst node in the input, which can be considered random as the instances are generated randomly.

Original: Farthest/nearest/random insertion The insertion heuristics represent a partial solution as a tour , and extends it by inserting nodes one node at the time.
Revised: Farthest/nearest/random insertion The insertion heuristics represent a partial solution as a tour , and extends it by inserting nodes one node at the time.

Original: In our implementation, we always insert the node using the cheapest insertion cost.
Revised: In our implementation, we always insert the node using the cheapest insertion cost.

Original: This means that when node i is inserted, the place of insertion (between adjacent nodes j and k in the tour) is selected such that it minimizes the insertion costs d ji + d ik − d jk , where d ji , d ik and d jk represent the distances from node j to i , i to k and j to k , respectively.
Revised: This means that when node i is inserted, the place of insertion (between adjacent nodes j and k in the tour) is selected such that it minimizes the insertion costs d ji + d ik − d jk , where d ji , d ik and d jk represent the distances from node j to i , i to k and j to k , respectively.

Original: The different variants of the insertion heuristic vary in the way in which the node which is inserted is selected.
Revised: The different variants of the insertion heuristic vary in the way in which the node which is inserted is selected.

Original: Let S be the set of nodes in the partial tour.
Revised: Let S be the set of nodes in the partial tour.

Original: Nearest insertion inserts the node i that is nearest to (any node in) the tour:
Revised: Nearest insertion inserts the node i that is nearest to (any node in) the tour:

Original: [Equation]
Revised: [Equation]

Original: Farthest insertion inserts the node i such that the distance to the tour (i.e. the distance from i to the nearest node j in the tour) is maximized:
Revised: Farthest insertion inserts the node i such that the distance to the tour (i.e. the distance from i to the nearest node j in the tour) is maximized:

Original: [Equation]
Revised: [Equation]

Original: Random insertion inserts a random node.
Revised: Random insertion inserts a random node.

Original: Similar to nearest neighbor, we consider the input order random so we simply insert the nodes in this order.
Revised: Similar to nearest neighbor, we consider the input order random so we simply insert the nodes in this order.

Original: B. E XTENDED RESULTS
Revised: B. C OMPARISON TO CONCURRENT WORK

Revised: Independently of our work, Deudon et al.

Revised: (2018) also developed a model for TSP based on the

Revised: Transformer (Vaswani et al., 2017).

Revised: There are important differences to this paper:

Revised: • As ‘context’ for the decoder, Deudon et al.

Revised: (2018) use the embeddings of the last K =visited nodes.

Revised: We use only the last (e.g. K = 1 ) node but add the ﬁrst visited node (as well as the graph embedding), since the ﬁrst node is important (it is the destination) while the order of the other nodes is irrelevant as we explain in Section 3.

Revised: • Deudon et al.

Revised: (2018) use a critic as baseline (which also uses the Transformer architecture).

Revised: We also experiment with using a critic (based on the Transformer architecture), but found that using a rollout baseline is much more effective (see Section 5).

Revised: • Deudon et al.

Revised: (2018) report results with sampling 128 solutions, with and without 2OPT local search.

Revised: We report results without 2OPT, using either a single greedy solution or sampling 1280 solutions and additionally show how this directly improves performance compared to Bello et al.

Revised: • By adding 2OPT on top of the best sampled solution, Deudon et al.

Revised: (2018) show that the model does not produce a local optimum and results can improve by using a ‘hybrid’ approach of a learned algorithm with local search.

Revised: This is a nice example of combining learned and traditional heuristics, but it is not compared against using the Pointer Network (Bello et al., 2016) with 2OPT.

Revised: • The model of Deudon et al.

Revised: (2018) uses a higher dimensionality internally in the decoder (for details see their paper).

Revised: Training is done with 20000 steps with a batch size of 256.

Revised: • Deudon et al.

Revised: (2018) apply Principal Component Analysis (PCA) on the input coordinates to eliminate rotation symmetry whereas we directly input node coordinates.

Revised: • Additionally to TSP, we also consider two variants of VRP, the OP with different prize distributions and the (stochastic) PCTSP.

Revised: We want to emphasize that this is independent work, but for completeness we include a full emperical comparison of performance.

Revised: Since the results presented in the paper by Deudon et al.

Revised: (2018) are not directly comparable, we ran their code 6 and report results under the same circumstances: using greedy decoding and sampling 1280 solutions on our test dataset (which has exactly the same generative procedure, e.g. uniform in the unit square).

Revised: Additionally, we include results of their model with 2OPT, showing that (even without 2OPT) ﬁnal performance of our model is better.

Revised: We use the hyperparameters in their code, but increase the batch size to 512 and number of training steps to 100 × 2500 = 250000 for a fair comparison (this increased the performance of their model).

Revised: As training with n = 100 gave out-of-memory errors, we train only on n = 20 and n = 50 and (following Deudon et al.

Revised: (2018)) report results for n = 100 using the model trained for n = 50 .

Revised: The training time as well as test run times are comparable.

Revised: B. E XTENDED RESULTS

Original: Hyperparameters We found in general that using a larger learning rate of 10 − 3 works better with decay but may be unstable in some cases.
Revised: Hyperparameters We found in general that using a larger learning rate of 10 − 3 works better with decay but may be unstable in some cases.

Original: A smaller learning rate 10 − 4 is more stable and does not require decay.
Revised: A smaller learning rate 10 − 4 is more stable and does not require decay.

Original: This is illustrated in Figure 6, which shows validation results over time using both 10 − 3 and 10 − 4 with and without decay for TSP20 and TSP50 (2 seeds).
Revised: This is illustrated in Figure 6, which shows validation results over time using both 10 − 3 and 10 − 4 with and without decay for TSP20 and TSP50 (2 seeds).

Original: As can be seen, without decay the method has not yet fully converged after 100 epochs and results may improve even further with longer training.
Revised: As can be seen, without decay the method has not yet fully converged after 100 epochs and results may improve even further with longer training.

Original: Table 2 shows the results in absolute terms as well as the relative optimality gap compared to Gurobi, for all runs using seeds 1234 and 1235 with the two different learning rate schedules.
Revised: Table 2 shows the results in absolute terms as well as the relative optimality gap compared to Gurobi, for all runs using seeds 1234 and 1235 with the two different learning rate schedules.

Original: We did not[Table] run ﬁnal experiments for n = 100 with the larger learning rate as we found training with the smaller learning rate to be more stable.
Revised: We did not run ﬁnal experiments for n = 100 with the larger learning rate as we found training with the smaller learning rate to be more stable.

Original: It can be seen that in most cases the end results with different learning rate schedules are similar, except for the larger models ( N = 5 , N = 8 ) where some of the runs diverged using the larger learning rate.
Revised: It can be seen that in most cases the end results with different learning rate schedules are similar, except for the larger models ( N = 5 , N = 8 ) where some of the runs diverged using the larger learning rate.

Original: Experiments with different number of layers N show that N = 3 and N = 5 achieve best performance, and we ﬁnd N = 3 is a good trade-off between quality of the results and computational complexity (runtime) of the model.
Revised: Experiments with different number of layers N show that N = 3 and N = 5 achieve best performance, and we ﬁnd N = 3 is a good trade-off between quality of the results and computational complexity (runtime) of the model.

Original: Generalization We test generalization performance on different n than trained for, which we plot in Figure 5 in terms of the relative optimality gap compared to Gurobi.
Revised: Generalization We test generalization performance on different n than trained for, which we plot in Figure 5 in terms of the relative optimality gap compared to Gurobi.

Original: The train sizes are indicated with vertical marker bars.
Revised: The train sizes are indicated with vertical marker bars.

Original: The models generalize when tested on different sizes, although quality degrades as the difference becomes bigger, which can be expected as there is no free lunch (Wolpert & Macready, 1997).
Revised: The models generalize when tested on different sizes, although quality degrades as the difference becomes bigger, which can be expected as there is no free lunch (Wolpert & Macready, 1997).

Original: Since the architectures are the same, these differences mean the models learn to specialize on the problem sizes trained for.
Revised: Since the architectures are the same, these differences mean the models learn to specialize on the problem sizes trained for.

Original: We can make a strong overall algorithm by selecting the trained model with highest validation performance for each instance size n (marked in Figureby the red bar).
Revised: We can make a strong overall algorithm by selecting the trained model with highest validation performance for each instance size n (marked in Figureby the red bar).

Original: For reference, we also include the baselines, where for the methods that perform search or sampling we do not connect the dots to prevent cluttering and to make the distinction with methods that consider only a single solution clear.
Revised: For reference, we also include the baselines, where for the methods that perform search or sampling we do not connect the [Table] Critic 6:10 3 . 87 (0 . (0 . 88%) AM / Rollout 5:30 3 . 85 (0 .

Revised: PN / Exponential 5:10 3 .

Original:  [Figure]
Revised: 93 (2 . 37%) PN / Critic 7:30 3 . 94 (2 . 84%) PN / Rollout 6:40 3 . [Figure]

Original: C V EHICLE R OUTING P ROBLEM
Revised: C V EHICLE R OUTING P ROBLEM

Original: The Capacitated Vehicle Routing Problem (CVRP) is a generalization of the TSP in which case there is a depot and multiple routes should be created, each starting and ending at the depot.
Revised: The Capacitated Vehicle Routing Problem (CVRP) is a generalization of the TSP in which case there is a depot and multiple routes should be created, each starting and ending at the depot.

Original: In our graph based formulation, we add a special depot node with index 0 and coordinates x 0 .
Revised: In our graph based formulation, we add a special depot node with index 0 and coordinates x 0 .

Original: A vehicle (route) has capacity D > 0 and each (regular) node i ∈ { 1 , . . .
Revised: A vehicle (route) has capacity D > 0 and each (regular) node i ∈ { 1 , . . .

Original: n } has a demand 0 < δ i ≤ D .
Revised: n } has a demand 0 < δ i ≤ D .

Original: Each route starts and ends at the depot and the total demand in each route should not exceed the capacity, so (cid:80) i ∈ R
Revised: Each route starts and ends at the depot and the total demand in each route should not exceed the capacity, so (cid:80) i ∈ R

Original: j δ i ≤ D , where R j is the set of node indices assigned to route j .
Revised: j δ i ≤ D , where R j is the set of node indices assigned to route j .

Original: Without loss of generality, we assume a normalized ˆ D = 1 as we can use normalized demands ˆ δ i =
Revised: Without loss of generality, we assume a normalized ˆ D = 1 as we can use normalized demands ˆ δ i =

Original: δ i D .
Revised: δ i D .

Original: The Split Delivery VRP (SDVRP) is a generalization of CVRP in which every node can be visited multiple times, and only a subset of the demand has to be delivered at each visit.
Revised: The Split Delivery VRP (SDVRP) is a generalization of CVRP in which every node can be visited multiple times, and only a subset of the demand has to be delivered at each visit.

Original: Instances for both
Revised: Instances for both

Original: CVRP and SDVRP are speciﬁed in the same way: an instance with size n as a depot location x 0 , n node locations x i , i = 1 . . .
Revised: CVRP and SDVRP are speciﬁed in the same way: an instance with size n as a depot location x 0 , n node locations x i , i = 1 . . .

Original: n and (normalized) demands 0 <
Revised: n and (normalized) demands 0 <

Original: ˆ δ i ≤ 1 , i = 1 . . .
Revised: ˆ δ i ≤ 1 , i = 1 . . .

Original: n .
Revised: n .

Original: C. I NSTANCE GENERATION
Revised: C. I NSTANCE GENERATION

Original: We follow Nazari et al.
Revised: We follow Nazari et al.

Original: (2018) in the generation of instances for n = 20 , 50 , 100 , but normalize the demands by the capacities.
Revised: (2018) in the generation of instances for n = 20 , 50 , 100 , but normalize the demands by the capacities.

Original: The depot location as well as n node locations are sampled uniformly at random in the unit square.
Revised: The depot location as well as n node locations are sampled uniformly at random in the unit square.

Original: The demands are deﬁned as ˆ δ i
Revised: The demands are deﬁned as ˆ δ i

Original: =
Revised: =

Original: δ i D n where δ i is discrete and sampled uniformly from { 1 , . .
Revised: δ i D n where δ i is discrete and sampled uniformly from { 1 , . .

Original: , 9 } and D 20 = 30 , D 50 = 40 and D 100 = 50.
Revised: , 9 } and D 20 = 30 , D 50 = 40 and D 100 = 50.

Original: C. A TTENTION M ODEL FOR THE VRP
Revised: C. A TTENTION M ODEL FOR THE VRP

Original: Encoder In order to allow our Attention Model to distinguish the depot node from the regular nodes, we use separate parameters W x 0 and b x 0 to compute the initial embedding h (0)0 of the depot node.
Revised: Encoder In order to allow our Attention Model to distinguish the depot node from the regular nodes, we use separate parameters W x 0 and b x 0 to compute the initial embedding h (0)0 of the depot node.

Original: Additionally, we provide the normalized demand δ i as input feature (and adjust the size of parameter W x accordingly):
Revised: Additionally, we provide the normalized demand δ i as input feature (and adjust the size of parameter W x accordingly):

Original: [Equation]
Revised: [Equation]

Original: Capacity constraints To facilitate the capacity constraints, we keep track of the remaining demands ˆ δ i,t for the nodes i ∈ { 1 , . . .
Revised: Capacity constraints To facilitate the capacity constraints, we keep track of the remaining demands ˆ δ i,t for the nodes i ∈ { 1 , . . .

Original: n } and remaining vehicle capacity ˆ D t at time t .
Revised: n } and remaining vehicle capacity ˆ D t at time t .

Original: At t = 1 , these are initialized as ˆ δ i,t
Revised: At t = 1 , these are initialized as ˆ δ i,t

Original: = ˆ
Revised: = ˆ

Original: δ i and ˆ D t = 1 , after which they are updated as follows (recall that π t is the index of the node selected at decoding step t ):
Revised: δ i and ˆ D t = 1 , after which they are updated as follows (recall that π t is the index of the node selected at decoding step t ):

Original: [Equation]
Revised: [Equation]

Original: If we do not allow split deliveries, ˆ δ i,t will be either 0 or ˆ
Revised: If we do not allow split deliveries, ˆ δ i,t will be either 0 or ˆ

Original: δ i for all t .
Revised: δ i for all t .

Original: Decoder context The context for the decoder for the VRP at time t is the current/last location π t − 1 and the remaining capacity ˆ D t .
Revised: Decoder context The context for the decoder for the VRP at time t is the current/last location π t − 1 and the remaining capacity ˆ D t .

Original: Compared to TSP, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot:
Revised: Compared to TSP, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Masking The depot can be visited multiple times, but we do not allow it to be visited at two subsequent timesteps.
Revised: Masking The depot can be visited multiple times, but we do not allow it to be visited at two subsequent timesteps.

Original: Therefore, in both layers of the decoder, we change the masking for the depot j = 0 and deﬁne u ( c )

Revised: Therefore, in both layers of the decoder, we change the masking for the depot j = 0 and deﬁne u ( c )

Original: =
Revised: =

Original: −∞ if (and only if) t = 1 or π t − 1 = 0 .
Revised: −∞ if (and only if) t = 1 or π t − 1 = 0 .

Original: The masking for the nodes depends on whether we allow split deliveries.
Revised: The masking for the nodes depends on whether we allow split deliveries.

Original: Without split deliveries, we do not allow nodes to be visited if their remaining demand is 0 (if the node was already visited) or exceeds the remaining capacity, so for j (cid:54)
Revised: Without split deliveries, we do not allow nodes to be visited if their remaining demand is 0 (if the node was already visited) or exceeds the remaining capacity, so for j (cid:54)

Original: = 

Revised: =

Original: we deﬁne u ( c ) j
Revised: we deﬁne u ( c ) j

Original: =
Revised: =

Original: −∞ if (and only if) ˆ δ i,t = 0 or ˆ δ i,t > ˆ D t .
Revised: −∞ if (and only if) ˆ δ i,t = 0 or ˆ δ i,t > ˆ D t .

Original: With split deliveries, we only forbid delivery when the remaining demand is 0, so we deﬁne u ( c )
Revised: With split deliveries, we only forbid delivery when the remaining demand is 0, so we deﬁne u ( c )

Original: j = −∞ if (and only if) ˆ δ i,t = 0 .
Revised: j = −∞ if (and only if) ˆ δ i,t = 0 .

Original: Split deliveries Without split deliveries, the remaining demand ˆ δ i,t is either 0 or ˆ δ i , corresponding to whether the node has been visited or not, and this information is conveyed to the model via the masking of the nodes already visited.
Revised: Split deliveries Without split deliveries, the remaining demand ˆ δ i,t is either 0 or ˆ δ i , corresponding to whether the node has been visited or not, and this information is conveyed to the model via the masking of the nodes already visited.

Original: However, when split deliveries are allowed, the remaining demand ˆ δ i,t can take any value 0 ≤ ˆ δ i,t ≤ ˆ δ i .
Revised: However, when split deliveries are allowed, the remaining demand ˆ δ i,t can take any value 0 ≤ ˆ δ i,t ≤ ˆ δ i .

Original: This information cannot be included in the context node as it corresponds to individual nodes.
Revised: This information cannot be included in the context node as it corresponds to individual nodes.

Original: Therefore we include it in the computation of the keys and values in both the attention layer (glimpse) and the output layer of the decoder, such that we compute queries, keys and values using:
Revised: Therefore we include it in the computation of the keys and values in both the attention layer (glimpse) and the output layer of the decoder, such that we compute queries, keys and values using:

Original: [Equation]
Revised: [Equation]

Original: Here we W Kd and W Vd are ( d k × 1) parameter matrices and we deﬁne ˆ δ i,t = 0 for the depot i = 0 .
Revised: Here we W Kd and W Vd are ( d k × 1) parameter matrices and we deﬁne ˆ δ i,t = 0 for the depot i = 0 .

Original: Summing the projection of both h i and ˆ δ i,t is equivalent to projecting the concatenation [ h i , ˆ δ i,t ] with a single (( d h + 1) ×
Revised: Summing the projection of both h i and ˆ δ i,t is equivalent to projecting the concatenation [ h i , ˆ δ i,t ] with a single (( d h + 1) ×

Original: d k ) matrix W K .
Revised: d k ) matrix W K .

Original: However, using this formulation we only need to compute the ﬁrst term once (instead for every t ) and by the weight initialization this puts more importance on ˆ δ i,t initially (which is otherwise just 1 of d h + 1 = 129 input values).
Revised: However, using this formulation we only need to compute the ﬁrst term once (instead for every t ) and by the weight initialization this puts more importance on ˆ δ i,t initially (which is otherwise just 1 of d h + 1 = 129 input values).

Original: Training For the VRP, the length of the output of the model depends on the number of times the depot is visited.
Revised: Training For the VRP, the length of the output of the model depends on the number of times the depot is visited.

Original: In general, the depot is visited multiple times, and in the case of SDVRP also some regular nodes are visited twice.
Revised: In general, the depot is visited multiple times, and in the case of SDVRP also some regular nodes are visited twice.

Original: Therefore the length of the solution is larger than n , which requires more memory such that we ﬁnd it necessary to limit the batch size B to 256 for n = 100 (on
Revised: Therefore the length of the solution is larger than n , which requires more memory such that we ﬁnd it necessary to limit the batch size B to 256 for n = 100 (on

Original: GPUs).
Revised: GPUs).

Original: To keep training times tractable and the total number of parameter updates equal, we still process 2500 batches per epoch, for a total of 0.64M training instances per epoch.
Revised: To keep training times tractable and the total number of parameter updates equal, we still process 2500 batches per epoch, for a total of 0.64M training instances per epoch.

Original: C. D ETAILS OF BASELINES
Revised: C. D ETAILS OF BASELINES

Original: For LKH3 5 by Helsgaun (2017) we build and run their code with the SPECIAL parameter as speciﬁed in their CVRP runscript 6 .
Revised: For LKH3 7 by Helsgaun (2017) we build and run their code with the SPECIAL parameter as speciﬁed in their CVRP runscript 8 .

Original: We perform 1 run with a maximum of 10000 trials, as we found performing 10 runs only marginally improves the quality of the results while taking much more time.
Revised: We perform 1 run with a maximum of 10000 trials, as we found performing 10 runs only marginally improves the quality of the results while taking much more time.

Original: C. E XAMPLE SOLUTIONS
Revised: C. E XAMPLE SOLUTIONS

Original: Figure 7 shows example solutions for the CVRP with n = 100 that were obtained by a single construction using the model with greedy decoding.
Revised: Figure 7 shows example solutions for the CVRP with n = 100 that were obtained by a single construction using the model with greedy decoding.

Original: These visualizations give insight in the heuristic that the model has learned.
Revised: These visualizations give insight in the heuristic that the model has learned.

Original: In general we see that the model constructs the routes from the bottom to the top, starting below the depot.
Revised: In general we see that the model constructs the routes from the bottom to the top, starting below the depot.

Original: Most routes are densely packed, except for the last route that has to serve some remaining (close to each other) customers.
Revised: Most routes are densely packed, except for the last route that has to serve some remaining (close to each other) customers.

Original: In most cases, the node in the route that is farthest from the depot is somewhere in the middle of the route, such that customers are served on the way to and from the farthest nodes.
Revised: In most cases, the node in the route that is farthest from the depot is somewhere in the middle of the route, such that customers are served on the way to and from the farthest nodes.

Original: In some cases, we see that the order of stops within some individual routes is suboptimal, which means that the method will likely beneﬁt from simple further optimizations on top, such as a beam search, a post-processing procedure based on local search (e.g.
Revised: In some cases, we see that the order of stops within some individual routes is suboptimal, which means that the method will likely beneﬁt from simple further optimizations on top, such as a beam search, a post-processing procedure based on local search (e.g.

Original: 2OPT) or solving the individual routes using a TSP solver.
Revised: 2OPT) or solving the individual routes using a TSP solver.

Original: [Figure]
Revised: [Figure]

Original: D O RIENTEERING P ROBLEM
Revised: D O RIENTEERING P ROBLEM

Original: In the Orienteering Problem (OP) each node has a prize ρ i and the goal is to maximize the total prize of nodes visited, while keeping the total length of the route below a maximum length T .
Revised: In the Orienteering Problem (OP) each node has a prize ρ i and the goal is to maximize the total prize of nodes visited, while keeping the total length of the route below a maximum length T .

Original: This problem is different from the TSP and the VRP because visiting each node is optional.
Revised: This problem is different from the TSP and the VRP because visiting each node is optional.

Original: Similar to the
Revised: Similar to the

Original: VRP, we add a special depot node with index 0 and coordinates x 0 .
Revised: VRP, we add a special depot node with index 0 and coordinates x 0 .

Original: If the model selects the depot, we consider the route to be ﬁnished.
Revised: If the model selects the depot, we consider the route to be ﬁnished.

Original: In order to prevent infeasible solutions, we only allow to visit a node if after visiting that node a return to the depot is still possible within the maximum length constraint.
Revised: In order to prevent infeasible solutions, we only allow to visit a node if after visiting that node a return to the depot is still possible within the maximum length constraint.

Original: Note that it is always suboptimal to visit the depot if additional nodes can be visited, but we do not enforce this knowledge.
Revised: Note that it is always suboptimal to visit the depot if additional nodes can be visited, but we do not enforce this knowledge.

Original: D. I NSTANCE GENERATION
Revised: D. I NSTANCE GENERATION

Original: The depot location as well as n node locations are sampled uniformly at random in the unit square.
Revised: The depot location as well as n node locations are sampled uniformly at random in the unit square.

Original: For the distribution of the prizes, we consider three different variants described by Fischetti et al.
Revised: For the distribution of the prizes, we consider three different variants described by Fischetti et al.

Original: but we normalize the prizes ρ i such that the normalized prizes
Revised: but we normalize the prizes ρ i such that the normalized prizes

Original: ˆ ρ i are between 0 and 1.
Revised: ˆ ρ i are between 0 and 1.

Original: Constant ρ i
Revised: Constant ρ i

Original: = ˆ ρ i = 1 .
Revised: = ˆ ρ i = 1 .

Original: Every node has the same prize so the goal becomes to visit as many nodes as possible within the length constraint.
Revised: Every node has the same prize so the goal becomes to visit as many nodes as possible within the length constraint.

Original: Uniform ρ i ∼ DiscreteUniform (1 , 100) , ˆ ρ i 
Revised: Uniform ρ i ∼ DiscreteUniform (1 , 100) , ˆ ρ i =

Original: = ρ
Revised: ρ

Original: i/ 100 .
Revised: i 100 .

Original: Every node has a prize that is (discretized) uniform.
Revised: Every node has a prize that is (discretized) uniform.

Original: Distance ρ i = 

Revised: Distance ρ i =

Original: + (cid:106) 99 ·d
Revised: + (cid:106) 99 ·

Original: [Equation]
Revised: [Equation]

Original: max nj =1 d 0 j (cid:107) , ˆρ i
Revised: (cid:107) , ˆ

Original: = ρ
Revised: [Equation]

Original:  i/ 100 , where d 0 i is the distance from the depot to node i .
Revised: = ρ i 100 , where d 0 i is the distance from the depot to node i .

Original: Every node has a (discretized) prize that is proportional to the distance to the depot.
Revised: Every node has a (discretized) prize that is proportional to the distance to the depot.

Original: This is designed to be challenging as the largest prizes are furthest away from the depot (Fischetti et al., 1998).
Revised: This is designed to be challenging as the largest prizes are furthest away from the depot (Fischetti et al., 1998).

Original: The maximum length T n for instances with n nodes (and a depot) is chosen to be (on average) approximately half of the length of the average TSP tour for uniform TSP instances with n nodes 7 .
Revised: The maximum length T n for instances with n nodes (and a depot) is chosen to be (on average) approximately half of the length of the average TSP tour for uniform TSP instances with n nodes 9 .

Original: This idea is that this way approximately (a little more than) half of the nodes can be visited, which results in the most difﬁcult problem instances (Vansteenwegen et al., 2011).
Revised: This idea is that this way approximately (a little more than) half of the nodes can be visited, which results in the most difﬁcult problem instances (Vansteenwegen et al., 2011).

Original: This is because the number of possible node selections (cid:0) nk (cid:1) is maximized if k = n/ 2 and additionally determining the actual path is harder with more nodes selected.
Revised: This is because the number of possible node selections (cid:0) nk (cid:1) is maximized if k = n 2 and additionally determining the actual path is harder with more nodes selected.

Original: We set ﬁxed maximum lengths T 20 = 2 , T 50 =
Revised: We set ﬁxed maximum lengths T 20 = 2 , T 50 =

Original: and T 100 = 4 instead of adjusting the constraint per instance, such that for some instances more or less nodes can be visited.
Revised: and T 100 = 4 instead of adjusting the constraint per instance, such that for some instances more or less nodes can be visited.

Original: Note that T n has the same unit as the node coordinates x i , so we do not normalize them.
Revised: Note that T n has the same unit as the node coordinates x i , so we do not normalize them.

Original: D. A TTENTION M ODEL FOR THE OP
Revised: D. A TTENTION M ODEL FOR THE OP

Original: Encoder Similar to the VRP, we use separate parameters for the depot node embedding.
Revised: Encoder Similar to the VRP, we use separate parameters for the depot node embedding.

Original: Additionally, we provide the node prize ˆ ρ i as input feature:
Revised: Additionally, we provide the node prize ˆ ρ i as input feature:

Original: [Equation]
Revised: [Equation]

Original: Max length constraint
Revised: Max length constraint

Original: In order to satisfy the max length constraint, we keep track of the remaining max length T t at time t .
Revised: In order to satisfy the max length constraint, we keep track of the remaining max length T t at time t .

Original: Starting at t = 1 , T 1 = T .
Revised: Starting at t = 1 , T 1 = T .

Original: Then for t > 0 , T is updated as
Revised: Then for t > 0 , T is updated as

Original: [Equation]
Revised: [Equation]

Original: Here d π t − 1 ,π t is the distance from node π t − 1 to π t and we conveniently deﬁne π 0 = 0 as we start at the depot.
Revised: Here d π t − 1 ,π t is the distance from node π t − 1 to π t and we conveniently deﬁne π 0 = 0 as we start at the depot.

Original: Decoder context The context for the decoder for the OP at time t is the current/last location π t −and the remaining max length T t .
Revised: Decoder context The context for the decoder for the OP at time t is the current/last location π t −and the remaining max length T t .

Original: Similar to VRP, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot.
Revised: Similar to VRP, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot.

Original: We do not need to provide information on the prizes gathered as this is irrelevant for the remaining decisions.
Revised: We do not need to provide information on the prizes gathered as this is irrelevant for the remaining decisions.

Original: The context is deﬁned as:
Revised: The context is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Masking In the OP, the depot node can always be visited so is never masked.
Revised: Masking In the OP, the depot node can always be visited so is never masked.

Original: Regular nodes are masked (i.e. cannot be visited) if either they are already visited or if they cannot be visited within the remaining length constraint:
Revised: Regular nodes are masked (i.e. cannot be visited) if either they are already visited or if they cannot be visited within the remaining length constraint:

Original: [Equation]
Revised: [Equation]

Original: D. D ETAILS OF BASELINES
Revised: D. D ETAILS OF BASELINES

Original: For Compass 8 by Kobeaga et al.
Revised: For Compass 10 by Kobeaga et al.

Original: (2018), we compile their code and run it with default parameters, only adding --op --op-ea4op to indicate that the Genetic Algorithm for the Orienteering Problem should be used.
Revised: (2018), we compile their code and run it with default parameters, only adding --op --op-ea4op to indicate that the Genetic Algorithm for the Orienteering Problem should be used.

Original: As Compass uses integer coordinates and prizes, we multiply all ﬂoats by 10and round to integers.
Revised: As Compass uses integer coordinates and prizes, we multiply all ﬂoats by 10and round to integers.

Original: We run the Python Genetic Algorithm 9 with default parameters.
Revised: We run the Python Genetic Algorithm 11 with default parameters.

Original: Tsiligirides Tsiligirides (1984) describes a heuristic procedure for solving the OP.
Revised: Tsiligirides Tsiligirides (1984) describes a heuristic procedure for solving the OP.

Original: It consists of sampling 3000 tours through a randomized construction procedure and applies local search on top.
Revised: It consists of sampling 3000 tours through a randomized construction procedure and applies local search on top.

Original: The randomized construction part of the heuristic is structurally exactly the same as the heuristic learned by our model, but with a manually engineered function to deﬁne the node probabilities.
Revised: The randomized construction part of the heuristic is structurally exactly the same as the heuristic learned by our model, but with a manually engineered function to deﬁne the node probabilities.

Original: We implement the construction part of the heuristic and compare it to our model (either greedy or sampling 1280 solutions), without the local search (as this can also be applied on top of our model).
Revised: We implement the construction part of the heuristic and compare it to our model (either greedy or sampling 1280 solutions), without the local search (as this can also be applied on top of our model).

Original: The ﬁnal heuristic used by Tsiligirides (1984) uses a formula with multiple terms to deﬁne the probability that a node should be selected, but by tuning the weights the form with only one simple term works best, showing the difﬁculty of manually deﬁning a good probability distribution.
Revised: The ﬁnal heuristic used by Tsiligirides (1984) uses a formula with multiple terms to deﬁne the probability that a node should be selected, but by tuning the weights the form with only one simple term works best, showing the difﬁculty of manually deﬁning a good probability distribution.

Original: In our terms, the heuristic deﬁnes a score s i for each node at time t as the prize divided by the distance from the current node π t − 1 , raised to the 4th power:
Revised: In our terms, the heuristic deﬁnes a score s i for each node at time t as the prize divided by the distance from the current node π t − 1 , raised to the 4th power:

Original: [Equation]
Revised: [Equation]

Original: Let S be the set with the min(4 , n − ( t − 1))
Revised: Let S be the set with the min(4 , n − ( t − 1))

Original: unvisited nodes with maximum score s i .
Revised: unvisited nodes with maximum score s i .

Original: Then the node probabilities p i at time t are deﬁned as
Revised: Then the node probabilities p i at time t are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: OR Tools For the Google OR Tools implementation, we modify the formulation for the CVRP 10 :
Revised: OR Tools For the Google OR Tools implementation, we modify the formulation for the CVRP 12 :

Original: • We replace the Manhattan distance by the Euclidian distance.
Revised: • We replace the Manhattan distance by the Euclidian distance.

Original: • We set the number of vehicles to 1.
Revised: • We set the number of vehicles to 1.

Original: • For each individual node
Revised: • For each individual node

Original: i , we add a Disjunction constraint with { i } as the set of nodes, and a penalty equal to the prize ˆ ρ i .
Revised: i , we add a Disjunction constraint with { i } as the set of nodes, and a penalty equal to the prize ˆ ρ i .

Original: This allows OR tools to skip node i at a cost ˆ ρ i .
Revised: This allows OR tools to skip node i at a cost ˆ ρ i .

Original: • We replace the capacity constraint by a maximum distance.
Revised: • We replace the capacity constraint by a maximum distance.

Original: constraint • We remove the objective to minimize the length.
Revised: constraint • We remove the objective to minimize the length.

Original: We multiply all ﬂoat inputs by 10 7 and round to integers.
Revised: We multiply all ﬂoat inputs by 10 7 and round to integers.

Original: Note that OR Tools computes penalties for skipped nodes rather than gains for nodes that are visited.
Revised: Note that OR Tools computes penalties for skipped nodes rather than gains for nodes that are visited.

Original: The problem is equivalent, but in order to compare the objective value against our method, we need to add the constant sum of all penalties (cid:80) i ˆ ρ i to the OR Tools objective.
Revised: The problem is equivalent, but in order to compare the objective value against our method, we need to add the constant sum of all penalties (cid:80) i ˆ ρ i to the OR Tools objective.

Original: python cvrp.py
Revised: https https python/cvrp py

Original: D. E XTENDED RESULTS
Revised: D. E XTENDED RESULTS

Original: [Table]
Revised: [Table]

Original: Table 3 displays the results for the OP with constant and uniform prize distributions.
Revised: Table 3 displays the results for the OP with constant and uniform prize distributions.

Original: The results are similar to the results for the prize distribution based on the distance to the depot, although by the calculation time for Gurobi it is conﬁrmed that indeed constant and uniform prize distributions are easier
Revised: The results are similar to the results for the prize distribution based on the distance to the depot, although by the calculation time for Gurobi it is conﬁrmed that indeed constant and uniform prize distributions are easier

Original: [Table]
Revised: [Table]

Original: E P RIZE C OLLECTING TSP
Revised: E P RIZE C OLLECTING TSP

Original: In the Prize Collecting TSP (PCTSP) each node has a prize ρ i and an associated penalty β i .
Revised: In the Prize Collecting TSP (PCTSP) each node has a prize ρ i and an associated penalty β i .

Original: The goal is to minimize the total length of the tour plus the sum of penalties for nodes which are not visited, while collecting at least a given minimum total prize.
Revised: The goal is to minimize the total length of the tour plus the sum of penalties for nodes which are not visited, while collecting at least a given minimum total prize.

Original: W.l.o.g.
Revised: W.l.o.g.

Original: we assume the minimum total prize is equal to 1 (as prizes can be normalized).
Revised: we assume the minimum total prize is equal to 1 (as prizes can be normalized).

Original: This problem is related to the OP but inverts the goal (minimizing tour length given a minimum total prize to collect instead of maximizing total prize given a maximum tour length) and additionally adds penalties.
Revised: This problem is related to the OP but inverts the goal (minimizing tour length given a minimum total prize to collect instead of maximizing total prize given a maximum tour length) and additionally adds penalties.

Original: Again, we add a special depot node with index 0 and coordinates x 0 and if the model selects the depot, the route is ﬁnished.
Revised: Again, we add a special depot node with index 0 and coordinates x 0 and if the model selects the depot, the route is ﬁnished.

Original: In the PCTSP, it can be beneﬁcial to visit additional nodes, even if the minimum total prize constraint is already satisﬁed, in order to avoid penalties.
Revised: In the PCTSP, it can be beneﬁcial to visit additional nodes, even if the minimum total prize constraint is already satisﬁed, in order to avoid penalties.

Original: E. I NSTANCE GENERATION
Revised: E. I NSTANCE GENERATION

Original: The depot location as well as n node locations are sampled uniformly at random in the unit square.
Revised: The depot location as well as n node locations are sampled uniformly at random in the unit square.

Original: Similar to the OP, we select the distribution for the prizes and penalties with the idea that for difﬁcult instances approximately half of the nodes should be visited.
Revised: Similar to the OP, we select the distribution for the prizes and penalties with the idea that for difﬁcult instances approximately half of the nodes should be visited.

Original: Additionally, neither the prize nor the penalty should dominate the node selection process.
Revised: Additionally, neither the prize nor the penalty should dominate the node selection process.

Original: Prizes We consider uniformly distributed prizes.
Revised: Prizes We consider uniformly distributed prizes.

Original: If we sample prizes ρ i ∼ Uniform (0 , 1) , then E ( ρ i ) = 12 , and the expected total prize of any subset of n 2 nodes (i.e. half of the nodes) would be n 4 .
Revised: If we sample prizes ρ i ∼ Uniform (0 , 1) , then E ( ρ i ) = 12 , and the expected total prize of any subset of n 2 nodes (i.e. half of the nodes) would be n 4 .

Original: Therefore, if S is the set of nodes that is visited, we require that (cid:80) i ∈ S
Revised: Therefore, if S is the set of nodes that is visited, we require that (cid:80) i ∈ S

Original: ρ i ≥ n 4 , or equivalently (cid:80) i ∈ S
Revised: ρ i ≥ n 4 , or equivalently (cid:80) i ∈ S

Original: ˆ ρ i ≥ 1 where ˆ ρ i
Revised: ˆ ρ i ≥ 1 where ˆ ρ i

Original: =
Revised: =

Original: ρ i · 4 n is the normalized prize.
Revised: ρ i · 4 n is the normalized prize.

Original: Note that it can be the case that (cid:80) ni
Revised: Note that it can be the case that (cid:80) ni

Original: =

Revised: =

Original: ˆ ρ
Revised: ˆ ρ

Original: i < 1 , in which case the prize constraint may be violated but it is only allowed to return to the depot after all nodes have been visited.
Revised: i < 1 , in which case the prize constraint may be violated but it is only allowed to return to the depot after all nodes have been visited.

Original: Penalties If penalties are too small, then node selection is determined almost entirely by the minimum total prize constraint.
Revised: Penalties If penalties are too small, then node selection is determined almost entirely by the minimum total prize constraint.

Original: If penalties are too large, we will always visit all nodes, making the minimum total prize constraint obsolete.
Revised: If penalties are too large, we will always visit all nodes, making the minimum total prize constraint obsolete.

Original: We argue that in order for the penalties to be meaningful, they should contribute a term in the objective approximately equal to the total length of the tour.
Revised: We argue that in order for the penalties to be meaningful, they should contribute a term in the objective approximately equal to the total length of the tour.

Original: If L n is the expected TSP tour length with n nodes, we try to achieve this by sampling β i ∼ Uniform (0 , 2 · L n n ) such that E ( β i ) =
Revised: If L n is the expected TSP tour length with n nodes, we try to achieve this by sampling β i ∼ Uniform (0 , 2 · L n n ) such that E ( β i ) =

Original: L n n and the expected total penalty for a subset of nnodes is L n 2 .
Revised: L n n and the expected total penalty for a subset of nnodes is L n 2 .

Original: Following the numbers we use for the OP, we roughly deﬁne L n 2 ≈ K n = 2 , 3 , 4 for n = 20 , 50 , 100 11 .
Revised: Following the numbers we use for the OP, we roughly deﬁne L n 2 ≈ K n = 2 , 3 , 4 for n = 20 , 50 , 100 13 .

Original: This means that we should sample β i ∼ Uniform (0 , 4 · K n n ) , but empirically we ﬁnd that ˆ β i ∼ Uniform (0 , 3 · K n n ) works better, which means that the prizes and penalties are balanced as the minimum total prize constraint is sometimes binding and sometimes not.
Revised: This means that we should sample β i ∼ Uniform (0 , 4 · K n n ) , but empirically we ﬁnd that ˆ β i ∼ Uniform (0 , 3 · K n n ) works better, which means that the prizes and penalties are balanced as the minimum total prize constraint is sometimes binding and sometimes not.

Original: E. A TTENTION M ODEL FOR THE PCTSP
Revised: E. A TTENTION M ODEL FOR THE PCTSP

Original: Encoder Again, we use separate parameters for the depot node embedding.
Revised: Encoder Again, we use separate parameters for the depot node embedding.

Original: Additionally, we provide the node prize ˆ ρ i and the penalty
Revised: Additionally, we provide the node prize ˆ ρ i and the penalty

Original: ˆ β i as input features:
Revised: ˆ β i as input features:

Original: [Equation]
Revised: [Equation]

Original: Minimum prize constraint In order to satisfy the minimum total prize constraint, we keep track of the remaining total prize P t to collect at time t .
Revised: Minimum prize constraint In order to satisfy the minimum total prize constraint, we keep track of the remaining total prize P t to collect at time t .

Original: At t = 1 , P 1 = 1 (as we normalized prizes).
Revised: At t = 1 , P 1 = 1 (as we normalized prizes).

Original: Then for t > 0 , P is updated as
Revised: Then for t > 0 , P is updated as

Original: [Equation]
Revised: [Equation]

Original: If the constraint is satisﬁed after visiting π t is visited at time t , then P t +1 will be 0.
Revised: If the constraint is satisﬁed after visiting π t is visited at time t , then P t +1 will be 0.

Original: Decoder context The context for the decoder for the PCTSP at time t is the current/last location π t − 1 and the remaining prize to collect P t .
Revised: Decoder context The context for the decoder for the PCTSP at time t is the current/last location π t − 1 and the remaining prize to collect P t .

Original: Again, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot.
Revised: Again, we do not need placeholders if t = 1 as the route starts at the depot and we do not need to provide information about the ﬁrst node as the route should end at the depot.

Original: The information about the prizes collected is implicitly provided to the model in the form of P t and we do not need to provide any information about the penalties as this is irrelevant for the remaining decisions:
Revised: The information about the prizes collected is implicitly provided to the model in the form of P t and we do not need to provide any information about the penalties as this is irrelevant for the remaining decisions:

Original: [Equation]
Revised: [Equation]

Original: Masking In the PCTSP, the depot node cannot be visited if the remaining prize to collect P t is larger than 0 and not yet all nodes have been visited (so t ≤ n ):
Revised: Masking In the PCTSP, the depot node cannot be visited if the remaining prize to collect P t is larger than 0 and not yet all nodes have been visited (so t ≤ n ):

Original: [Equation]
Revised: [Equation]

Original: Regular nodes are masked (i.e. cannot be visited) only if they are already visited:
Revised: Regular nodes are masked (i.e. cannot be visited) only if they are already visited:

Original: [Equation]
Revised: [Equation]

Original: E. D ETAILS OF BASELINES
Revised: E. D ETAILS OF BASELINES

Original: For the C++ Iterated Local Search (ILS) algorithm 12 , we perform 1 run as this takes alreadyminutes per instance (single thread) on average.
Revised: For the C++ Iterated Local Search (ILS) algorithm 14 , we perform 1 run as this takes alreadyminutes per instance (single thread) on average.

Original: For the Python ILS algorithm 13 we perform 10 runs as this algorithm is fast.
Revised: For the Python ILS algorithm 15 we perform 10 runs as this algorithm is fast.

Original: This improved results somewhat for n = 20 .
Revised: This improved results somewhat for n = 20 .

Original: OR Tools For the Google OR Tools implementation, we modify the formulation for the CVRP 14 :
Revised: OR Tools For the Google OR Tools implementation, we modify the formulation for the CVRP 16 :

Original: • We replace the Manhattan distance by the Euclidian distance.
Revised: • We replace the Manhattan distance by the Euclidian distance.

Original: • We set the number of vehicles to 1.
Revised: • We set the number of vehicles to 1.

Original: • For each individual node
Revised: • For each individual node

Original: i , we add a Disjunction constraint with { i } as the set of nodes, and a penalty equal to the penalty ˆ β i .
Revised: i , we add a Disjunction constraint with { i } as the set of nodes, and a penalty equal to the penalty ˆ β i .

Original: This allows OR tools to skip node i at a cost ˆ β i .
Revised: This allows OR tools to skip node i at a cost ˆ β i .

Original: • We replace the capacity constraint by a minimum total prize constraint by adding the prizes as a Dimension .
Revised: • We replace the capacity constraint by a minimum total prize constraint by adding the prizes as a Dimension .

Original: We multiply all ﬂoat inputs by 10 7 and round to integers.
Revised: We multiply all ﬂoat inputs by 10 7 and round to integers.

Original: Note that we keep the total length objective from the CVRP and add the Disjunction constraint with penalties to obtain the right objective.
Revised: Note that we keep the total length objective from the CVRP and add the Disjunction constraint with penalties to obtain the right objective.

Original: examples/python/cvrp.py

Original: F S TOCHASTIC PCTSP (SPCTSP)
Revised: F S TOCHASTIC PCTSP (SPCTSP)

Original: For the SPCTSP, we assume that the real prize collected ˆ ρ ∗ i at each node only becomes known when visiting the node, and ˆ ρ i =
Revised: For the SPCTSP, we assume that the real prize collected ˆ ρ ∗ i at each node only becomes known when visiting the node, and ˆ ρ i =

Original: E [ˆ ρ ∗ i ] is the expected prize.
Revised: E [ˆ ρ ∗ i ] is the expected prize.

Original: We assume the real prizes follow a uniform distribution, so
Revised: We assume the real prizes follow a uniform distribution, so

Original: ˆ ρ ∗ i ∼ Uniform (0 , 2ˆ ρ i ) .
Revised: ˆ ρ ∗ i ∼ Uniform (0 , 2ˆ ρ i ) .

Original: F. A TTENTION M ODEL FOR THE SPCTSP
Revised: F. A TTENTION M ODEL FOR THE SPCTSP

Original: In order to apply the Attention Model to the Stochastic PCTSP, the only change we need is that we use the real ˆ ρ ∗ i to update the remaining prize to collect P t in equation 35:
Revised: In order to apply the Attention Model to the Stochastic PCTSP, the only change we need is that we use the real ˆ ρ ∗ i to update the remaining prize to collect P t in equation 31:

Original: [Equation]
Revised: [Equation]

Original: We could theoretically use the model trained for PCTSP without retraining, but we choose to retrain.
Revised: We could theoretically use the model trained for PCTSP without retraining, but we choose to retrain.

Original: This way the model could (for example) learn that if it needs to gather a remaining (normalized) prize of 0 .
Revised: This way the model could (for example) learn that if it needs to gather a remaining (normalized) prize of 0 .

Original: 1 , it might prefer to visit a node with expected prize 0 .
Revised: 1 , it might prefer to visit a node with expected prize 0 .

Original: 2 over a node with expected prize 0 .
Revised: 2 over a node with expected prize 0 .

Original: 1 as the ﬁrst real prize will be ≥ 0 .
Revised: 1 as the ﬁrst real prize will be ≥ 0 .

Original: 1 with probability 75% (uniform prizes) whereas the latter only with 50% and thus has a probability of 50% to not satisfy the constraint.
Revised: 1 with probability 75% (uniform prizes) whereas the latter only with 50% and thus has a probability of 50% to not satisfy the constraint.

Original: F. R OLLOUT BASELINE IN THE STOCHASTIC SETTING
Revised: F. R OLLOUT BASELINE IN THE STOCHASTIC SETTING

Original: Instead of sampling the real prizes online, we already sample them when creating the dataset but keep them hidden to the algorithm.
Revised: Instead of sampling the real prizes online, we already sample them when creating the dataset but keep them hidden to the algorithm.

Original: This way, when using a rollout baseline, both the greedy rollout baseline as well as the sample (rollout) from the model use the same real prizes, such that any difference between the two is not a result of stochasticity.
Revised: This way, when using a rollout baseline, both the greedy rollout baseline as well as the sample (rollout) from the model use the same real prizes, such that any difference between the two is not a result of stochasticity.

Original: This can be seen as a variant of using
Revised: This can be seen as a variant of using

Original: Common Random Numbers for variance reduction (Glasserman & Yao, 1992).
Revised: Common Random Numbers for variance reduction (Glasserman & Yao, 1992).

Original: F. D ETAILS OF BASELINES
Revised: F. D ETAILS OF BASELINES

Original: For the SPCTSP, it is not possible to formulate an exact model that constructs a tour ofﬂine (as any tour can be infeasible with nonzero probability) and an algorithm that computes the optimal decision online should take into account an inﬁnite number of scenarios.
Revised: For the SPCTSP, it is not possible to formulate an exact model that constructs a tour ofﬂine (as any tour can be infeasible with nonzero probability) and an algorithm that computes the optimal decision online should take into account an inﬁnite number of scenarios.

Original: As a baseline we implement a strategy that:
Revised: As a baseline we implement a strategy that:

Original: Plans a tour using the expected prizes ˆ ρ i 2.
Revised: Plans a tour using the expected prizes ˆ ρ i 2.

Original: Executes part of the tour (not returning to the depot), observing the real prizes ˆ ρ ∗ i 3.
Revised: Executes part of the tour (not returning to the depot), observing the real prizes ˆ ρ ∗ i 3.

Original: Computes the remaining total prize that needs to be collected 4.
Revised: Computes the remaining total prize that needs to be collected 4.

Original: Computes a new tour (again using expected prizes ˆ ρ i ), starting from the last node that was visited, through nodes that have not yet been visited and ending at the depot 5.
Revised: Computes a new tour (again using expected prizes ˆ ρ i ), starting from the last node that was visited, through nodes that have not yet been visited and ending at the depot 5.

Original: Repeats the steps (2) - (4) above until the minimum total prize has been collected or all nodes have been visited 6.
Revised: Repeats the steps (2) - (4) above until the minimum total prize has been collected or all nodes have been visited 6.

Original: Returns to the depot
Revised: Returns to the depot

Original: Planning of the tours using deterministic prizes means we need to solve a (deterministic) PCTSP, for which we use the ILS C++ algorithm as this was the strongest algorithm for PCTSP (for large n ).
Revised: Planning of the tours using deterministic prizes means we need to solve a (deterministic) PCTSP, for which we use the ILS C++ algorithm as this was the strongest algorithm for PCTSP (for large n ).

Original: Note that in (4), we have a variant of the PCTSP where we do not have a single depot, but rather separate start and end points, whereas the ILS C++ implementation assumes starting and ending at a single depot.
Revised: Note that in (4), we have a variant of the PCTSP where we do not have a single depot, but rather separate start and end points, whereas the ILS C++ implementation assumes starting and ending at a single depot.

Original: However, as the ILS C++ implementation uses a distance matrix, we can effectively plan with a start and end node by deﬁning the distance from the ‘depot’ to node j as the distance from the start node (the last visited node) to node j , whereas we leave the distance from node j to the depot/end node unchanged (so the distance matrix becomes asymmetrical).
Revised: However, as the ILS C++ implementation uses a distance matrix, we can effectively plan with a start and end node by deﬁning the distance from the ‘depot’ to node j as the distance from the start node (the last visited node) to node j , whereas we leave the distance from node j to the depot/end node unchanged (so the distance matrix becomes asymmetrical).

Original: Additionally, we remove all nodes (rows/columns in the distance matrix) that have already been visited from the problem.
Revised: Additionally, we remove all nodes (rows/columns in the distance matrix) that have already been visited from the problem.

Original: We consider three variants that differ in the number of nodes that are visited before replanning the tour, for a tradeoff between adaptivity and run time:
Revised: We consider three variants that differ in the number of nodes that are visited before replanning the tour, for a tradeoff between adaptivity and run time:

Original: All nodes in the planned tour are visited (except the ﬁnal return to the depot).
Revised: All nodes in the planned tour are visited (except the ﬁnal return to the depot).

Original: We only need to replan and visit additional nodes if the constraint is not satisﬁed, otherwise we return to the depot.
Revised: We only need to replan and visit additional nodes if the constraint is not satisﬁed, otherwise we return to the depot.

Original: Half of the nodes in the planned tour are visited, where we visit k nodes if there are 2 k + nodes (excluding the return to the depot), so we round down if an odd number of visits is planned.
Revised: Half of the nodes in the planned tour are visited, where we visit k nodes if there are 2 k + nodes (excluding the return to the depot), so we round down if an odd number of visits is planned.

Original: This way, we will have O (log n ) replanning iterations, while being more adaptive when we are closer to satisfying the total prize constraint.
Revised: This way, we will have O (log n ) replanning iterations, while being more adaptive when we are closer to satisfying the total prize constraint.

Original: This is a trade-off of adaptivity vs computation time.
Revised: This is a trade-off of adaptivity vs computation time.

Original: Only the ﬁrst node is visited, after which we directly replan.
Revised: Only the ﬁrst node is visited, after which we directly replan.

Original: This allows the algorithm to take new online information about the real prizes into account directly, but is very expensive to compute as it requires O ( n ) iterations.
Revised: This allows the algorithm to take new online information about the real prizes into account directly, but is very expensive to compute as it requires O ( n ) iterations.