Original: L EARNING TO S OLVE C OMBINATORIAL G RAPH P ARTITIONING P ROBLEMS VIA E FFICIENT E XPLORATION
Revised: L EARNING TO S OLVE C OMBINATORIAL G RAPH P ARTITIONING P ROBLEMS VIA E FFICIENT E XPLORATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: From logistics to the natural sciences, combinatorial optimisation on graphs underpins numerous real-world applications.
Revised: From logistics to the natural sciences, combinatorial optimisation on graphs underpins numerous real-world applications.

Original: Reinforcement learning (RL) has shown particular promise in this setting as it can adapt to speciﬁc problem structures and does not require pre-solved instances for these, often NP-hard, problems.
Revised: Reinforcement learning (RL) has shown particular promise in this setting as it can adapt to speciﬁc problem structures and does not require pre-solved instances for these, often NP-hard, problems.

Original: However, state-of-the-art (SOTA) approaches typically suffer from severe scalability issues, primarily due to their reliance on expensive graph neural networks (GNNs) at each decision step.
Revised: However, state-of-the-art (SOTA) approaches typically suffer from severe scalability issues, primarily due to their reliance on expensive graph neural networks (GNNs) at each decision step.

Original: We introduce ECORD; a novel RL algorithm that alleviates this expense by restricting the GNN to a single pre-processing step, before entering a fast-acting exploratory phase directed by a recurrent unit.
Revised: We introduce ECORD; a novel RL algorithm that alleviates this expense by restricting the GNN to a single pre-processing step, before entering a fast-acting exploratory phase directed by a recurrent unit.

Original: Experimentally, we demonstrate that ECORD achieves a new SOTA for RL algorithms on the Maximum Cut problem, whilst also providing orders of magnitude improvement in speed and scalability.
Revised: Experimentally, we demonstrate that ECORD achieves a new SOTA for RL algorithms on the Maximum Cut problem, whilst also providing orders of magnitude improvement in speed and scalability.

Original: Compared to the nearest competitor, ECORD reduces the optimality gap by up to 73 % on 500 vertex graphs with a decreased wall-clock time.
Revised: Compared to the nearest competitor, ECORD reduces the optimality gap by up to 73 % on 500 vertex graphs with a decreased wall-clock time.

Original: Moreover, ECORD retains strong performance when generalising to larger graphs with up to 10 000 vertices.
Revised: Moreover, ECORD retains strong performance when generalising to larger graphs with up to 10 000 vertices.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Combinatorial optimisation (CO) problems seek to ﬁnd the ordering, labelling, or subset of discrete elements that maximises some objective function.
Revised: Combinatorial optimisation (CO) problems seek to ﬁnd the ordering, labelling, or subset of discrete elements that maximises some objective function.

Original: Despite this seemingly abstract mathematical formulation, CO arises at the heart of many practical applications, from logistics (Yanling et al., 2010) to protein folding (Perdomo-Ortiz et al., 2012) and fundamental science (Barahona, 1982).
Revised: Despite this seemingly abstract mathematical formulation, CO arises at the heart of many practical applications, from logistics (Yanling et al., 2010) to protein folding (Perdomo-Ortiz et al., 2012) and fundamental science (Barahona, 1982).

Original: However, with these tasks often being NP-hard, solving CO problems becomes increasingly challenging for all but the simplest of systems.
Revised: However, with these tasks often being NP-hard, solving CO problems becomes increasingly challenging for all but the simplest of systems.

Original: This combination of conceptual simplicity, computational complexity, and practical importance has made CO a canonical challenge, and motivated signiﬁcant efforts into developing approximate and heuristic algorithms for these tasks.
Revised: This combination of conceptual simplicity, computational complexity, and practical importance has made CO a canonical challenge, and motivated signiﬁcant efforts into developing approximate and heuristic algorithms for these tasks.

Original: Whilst approximate methods can offer guarantees on the solution quality, in practice they frequently lack sufﬁciently strong bounds and have limited scalability (Williamson & Shmoys, 2011).
Revised: Whilst approximate methods can offer guarantees on the solution quality, in practice they frequently lack sufﬁciently strong bounds and have limited scalability (Williamson & Shmoys, 2011).

Original: By contrast, well-designed heuristics offer no such guarantees but can prove highly efﬁcient (Halim & Ismail, 2019).
Revised: By contrast, well-designed heuristics offer no such guarantees but can prove highly efﬁcient (Halim & Ismail, 2019).

Original: As a result, recent years have seen a surge in the application of automated learning methods that parameterise CO heuristics with deep neural networks (Bengio et al., 2021).
Revised: As a result, recent years have seen a surge in the application of automated learning methods that parameterise CO heuristics with deep neural networks (Bengio et al., 2021).

Original: In particular, reinforcement learning (RL) has become a popular paradigm, as it can facilitate the discovery of novel heuristics without the need for labelled data.
Revised: In particular, reinforcement learning (RL) has become a popular paradigm, as it can facilitate the discovery of novel heuristics without the need for labelled data.

Original: Moreover, many CO problems are naturally formulated as Markov decision processes (MDPs) on graphs, where vertices correspond to discrete variables and edges denote their interaction or dependence.
Revised: Moreover, many CO problems are naturally formulated as Markov decision processes (MDPs) on graphs, where vertices correspond to discrete variables and edges denote their interaction or dependence.

Original: Accordingly, graph neural networks (GNNs) have become the de-facto function approximator of choice as they reﬂect the underlying structure of the problem whilst seamlessly handling variable problem sizes and irregular topological structures.
Revised: Accordingly, graph neural networks (GNNs) have become the de-facto function approximator of choice as they reﬂect the underlying structure of the problem whilst seamlessly handling variable problem sizes and irregular topological structures.

Original: However, despite the demonstrated success of RL-GNN approaches, scalability remains an outstanding challenge.
Revised: However, despite the demonstrated success of RL-GNN approaches, scalability remains an outstanding challenge.

Original: Running a GNN for every decision results in impractical computational overheads for anything beyond small- to medium-sized problems.
Revised: Running a GNN for every decision results in impractical computational overheads for anything beyond small- to medium-sized problems.

Original: This is exacerbated by the fact that directly predicting the solution to an NP-hard problem is typically unrealistic.
Revised: This is exacerbated by the fact that directly predicting the solution to an NP-hard problem is typically unrealistic.

Original: As such, leading approaches often utilise stochastic exploration or structured search to generate multiple candidate solutions (Chen &
Revised: As such, leading approaches often utilise stochastic exploration or structured search to generate multiple candidate solutions (Chen &

Original: Tian, 2019; Joshi et al., 2019; Gupta et al., 2020; Barrett et al., 2020; Bresson & Laurent, 2021) – which ultimately requires longer solving times and an increased computational burden.
Revised: Tian, 2019; Joshi et al., 2019; Gupta et al., 2020; Barrett et al., 2020; Bresson & Laurent, 2021) – which ultimately requires longer solving times and an increased computational burden.

Original: The notion of exploratory combinatorial optimisation – reframing the task from predicting a single solution to exploring the solution space using reversible single-vertex actions (Barrett et al., 2020) – also hints at a mitigation to the scalability issue it highlights.
Revised: The notion of exploratory combinatorial optimisation – reframing the task from predicting a single solution to exploring the solution space using reversible single-vertex actions (Barrett et al., 2020) – also hints at a mitigation to the scalability issue it highlights.

Original: Intuitively, when any decisions can be reversed, the quality of any single decision is less critical, so long as overall improvements are made over the course of many such actions.
Revised: Intuitively, when any decisions can be reversed, the quality of any single decision is less critical, so long as overall improvements are made over the course of many such actions.

Original: In this work, we leverage this notion to introduce a new algorithm, ECORD
Revised: In this work, we leverage this notion to introduce a new algorithm, ECORD

Original: (Exploratory Combinatorial Optimisation with Rapid Decoding), that combines a single GNN preprocessing step with fast action-decoding that replaces further geometric inference with simple per-vertex observations and a learnt representation of the ongoing optimisation trajectory.
Revised: (Exploratory Combinatorial Optimisation with Rapid Decoding), that combines a single GNN preprocessing step with fast action-decoding that replaces further geometric inference with simple per-vertex observations and a learnt representation of the ongoing optimisation trajectory.

Original: The result is a theoretical and demonstrated speed-up over expensive GNN action-decoding with the action-selection time of ECORD being independent of the graph topology and, in practice, near constant regardless of graph size.
Revised: The result is a theoretical and demonstrated speed-up over expensive GNN action-decoding with the action-selection time of ECORD being independent of the graph topology and, in practice, near constant regardless of graph size.

Original: Experimentally, we consider the Maximum Cut (Max-Cut) problem, a canonical CO problem chosen because of its generality (11 of the 21 NP-complete problems presented by Karp (1972) can be reduced to Max-Cut , including graph coloring, clique cover, knapsack and Steiner Tree) and the fact that it presents a challenging problem for scalable CO as the optimal solution requires every vertex to be correctly labelled (rather than simply a subset).
Revised: Experimentally, we consider the Maximum Cut (Max-Cut) problem, a canonical CO problem chosen because of its generality (11 of the 21 NP-complete problems presented by Karp (1972) can be reduced to Max-Cut, including graph coloring, clique cover, knapsack and Steiner Tree) and the fact that it presents a challenging problem for scalable CO as the optimal solution requires every vertex to be correctly labelled (rather than simply a subset).

Original: This combination of wide-ranging applicability and intractability has motivated signiﬁcant commercial and research efforts into Max-Cut solvers, from bespoke hardware based on classical (Goto et al., 2019) and quantum annealing (Yamamoto et al., 2017;
Revised: This combination of wide-ranging applicability and intractability has motivated signiﬁcant commercial and research efforts into Max-Cut solvers, from bespoke hardware based on classical (Goto et al., 2019) and quantum annealing (Yamamoto et al., 2017;

Original: Djidjev et al., 2018) to hand-crafted (Goemans & Williamson, 1995; Benlic & Hao, 2013) or learnt (Barrett et al., 2020) heuristic algorithms.
Revised: Djidjev et al., 2018) to hand-crafted (Goemans & Williamson, 1995; Benlic & Hao, 2013) or learnt (Barrett et al., 2020) heuristic algorithms.

Original: ECORD is found to equal or surpass the performance of expensive SOTA RL-GNN methods on graphs with up to 500 vertices (where all methods are computationally feasible), even when compared on number of actions instead of wall-clock time.
Revised: ECORD is found to equal or surpass the performance of expensive SOTA RL-GNN methods on graphs with up to 500 vertices (where all methods are computationally feasible), even when compared on number of actions instead of wall-clock time.

Original: Moreover, the low computational overhead of
Revised: Moreover, the low computational overhead of

Original: ECORD is seen to provide orders of magnitude improvements in speed and scalability, with strong performance, and a nearly 300 × increase in throughput when compared to conventional RL-GNN methods, demonstrated on graphs with up to 10 000 vertices.
Revised: ECORD is seen to provide orders of magnitude improvements in speed and scalability, with strong performance, and a nearly 300 × increase in throughput when compared to conventional RL-GNN methods, demonstrated on graphs with up to 10 000 vertices.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The application of neural networks to graph-based CO problems dates back to Hopﬁeld & Tank (1985) who considered the Travelling Salesman Problem (TSP).
Revised: The application of neural networks to graph-based CO problems dates back to Hopﬁeld & Tank (1985) who considered the Travelling Salesman Problem (TSP).

Original: However RL techniques were not applied until a decade later in the work of Zhang & Dietterich (1995) on the NP-hard job-shop problem.
Revised: However RL techniques were not applied until a decade later in the work of Zhang & Dietterich (1995) on the NP-hard job-shop problem.

Original: More recently, the advancement of deep learning and RL has triggered a resurgence in the ML community’s interest in developing CO solvers, with multiple reviews providing detailed taxonomies (Bengio et al., 2021; Mazyavkina et al., 2020; Vesselinova et al., 2020).
Revised: More recently, the advancement of deep learning and RL has triggered a resurgence in the ML community’s interest in developing CO solvers, with multiple reviews providing detailed taxonomies (Bengio et al., 2021; Mazyavkina et al., 2020; Vesselinova et al., 2020).

Original: Learning to solve non-Euclidean CO .
Revised: Learning to solve non-Euclidean CO .

Original: This resurgence began by considering Euclidean approaches that did not reﬂect the underlying graph structure of the problems.
Revised: This resurgence began by considering Euclidean approaches that did not reﬂect the underlying graph structure of the problems.

Original: In this context, Bello et al.
Revised: In this context, Bello et al.

Original: (2016) used RL to train pointer networks (PNs), which treat the discrete variables of the CO problem as a sequence of inputs to a recurrent unit (Vinyals et al., 2015) to solve TSP.
Revised: (2016) used RL to train pointer networks (PNs), which treat the discrete variables of the CO problem as a sequence of inputs to a recurrent unit (Vinyals et al., 2015) to solve TSP.

Original: By avoiding the need for labelled data sets, they were able to scale beyond the 40 vertex limit of Vinyals et al.
Revised: By avoiding the need for labelled data sets, they were able to scale beyond the 40 vertex limit of Vinyals et al.

Original: , inferring on 20 to 100 vertex graphs.
Revised: , inferring on 20 to 100 vertex graphs.

Original: Gu & Yang (2020) further scaled this approach to instances of up to 300 , using a hybrid supervised-reinforcement learning framework that combined PNs and
Revised: Gu & Yang (2020) further scaled this approach to instances of up to 300 , using a hybrid supervised-reinforcement learning framework that combined PNs and

Original: A3C (Mnih et al., 2016).
Revised: A3C (Mnih et al., 2016).

Original: However, although PNs can handle graphs of different sizes (with the help of manual input/output engineering, such as zero padding), these Euclidean approaches fail to capture the topological structures and intricate relationships contained within graphs, and typically require a large number of training instances in order to generalise.
Revised: However, although PNs can handle graphs of different sizes (with the help of manual input/output engineering, such as zero padding), these Euclidean approaches fail to capture the topological structures and intricate relationships contained within graphs, and typically require a large number of training instances in order to generalise.

Original: This issue was addressed by Dai et al.
Revised: This issue was addressed by Dai et al.

Original: (2017), who trained a Structure-to-Vector (S2V) GNN with
Revised: (2017), who trained a Structure-to-Vector (S2V) GNN with

Original: DQN to solve TSP and Max-Cut.
Revised: DQN to solve TSP and Max-Cut.

Original: The resulting algorithm, S2V-DQN, generalised to graphs with different size and structure to the training set and achieved excellent performance across a range of problems without the need for manual algorithmic design, demonstrating the value in exploiting underlying graph structure.
Revised: The resulting algorithm, S2V-DQN, generalised to graphs with different size and structure to the training set and achieved excellent performance across a range of problems without the need for manual algorithmic design, demonstrating the value in exploiting underlying graph structure.

Original: Advances in optimality .
Revised: Advances in optimality .

Original: Various works since Dai et al.
Revised: Various works since Dai et al.

Original: (2017) have sought to harness GNN embeddings to improve solution quality.
Revised: (2017) have sought to harness GNN embeddings to improve solution quality.

Original: Abe et al.
Revised: Abe et al.

Original: combined a GNN with a Monte Carlo tree search approach to learn a high-quality constructive heuristic.
Revised: combined a GNN with a Monte Carlo tree search approach to learn a high-quality constructive heuristic.

Original: Ultimately, they demonstrated a greater ability to generalise to more graph types than S2V-DQN on Max-Cut, but their method could only scale to 100 vertex Erd˝os-Rényi and Barabasi-Albert graphs.
Revised: Ultimately, they demonstrated a greater ability to generalise to more graph types than S2V-DQN on Max-Cut, but their method could only scale to 100 vertex Erd˝os-Rényi and Barabasi-Albert graphs.

Original: Li et al.
Revised: Li et al.

Original: combined a graph convolution network (GCN) with guided tree search to synthesise a diverse set of solutions and thereby more fully explore the space of possible solutions.
Revised: combined a graph convolution network (GCN) with guided tree search to synthesise a diverse set of solutions and thereby more fully explore the space of possible solutions.

Original: However, they used supervised learning and so required labelled data which limits scalability, and they did not consider Max-Cut.
Revised: However, they used supervised learning and so required labelled data which limits scalability, and they did not consider Max-Cut.

Original: Barrett et al.
Revised: Barrett et al.

Original: (2020) proposed ECO-DQN, the SOTA RL algorithm for Max-Cut, which reframed the role of the
Revised: (2020) proposed ECO-DQN, the SOTA RL algorithm for Max-Cut, which reframed the role of the

Original: RL agent to looking to improve on any given solution, rather than directly predict the optimal solution.
Revised: RL agent to looking to improve on any given solution, rather than directly predict the optimal solution.

Original: The key insight is that exploration at test time is beneﬁcial, since individual sub-optimal decisions (which are to be expected for NP-hard problems) need not matter so long as the ﬁnal solution is of high quality.
Revised: The key insight is that exploration at test time is beneﬁcial, since individual sub-optimal decisions (which are to be expected for NP-hard problems) need not matter so long as the ﬁnal solution is of high quality.

Original: However, ECO-DQN utilises an expensive GNN at each decision step and extends the overall number of decisions taken to be theoretically limitless, thereby restricting its scalability to be even worse than that of S2V-DQN.
Revised: However, ECO-DQN utilises an expensive GNN at each decision step and extends the overall number of decisions taken to be theoretically limitless, thereby restricting its scalability to be even worse than that of S2V-DQN.

Original: ECORD remedies this by using an initial GNN embedding followed by a recurrent unit to balance the richness provided by graph networks with fast-action selection.
Revised: ECORD remedies this by using an initial GNN embedding followed by a recurrent unit to balance the richness provided by graph networks with fast-action selection.

Original: Advances in scalability .
Revised: Advances in scalability .

Original: Manchanda et al.
Revised: Manchanda et al.

Original: (2020) furthered the work of Dai et al.
Revised: (2020) furthered the work of Dai et al.

Original: (2017) by ﬁrst training an embedding GCN in a supervised manner, and then training a Q-network with RL to predict per-vertex action values.
Revised: (2017) by ﬁrst training an embedding GCN in a supervised manner, and then training a Q-network with RL to predict per-vertex action values.

Original: By using the GCN embeddings to prune nodes unlikely to be in the solution set, their method provided signiﬁcantly more scalability than S2V-DQN on the the
Revised: By using the GCN embeddings to prune nodes unlikely to be in the solution set, their method provided signiﬁcantly more scalability than S2V-DQN on the the

Original: Maximum Vertex Cover problem.
Revised: Maximum Vertex Cover problem.

Original: However, it was not applicable to problems whose nodes cannot be pruned, which precludes it from solving some of the most fundamental CO problems such as
Revised: However, it was not applicable to problems whose nodes cannot be pruned, which precludes it from solving some of the most fundamental CO problems such as

Original: Max-Cut.
Revised: Max-Cut.

Original: Drori et al.
Revised: Drori et al.

Original: (2020) took a different approach, proposing a general RL-GNN framework that uses a graph attention network to create a dense embedding of the input graph followed by a recurrent attention mechanism for action selection.
Revised: (2020) took a different approach, proposing a general RL-GNN framework that uses a graph attention network to create a dense embedding of the input graph followed by a recurrent attention mechanism for action selection.

Original: They achieved impressive scalability, reaching instance sizes of 1000 vertices on the minimum spanning tree problem and TSP.
Revised: They achieved impressive scalability, reaching instance sizes of 1000 vertices on the minimum spanning tree problem and TSP.

Original: However, unlike
Revised: However, unlike

Original: ECORD, the Drori et al.
Revised: ECORD, the Drori et al.

Original: (2020) framework restricts the decoding stage to condition only on the previously selected action, and considers only node ordering problems in a non-exploratory setting.
Revised: (2020) framework restricts the decoding stage to condition only on the previously selected action, and considers only node ordering problems in a non-exploratory setting.

Original:  M ETHODS 3. B ACKGROUND
Revised:  M ETHODS 3. B ACKGROUND

Original: Max-Cut problem .
Revised: Max-Cut problem .

Original: The Max-Cut problem seeks a binary labelling of all nodes in a graph, such that the number (or cumulative weight) of edges connecting nodes of opposite labels is maximised.
Revised: The Max-Cut problem seeks a binary labelling of all nodes in a graph, such that the number (or cumulative weight) of edges connecting nodes of opposite labels is maximised.

Original: Concretely, for a graph, G ( V, E ) , with vertices V and edges E , we wish to ﬁnd a subset of vertices,
Revised: Concretely, for a graph, G ( V, E ) , with vertices V and edges E , we wish to ﬁnd a subset of vertices,

Original: S ⊂ V that maximises the ‘cut-value’, C ( S, G ) =
Revised: S ⊂ V that maximises the ‘cut-value’, C ( S, G ) =

Original: (cid:80) i ∈ S,j ∈ V \ S w ( e ij ) , where w ( e ij ) is the weight of an edge e ij ∈ E .
Revised: (cid:80) i ∈ S,j ∈ V \ S w ( e ij ) , where w ( e ij ) is the weight of an edge e ij ∈ E .

Original: Q-learning .
Revised: Q-learning .

Original: We formulate the optimisation as a Markov Decision Process (MDP) deﬁned by a tuple {S , A , T , R , γ } , where S and A are the state and action spaces, respectively, T : S×A×S → [0 , 1] is the transition function, R :
Revised: We formulate the optimisation as a Markov Decision Process (MDP) deﬁned by a tuple {S , A , T , R , γ } , where S and A are the state and action spaces, respectively, T : S×A×S → [0 , 1] is the transition function, R :

Original: S
Revised: S

Original: → R the reward function and γ ∈ [0 , 1] the discount factor.
Revised: → R the reward function and γ ∈ [0 , 1] the discount factor.

Original: Q-learning methods seek to directly learn the Q-value function mapping state-action pairs, ( s ∈
Revised: Q-learning methods seek to directly learn the Q-value function mapping state-action pairs, ( s ∈

Original: S , a ∈ A ) , to the expected discounted sum of immediate and future rewards when following a policy π : S → A , Q π ( s, a ) =
Revised: S , a ∈ A ) , to the expected discounted sum of immediate and future rewards when following a policy π : S → A , Q π ( s, a ) =

Original: E π [ (cid:80) ∞ t (cid:48) =
Revised: E π [ (cid:80) ∞ t (cid:48) =

Original: t +1 γ t (cid:48) − 1 r ( s ( t (cid:48) ) )
Revised: t +1 γ t (cid:48) − 1 r ( s ( t (cid:48) ) )

Original: | s ( t ) =
Revised: | s ( t ) =

Original: s, a ( t ) = a ] .
Revised: s, a ( t ) = a ] .

Original: Conventional DQN (Mnih et al., 2015) optimises Q θ by minimising the mean-squared-error between the network predictions and a bootstrapped estimate of the Q-value.
Revised: Conventional DQN (Mnih et al., 2015) optimises Q θ by minimising the mean-squared-error between the network predictions and a bootstrapped estimate of the Q-value.

Original: By deﬁnition, an optimal policy maximises the true
Revised: By deﬁnition, an optimal policy maximises the true

Original: Q-value of every selected action, therefore, after training, an approximation of an optimal policy is obtained by acting greedily with respect to the learnt value function, π θ ( s ) =
Revised: Q-value of every selected action, therefore, after training, an approximation of an optimal policy is obtained by acting greedily with respect to the learnt value function, π θ ( s ) =

Original: arg max a (cid:48)
Revised: arg max a (cid:48)

Original: Q θ ( s, a (cid:48) ) .
Revised: Q θ ( s, a (cid:48) ) .

Original: Munchausen DQN .
Revised: Munchausen DQN .

Original: Munchausen DQN (M-DQN) (Vieillard et al., 2020) makes two fundamental adjustments to conventional DQN.
Revised: Munchausen DQN (M-DQN) (Vieillard et al., 2020) makes two fundamental adjustments to conventional DQN.

Original: Firstly, the Q-values are considered to deﬁne a stochastic policy with action probabilities given by π θ ( ·| s ) =
Revised: Firstly, the Q-values are considered to deﬁne a stochastic policy with action probabilities given by π θ ( ·| s ) =

Original: softmax( Q θ ( s, · ) τ ) , where τ is a temperature parameter.
Revised: softmax( Q θ ( s, · ) τ ) , where τ is a temperature parameter.

Original: Secondly, M-DQN adds the log-probability of the selected action to the reward at each step.
Revised: Secondly, M-DQN adds the log-probability of the selected action to the reward at each step.

Original: All together, the regression target for the Q-function is modiﬁed to
Revised: All together, the regression target for the Q-function is modiﬁed to

Original: [Equation]
Revised: [Equation]

Original: (1) where α scales the additional log-policy contribution to the reward.
Revised: (1) where α scales the additional log-policy contribution to the reward.

Original: Note that as α → 0 and τ →we recover the standard DQN regression target.
Revised: Note that as α → 0 and τ →we recover the standard DQN regression target.

Original: 3. ECORD
Revised: 3. ECORD

Original: MDP formulation .
Revised: MDP formulation .

Original: The state, s ( t ) ≡ ( G ( V, E ) , S ) ∈ S , at a given time-step corresponds to the graph and a binary labelling of all vertices.
Revised: The state, s ( t ) ≡ ( G ( V, E ) , S ) ∈ S , at a given time-step corresponds to the graph and a binary labelling of all vertices.

Original: An action selects a single vertex and ‘ﬂips’ its label to the opposite state, adding or removing it from the solution subset.
Revised: An action selects a single vertex and ‘ﬂips’ its label to the opposite state, adding or removing it from the solution subset.

Original: Rewards correspond to the (normalised and clipped) increase in best cut-value, r ( s ( t ) )
Revised: Rewards correspond to the (normalised and clipped) increase in best cut-value, r ( s ( t ) )

Original: =
Revised: =

Original: max(( C ( s ( t ) )
Revised: max(( C ( s ( t ) )

Original: − C ( s ∗ ))) /
Revised: − C ( s ∗ ))) /

Original: | V | , 0) , where C ( s ∗ ) is the the best solution found in the episode thus far.
Revised: | V | , 0) , where C ( s ∗ ) is the the best solution found in the episode thus far.

Original: This choice of reward structure motivates the agent to maximise the cut-value at any step within the episode, without discouraging it from searching through unseen solutions.
Revised: This choice of reward structure motivates the agent to maximise the cut-value at any step within the episode, without discouraging it from searching through unseen solutions.

Original: Naturally, the transition function is known and fully deterministic for our mathematically deﬁned optimisation problem.
Revised: Naturally, the transition function is known and fully deterministic for our mathematically deﬁned optimisation problem.

Original: Concretely, we represent the state, s ( t ) , with per-vertex observations, o ( t ) i ∈
Revised: Concretely, we represent the state, s ( t ) , with per-vertex observations, o ( t ) i ∈

Original: R 3 for i ∈ V , and a global observation of the current state, o ( t )G ∈ R 2 .
Revised: R 3 for i ∈ V , and a global observation of the current state, o ( t )G ∈ R 2 .

Original: For each vertex, we provide: (i) the current label, (ii) the immediate change in cut-value if the vertex is ﬂipped (which we refer to as a ‘peek’) and (iii) the number of steps since the node was last ﬂipped.
Revised: For each vertex, we provide: (i) the current label, (ii) the immediate change in cut-value if the vertex is ﬂipped (which we refer to as a ‘peek’) and (iii) the number of steps since the node was last ﬂipped.

Original: Our global observations are the (normalised) difference in cut-value between the current and best observed state, and the maximum peek of any vertex in the current state.
Revised: Our global observations are the (normalised) difference in cut-value between the current and best observed state, and the maximum peek of any vertex in the current state.

Original: Importantly, these features are readily available without introducing any signiﬁcant computational burden.
Revised: Importantly, these features are readily available without introducing any signiﬁcant computational burden.

Original: Indeed, even the peeks – one-step look-aheads for each action – can be calculated for all vertices once at the beginning of each episode, after which they can be efﬁciently updated quickly at each step for only the ﬂipped vertex and it’s immediate neighbours (details can be found appendix
Revised: Indeed, even the peeks – one-step look-aheads for each action – can be calculated for all vertices once at the beginning of each episode, after which they can be efﬁciently updated quickly at each step for only the ﬂipped vertex and it’s immediate neighbours (details can be found appendix

Original: A.3).
Revised: A.3).

Original: Besides this being a cheap operation, this approach is also the most efﬁcient way to track the cut-value over the course of multiple vertex ﬂips, and thus is a necessary calculation to evaluate each environmental state regardless of whether the peeks form part of the observation.
Revised: Besides this being a cheap operation, this approach is also the most efﬁcient way to track the cut-value over the course of multiple vertex ﬂips, and thus is a necessary calculation to evaluate each environmental state regardless of whether the peeks form part of the observation.

Original: Architecture .
Revised: Architecture .

Original: ECORD splits an optimisation into two stages (see ﬁgure 1 for an overview of the entire architecture).
Revised: ECORD splits an optimisation into two stages (see ﬁgure 1 for an overview of the entire architecture).

Original: (i)
Revised: (i)

Original: A GNN prepares per-vertex embeddings conditioned on only the geometric structure (i.e. weighted adjacency matrix) of the graph itself.
Revised: A GNN prepares per-vertex embeddings conditioned on only the geometric structure (i.e. weighted adjacency matrix) of the graph itself.

Original: (ii)
Revised: (ii)

Original: Starting from a random labelling of the vertices, ECORD sequentially ﬂips vertices between subsets, with each action conditioned on the static GNN embeddings, the previous trajectory steps (through the hidden state of an RNN) and simple observations of the current state (described above).
Revised: Starting from a random labelling of the vertices, ECORD sequentially ﬂips vertices between subsets, with each action conditioned on the static GNN embeddings, the previous trajectory steps (through the hidden state of an RNN) and simple observations of the current state (described above).

Original: Full details of the architecture can be found in appendix A.1, with this section providing a higher level summary.
Revised: Full details of the architecture can be found in appendix A.1, with this section providing a higher level summary.

Original: To generate the per-vertex embeddings, we use a gated graph convolution network (Li et al., 2015).
Revised: To generate the per-vertex embeddings, we use a gated graph convolution network (Li et al., 2015).

Original: The ﬁnal embeddings after L rounds of message-passing are linearly projected to per-vertex embeddings, x i =
Revised: The ﬁnal embeddings after L rounds of message-passing are linearly projected to per-vertex embeddings, x i =

Original: W p
Revised: W p

Original: x
Revised: x

Original: ( L ) i where i denotes the vertex, that remain ﬁxed for the remainder of the episode.
Revised: ( L ) i where i denotes the vertex, that remain ﬁxed for the remainder of the episode.

Original: To select an action at time t , we ﬁrst combine the GNN outputs with the observation to obtain pervertex embeddings of the form v ( t )
Revised: To select an action at time t , we ﬁrst combine the GNN outputs with the observation to obtain pervertex embeddings of the form v ( t )

Original: i =
Revised: i =

Original: [ x i , W o o ( t ) i ] , where square brackets denote concatenation.
Revised: [ x i , W o o ( t ) i ] , where square brackets denote concatenation.

Original: This local information is then used in conjunction with the RNN hidden state, h ( t ) , to predict the
Revised: This local information is then used in conjunction with the RNN hidden state, h ( t ) , to predict the

Original: Q-value for ﬂipping vertex i .
Revised: Q-value for ﬂipping vertex i .

Original: The Q-network itself uses a duelling architecture (Wang et al., 2016), where separate MLPs estimate the value of the state, V ( · ) , and advantage for each action, A ( · ) ,
Revised: The Q-network itself uses a duelling architecture (Wang et al., 2016), where separate MLPs estimate the value of the state, V ( · ) , and advantage for each action, A ( · ) ,

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: h ( t ) is shared across each vertex when calculating the advantage function as our reward function depends on the current best score within the trajectory’s history, a global feature contained in the state-level embedding.
Revised: h ( t ) is shared across each vertex when calculating the advantage function as our reward function depends on the current best score within the trajectory’s history, a global feature contained in the state-level embedding.

Original: However, before concatenating them together, we project the high dimensional (1024 in our case) hidden state to match the lower dimensional (32) per-vertex embeddings.
Revised: However, before concatenating them together, we project the high dimensional (1024 in our case) hidden state to match the lower dimensional (32) per-vertex embeddings.

Original: Finally, we update the RNN using the embedding of the selected action, v ( t ) ∗ , and our global observation after taking it, o ( t +1) G , h ( t +1) =
Revised: Finally, we update the RNN using the embedding of the selected action, v ( t ) ∗ , and our global observation after taking it, o ( t +1) G , h ( t +1) =

Original: GRU ( h ( t ) , m ( t +1) ) , where m ( t +1) =
Revised: GRU ( h ( t ) , m ( t +1) ) , where m ( t +1) =

Original: MLP m ([ v ( t ) ∗ , o ( t +1)G ]) .
Revised: MLP m ([ v ( t ) ∗ , o ( t +1)G ]) .

Original: Training .
Revised: Training .

Original: ECORD is trained using M-DQN.
Revised: ECORD is trained using M-DQN.

Original: As the Q-values are conditioned upon the internal state of an RNN, see eq.
Revised: As the Q-values are conditioned upon the internal state of an RNN, see eq.

Original: (2), we train using truncated backpropagation through time (BPTT).
Revised: (2), we train using truncated backpropagation through time (BPTT).

Original: Concretely, when calculating the Q-values at time t during training, we reset the environment and the internal state of the RNN to their state at time t − k BPTT and replay the trajectory to time t .
Revised: Concretely, when calculating the Q-values at time t during training, we reset the environment and the internal state of the RNN to their state at time t − k BPTT and replay the trajectory to time t .

Original: In doing so, we only have to backpropagate through the previous k BPTT time steps when minimising the loss.
Revised: In doing so, we only have to backpropagate through the previous k BPTT time steps when minimising the loss.

Original: The intuition behind M-DQN’s modiﬁcations are (i) the entropy of the policy is jointly optimised with the returns, in the spirit of maximum entropy RL (Haarnoja et al., 2018), and (ii) the agent is rewarded for being more conﬁdent about the correct action.
Revised: The intuition behind M-DQN’s modiﬁcations are (i) the entropy of the policy is jointly optimised with the returns, in the spirit of maximum entropy RL (Haarnoja et al., 2018), and (ii) the agent is rewarded for being more conﬁdent about the correct action.

Original: The second point is based on the assumption that an optimal policy is deterministic, since it will always take the action with maximum
Revised: The second point is based on the assumption that an optimal policy is deterministic, since it will always take the action with maximum

Original: Q-value.
Revised: Q-value.

Original: One could observe that as ECORD’s action space is large (equal to the number of vertices in the graph, | V | ) the structured exploration of M-DQN’s stochastic policy may allow for more meaningful trajectories than the standard epsilon-greedy approaches used in DQN.
Revised: One could observe that as ECORD’s action space is large (equal to the number of vertices in the graph, | V | ) the structured exploration of M-DQN’s stochastic policy may allow for more meaningful trajectories than the standard epsilon-greedy approaches used in DQN.

Original: Moreover, the underlying postulate of M-DQN, that despite the stochastic exploration policy, the true optimal policy is deterministic, aligns exactly with our problem setting.
Revised: Moreover, the underlying postulate of M-DQN, that despite the stochastic exploration policy, the true optimal policy is deterministic, aligns exactly with our problem setting.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Baselines .
Revised: Baselines .

Original: We compare ECORD to the previous two SOTA RL-GNN algorithms for Max-Cut, S2VDQN (Khalil et al., 2017) and ECO-DQN (Barrett et al., 2020).
Revised: We compare ECORD to the previous two SOTA RL-GNN algorithms for Max-Cut, S2VDQN (Khalil et al., 2017) and ECO-DQN (Barrett et al., 2020).

Original: Our implementation of ECORD contains several speed improvements in comparison to the publicly available implementation of
Revised: Our implementation of ECORD contains several speed improvements in comparison to the publicly available implementation of

Original: ECO-DQN (e.g. parallelised optimisation trajectories, compiled calculations, sparse matrix operations).
Revised: ECO-DQN (e.g. parallelised optimisation trajectories, compiled calculations, sparse matrix operations).

Original: To ensure the fairest possible comparison, when directly comparing performance independent of speed (section 4.1), we use the public implementation, however when comparing scalability (section 4.2 and 4.3) we use a re-implementation of ECO-DQN within the same codebase as ECORD.
Revised: To ensure the fairest possible comparison, when directly comparing performance independent of speed (section 4.1), we use the public implementation, however when comparing scalability (section 4.2 and 4.3) we use a re-implementation of ECO-DQN within the same codebase as ECORD.

Original: Additionally, a simple heuristic, Max-Cut Approx (MCA), that greedily selects actions that maximise the immediate increase in cut value is also considered.
Revised: Additionally, a simple heuristic, Max-Cut Approx (MCA), that greedily selects actions that maximise the immediate increase in cut value is also considered.

Original: Besides from providing surprisingly strong performance, the choice of MCA baselines is motivated by the observation that the one-step look-ahead ‘peek’ features make learning a greedy policy straightforward.
Revised: Besides from providing surprisingly strong performance, the choice of MCA baselines is motivated by the observation that the one-step look-ahead ‘peek’ features make learning a greedy policy straightforward.

Original: Moreover, whilst MCA terminates once a locally optimal solution is found (i.e. one from which no single action can further increase the cut value), a network learning to only approximate a greedy policy may fortuitously escape these solutions and ultimately obtain better results.
Revised: Moreover, whilst MCA terminates once a locally optimal solution is found (i.e. one from which no single action can further increase the cut value), a network learning to only approximate a greedy policy may fortuitously escape these solutions and ultimately obtain better results.

Original: To address this concern, we introduce a simple extension of (and signiﬁcant improvement over) MCA called MCA-soft, where actions are selected by a soft-greedy policy with the temperature tuned to maximise performance on the target dataset (see appendix A.4 for details).
Revised: To address this concern, we introduce a simple extension of (and signiﬁcant improvement over) MCA called MCA-soft, where actions are selected by a soft-greedy policy with the temperature tuned to maximise performance on the target dataset (see appendix A.4 for details).

Original: Datasets .
Revised: Datasets .

Original: We consider graph datasets for which the optimal (or best known) solutions are publicly available.
Revised: We consider graph datasets for which the optimal (or best known) solutions are publicly available.

Original: The dataset published by Barrett et al.
Revised: The dataset published by Barrett et al.

Original: (2020) consists of Erd ˝
Revised: (2020) consists of Erd ˝

Original: os-Rényi
Revised: os-Rényi

Original: (Erd ˝ os & Rényi,
Revised: (Erd ˝ os & Rényi,

Original: and Barabasi-Albert (Albert & Barabási, 2002) graphs (ER and BA, respectively) with edge weights w ( e ij ) ∈ { 0 , ± 1 } and up to 500 vertices.
Revised: and Barabasi-Albert (Albert & Barabási, 2002) graphs (ER and BA, respectively) with edge weights w ( e ij ) ∈ { 0 , ± 1 } and up to 500 vertices.

Original: Each graph type and size is represented byrandom instances, with 50 used for model selection and results reported on the remaining 100 at test time.
Revised: Each graph type and size is represented byrandom instances, with 50 used for model selection and results reported on the remaining 100 at test time.

Original: We refer to these distributions as ER40/BA40 to ER500/BA500.
Revised: We refer to these distributions as ER40/BA40 to ER500/BA500.

Original: To test on larger graphs we use the GSet (Benlic & Hao, 2013), a well-studied dataset containing multiple distribution, from which we focus on random (ER) graphs with binary edge weights and 800 to 10 000 vertices.
Revised: To test on larger graphs we use the GSet (Benlic & Hao, 2013), a well-studied dataset containing multiple distribution, from which we focus on random (ER) graphs with binary edge weights and 800 to 10 000 vertices.

Original: To aide model selection and parameter tuning, we generate 10 additional graphs for each distribution in the GSet.
Revised: To aide model selection and parameter tuning, we generate 10 additional graphs for each distribution in the GSet.

Original: We refer to these validation sets as ER800 to ER10000.
Revised: We refer to these validation sets as ER800 to ER10000.

Original: Metrics .
Revised: Metrics .

Original: Our analysis considers the raw performance, wall-clock speed, and memory usage of algorithms.
Revised: Our analysis considers the raw performance, wall-clock speed, and memory usage of algorithms.

Original: Following prior work, we use the approximation ratio, given by AR ( s ∗ ) =
Revised: Following prior work, we use the approximation ratio, given by AR ( s ∗ ) =

Original: C ( s ∗ ) /C
Revised: C ( s ∗ ) /C

Original: opt where C opt is the best known cut value, as a metric of solution quality.
Revised: opt where C opt is the best known cut value, as a metric of solution quality.

Original: All experiments were performed on the same system with an Nvidia GeForce RTX 2080 Ti 11 GB GPU and 80 processors (Intel(R)
Revised: All experiments were performed on the same system with an Nvidia GeForce RTX 2080 Ti 11 GB GPU and 80 processors (Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6248 CPU @ 2 .
Revised: Gold 6248 CPU @ 2 .

Original: 50 GHz ).
Revised: 50 GHz ).

Original: Reproducability .
Revised: Reproducability .

Original: All datasets considered in this work are either available or linked in the supporting code at https://github.com/[MASKED-FOR-BLIND-REVIEW] , along with source code and scripts to reproduce the reported results.
Revised: All datasets considered in this work are either available or linked in the supporting code at https://github.com/[MASKED-FOR-BLIND-REVIEW] , along with source code and scripts to reproduce the reported results.

Original: 4. C OMPARISON TO SOTA RL METHODS
Revised: 4. C OMPARISON TO SOTA RL METHODS

Original: [Table]
Revised: [Table]

Original: Methods .
Revised: Methods .

Original: Our ﬁrst set of experiments compares ECORD to both heuristic node-ﬂipping, and SOTA
Revised: Our ﬁrst set of experiments compares ECORD to both heuristic node-ﬂipping, and SOTA

Original: RL, baselines.
Revised: RL, baselines.

Original: To facilitate a fair and direct comparison, we use exactly the models, datasets, and results published by Barrett et al.
Revised: To facilitate a fair and direct comparison, we use exactly the models, datasets, and results published by Barrett et al.

Original: (2020) for S2V-DQN and ECO-DQN.
Revised: (2020) for S2V-DQN and ECO-DQN.

Original: All RL algorithms are trained on 40 vertex ER graphs and evaluated on both ER and BA graphs with up to 500 vertices.
Revised: All RL algorithms are trained on 40 vertex ER graphs and evaluated on both ER and BA graphs with up to 500 vertices.

Original: S2VDQN is deterministic and incrementally constructs the solution set one vertex at a time, therefore each optimisation trajectory consists of | V | sequential actions.
Revised: S2VDQN is deterministic and incrementally constructs the solution set one vertex at a time, therefore each optimisation trajectory consists of | V | sequential actions.

Original: MCA-soft, ECO-DQN and ECORD allow any vertex to be ‘ﬂipped’ at each step, and therefore can in principle run indeﬁnitely on a target graph.
Revised: MCA-soft, ECO-DQN and ECORD allow any vertex to be ‘ﬂipped’ at each step, and therefore can in principle run indeﬁnitely on a target graph.

Original: The baselines presented are chosen as they consider the same sequential node-ﬂipping paradigm as ECORD, however their exits multiple algorithms for solving Max-Cut that do not ﬁt this paradigm – notably simulated annealing (SA) (Tiunov et al., 2019; Leleu et al., 2019), semideﬁnite programming (SDP) (Goemans & Williamson, 1995) and mixed integer programming (CPLEX,
Revised: The baselines presented are chosen as they consider the same sequential node-ﬂipping paradigm as ECORD, however their exits multiple algorithms for solving Max-Cut that do not ﬁt this paradigm – notably simulated annealing (SA) (Tiunov et al., 2019; Leleu et al., 2019), semideﬁnite programming (SDP) (Goemans & Williamson, 1995) and mixed integer programming (CPLEX,

Original: ILOG, 2007).
Revised: ILOG, 2007).

Original: We provide additional results using SOTA or commercial algorithms spanning these paradigms in appendix A.5, where ECORD is found to outperform SDP and MIP and either beat or be competitive with SOTA SA methodologies.
Revised: We provide additional results using SOTA or commercial algorithms spanning these paradigms in appendix A.5, where ECORD is found to outperform SDP and MIP and either beat or be competitive with SOTA SA methodologies.

Original: In practice, we use ECORD and the MCA heuristics with ECO-DQN’s default settings; 50 optimisation trajectories per graph, each starting from a random node labelling, acting greedily with respect to the learnt Q-values, and terminating after 2 | V | sequential actions.
Revised: In practice, we use ECORD and the MCA heuristics with ECO-DQN’s default settings; 50 optimisation trajectories per graph, each starting from a random node labelling, acting greedily with respect to the learnt Q-values, and terminating after 2 | V | sequential actions.

Original: We note that this disadvantages
Revised: We note that this disadvantages

Original: ECORD as (i) despite learning a stochastic policy, ECORD acts deterministically at test time, and (ii) ECORD’s signiﬁcant speed and memory advantages over ECO-DQN are not accounted for.
Revised: ECORD as (i) despite learning a stochastic policy, ECORD acts deterministically at test time, and (ii) ECORD’s signiﬁcant speed and memory advantages over ECO-DQN are not accounted for.

Original: Results .
Revised: Results .

Original: Despite the disadvantages described above, ECORD either outperforms or essentially matches ECO-DQN on all tests (see table 1), whilst signiﬁcantly improving over all other baselines
Revised: Despite the disadvantages described above, ECORD either outperforms or essentially matches ECO-DQN on all tests (see table 1), whilst signiﬁcantly improving over all other baselines

Original: [Figure]
Revised: [Figure]

Original: and generalising to unseen graph topologies and sizes.
Revised: and generalising to unseen graph topologies and sizes.

Original: This seems surprising when we consider that, in contrast to ECO-DQN, ECORD does not directly condition per-node decisions on the state of other nodes.
Revised: This seems surprising when we consider that, in contrast to ECO-DQN, ECORD does not directly condition per-node decisions on the state of other nodes.

Original: However, ECORD does have access to the optimisation trajectory via the RNN hidden state suggesting that in exploratory settings, the temporal structure (in addition to the geometric structure of the graph) is highly informative.
Revised: However, ECORD does have access to the optimisation trajectory via the RNN hidden state suggesting that in exploratory settings, the temporal structure (in addition to the geometric structure of the graph) is highly informative.

Original: This experiment was repeated for a model trained instead on BA40 with the results (provided in appendix A.5) being qualitatively the same
Revised: This experiment was repeated for a model trained instead on BA40 with the results (provided in appendix A.5) being qualitatively the same

Original: 4. C OMPUTATIONAL COMPLEXITY AND PRACTICAL SCALING PERFORMANCE
Revised: 4. C OMPUTATIONAL COMPLEXITY AND PRACTICAL SCALING PERFORMANCE

Original: Whilst ECORD has already been shown to match or surpass SOTA RL baselines when ignoring computational cost, this is not a sufﬁcient metric of algorithmic utility.
Revised: Whilst ECORD has already been shown to match or surpass SOTA RL baselines when ignoring computational cost, this is not a sufﬁcient metric of algorithmic utility.

Original: In this section, we consider the theoretical complexity and practical scaling cost of ECORD.
Revised: In this section, we consider the theoretical complexity and practical scaling cost of ECORD.

Original: Theoretical complexity .
Revised: Theoretical complexity .

Original: The runtime complexity of any optimisation trajectory scales linearly with the number of actions.
Revised: The runtime complexity of any optimisation trajectory scales linearly with the number of actions.

Original: In CO problems where every vertex must be correctly labelled, such as MaxCut, episode length scales at best linearly with the number of nodes.
Revised: In CO problems where every vertex must be correctly labelled, such as MaxCut, episode length scales at best linearly with the number of nodes.

Original: A typical approach, which encapsulates both S2V-DQN and ECO-DQN, for applying RL to CO on graphs parameterises the policy or value function with a GNN.
Revised: A typical approach, which encapsulates both S2V-DQN and ECO-DQN, for applying RL to CO on graphs parameterises the policy or value function with a GNN.

Original: The precise complexity of a GNN depends on the chosen architecture and graph topology, however, typically the per-layer performance scales linearly with the number of edges in the graph, as this is the number of unique communication channels along which information must be passed.
Revised: The precise complexity of a GNN depends on the chosen architecture and graph topology, however, typically the per-layer performance scales linearly with the number of edges in the graph, as this is the number of unique communication channels along which information must be passed.

Original: In practice, this results in a (worst case and typical) polynomial scaling of O ( | V | 2 ) per-action and O ( | V | 3 ) per optimisation, which makes even modest sized graphs with the order of hundreds of nodes very computationally expensive in practice.
Revised: In practice, this results in a (worst case and typical) polynomial scaling of O ( | V | 2 ) per-action and O ( | V | 3 ) per optimisation, which makes even modest sized graphs with the order of hundreds of nodes very computationally expensive in practice.

Original: In contrast, ECORD only runs the GNN once, regardless of the graph size or episode length, and then selects actions without any additional message passing between nodes.
Revised: In contrast, ECORD only runs the GNN once, regardless of the graph size or episode length, and then selects actions without any additional message passing between nodes.

Original: As a result, per-action computational cost scales linearly as O ( | V | ) , regardless of the graph topology, and the entire optimisation scales as O ( | V | 2 ) .
Revised: As a result, per-action computational cost scales linearly as O ( | V | ) , regardless of the graph topology, and the entire optimisation scales as O ( | V | 2 ) .

Original: Moreover, action selection in ECORD also has a far smaller memory footprint than using a GNN over the entire graph, and each vertex can be processed in parallel up to the limits of hardware.
Revised: Moreover, action selection in ECORD also has a far smaller memory footprint than using a GNN over the entire graph, and each vertex can be processed in parallel up to the limits of hardware.

Original: Therefore, in practice, we typically obtain a constant, O (1) , scaling of the per-action computational cost and, as the single graph network pass is typically negligible compared to the long exploratory phase, complete the entire optimisation in O ( | V | ) .
Revised: Therefore, in practice, we typically obtain a constant, O (1) , scaling of the per-action computational cost and, as the single graph network pass is typically negligible compared to the long exploratory phase, complete the entire optimisation in O ( | V | ) .

Original: Practical performance .
Revised: Practical performance .

Original: Figure 2 demonstrates the practical performance realised from these theoretical improvements.
Revised: Figure 2 demonstrates the practical performance realised from these theoretical improvements.

Original: In 2a, the action time of ECORD and ECO-DQN are compared on graphs with up to 500 k vertices.
Revised: In 2a, the action time of ECORD and ECO-DQN are compared on graphs with up to 500 k vertices.

Original: Whilst ECORD is always faster, both take near constant time for small graphs where batched inference across all nodes is still efﬁcient.
Revised: Whilst ECORD is always faster, both take near constant time for small graphs where batched inference across all nodes is still efﬁcient.

Original: However, even when ECO-DQN begins to increase in cost ( | V | ≈ 500 ) and eventually ﬁlls the entire GPU memory ( | V | ≈ 10 k ), ECORD retains a ﬁxed low-cost inference which only appreciably begins to increase for large graphs ( | V | > 100 k ).
Revised: However, even when ECO-DQN begins to increase in cost ( | V | ≈ 500 ) and eventually ﬁlls the entire GPU memory ( | V | ≈ 10 k ), ECORD retains a ﬁxed low-cost inference which only appreciably begins to increase for large graphs ( | V | > 100 k ).

Original: Moreover, the reduced memory footprint of ECORD also allows for more (randomly initialised) optimisation trajectories to be run in parallel.
Revised: Moreover, the reduced memory footprint of ECORD also allows for more (randomly initialised) optimisation trajectories to be run in parallel.

Original: Figure 2b contrasts ECORD and ECO-DQN by running as many parallel trajectories as possible for a single graph with up to | V | =10 k , and plotting the effective step time (action selection plus environmental step) per trajectory.
Revised: Figure 2b contrasts ECORD and ECO-DQN by running as many parallel trajectories as possible for a single graph with up to | V | =10 k , and plotting the effective step time (action selection plus environmental step) per trajectory.

Original: The practical increase in throughput of ECORD compared to ECO-DQN increases with graph size from a modest 1 . 8 × with | V | =30 to 100 × for | V | =3 k and 298 .
Revised: The practical increase in throughput of ECORD compared to ECO-DQN increases with graph size from a modest 1 . 8 × with | V | =30 to 100 × for | V | =3 k and 298 .

Original: 9 × for | V | =100 k .
Revised: 9 × for | V | =100 k .

Original: 4. S CALING TO LARGE GRAPHS
Revised: 4. S CALING TO LARGE GRAPHS

Original: Methods .
Revised: Methods .

Original: ECORD is trained on graphs with binary edge weights, w ( e ij ) ∈ { 0 , 1 } and | V | = 500 .
Revised: ECORD is trained on graphs with binary edge weights, w ( e ij ) ∈ { 0 , 1 } and | V | = 500 .

Original: The optimal parameters are selected according to the performance of the greedy policy on the generated ER10000 set.
Revised: The optimal parameters are selected according to the performance of the greedy policy on the generated ER10000 set.

Original: For the computational reasons discussed previously, ECO-DQN was trained on ER200 graphs with binary edge weights, and used ER500 graphs for model selection.
Revised: For the computational reasons discussed previously, ECO-DQN was trained on ER200 graphs with binary edge weights, and used ER500 graphs for model selection.

Original: The temperature of MCA-soft was tuned independently on each graph, as detailed in appendix A.4.
Revised: The temperature of MCA-soft was tuned independently on each graph, as detailed in appendix A.4.

Original: Stochastic exploration .
Revised: Stochastic exploration .

Original: Despite ECORD using a stochastic soft-greedy policy during training, at test time we previously used a deterministic policy (which, ultimately, still provided near optimal performance on small and intermediate graphs).
Revised: Despite ECORD using a stochastic soft-greedy policy during training, at test time we previously used a deterministic policy (which, ultimately, still provided near optimal performance on small and intermediate graphs).

Original: To investigate the performance of a soft-greedy policy, we evaluated ECORD on the large-graph datasets ER5000, ER7000 and ER10000 – using 20 trajectories of 4 | V | steps for all graphs in the test sets – across a range of temperatures.
Revised: To investigate the performance of a soft-greedy policy, we evaluated ECORD on the large-graph datasets ER5000, ER7000 and ER10000 – using 20 trajectories of 4 | V | steps for all graphs in the test sets – across a range of temperatures.

Original: [Figure]
Revised: [Figure]

Original: The dashed lines are simply to guide the eye.
Revised: The dashed lines are simply to guide the eye.

Original: The results are summarised in ﬁgure 3.
Revised: The results are summarised in ﬁgure 3.

Original: The key points are that a non-zero temperature is optimal for larger graphs, and that the optimal temperature decreases with graph size.
Revised: The key points are that a non-zero temperature is optimal for larger graphs, and that the optimal temperature decreases with graph size.

Original: The interpretation is that even an exploratory agent such as ECORD can still eventually get stuck in closed trajectories when acting deterministically.
Revised: The interpretation is that even an exploratory agent such as ECORD can still eventually get stuck in closed trajectories when acting deterministically.

Original: Therefore, introducing some stochasticity allows ECORD to escape these regions of the solution space and ultimately continue to search for improved conﬁgurations.
Revised: Therefore, introducing some stochasticity allows ECORD to escape these regions of the solution space and ultimately continue to search for improved conﬁgurations.

Original: However, this must be balanced with the need for a sufﬁciently deterministic policy that a good solution is found with high probability.
Revised: However, this must be balanced with the need for a sufﬁciently deterministic policy that a good solution is found with high probability.

Original: Intuitively, longer sequences of actions are required to reach a good conﬁguration for larger graphs, which aligns with the observed dependence of the optimal temperature with graph size.
Revised: Intuitively, longer sequences of actions are required to reach a good conﬁguration for larger graphs, which aligns with the observed dependence of the optimal temperature with graph size.

Original: GSet results .
Revised: GSet results .

Original: We validate ECORD’s performance on large graphs by testing it on all GSet graphs with random binary edges, as summarised in table 2.
Revised: We validate ECORD’s performance on large graphs by testing it on all GSet graphs with random binary edges, as summarised in table 2.

Original: For each graph we run 20 parallel optimisation trajectories for a maximum of 180 s .
Revised: For each graph we run 20 parallel optimisation trajectories for a maximum of 180 s .

Original: For G5000, G7000 and G10000 we use tuned temperatures, τ , of 3 .
Revised: For G5000, G7000 and G10000 we use tuned temperatures, τ , of 3 .

Original: 5e − 4 , 2e − 4 and 1e − 4 respectively, and set τ =5e − 4 for all other graphs.
Revised: 5e − 4 , 2e − 4 and 1e − 4 respectively, and set τ =5e − 4 for all other graphs.

Original: ECORD shows consistently strong performance, matching or beating ECO-DQN and MCA-soft on every instance by a margin that increases with graph size.
Revised: ECORD shows consistently strong performance, matching or beating ECO-DQN and MCA-soft on every instance by a margin that increases with graph size.

Original: The speed of ECORD is emphasised by the fact that it obtains approximation ratios of above 0 .
Revised: The speed of ECORD is emphasised by the fact that it obtains approximation ratios of above 0 .

Original: 99 on all graphs with | V | ≤ 2 k in under 10 s .
Revised: 99 on all graphs with | V | ≤ 2 k in under 10 s .

Original: Whilst in principle it is possible to reach the optimal solution in | V | actions, on larger graphs the obtained cut values consistently increase with longer exploration (for reference, ECORD takes > 28 k actions in 180 s when | V | =10 k ).
Revised: Whilst in principle it is possible to reach the optimal solution in | V | actions, on larger graphs the obtained cut values consistently increase with longer exploration (for reference, ECORD takes > 28 k actions in 180 s when | V | =10 k ).

Original: This clearly demonstrates that ECORD learns to search through the solution space without getting stuck, rather than generating a single mode of solutions.
Revised: This clearly demonstrates that ECORD learns to search through the solution space without getting stuck, rather than generating a single mode of solutions.

Original: A natural ﬁnal question is whether the slight decrease in performance on the largest graphs is due to an insufﬁcient time budget, or the learnt policy being less suitable for these problems.
Revised: A natural ﬁnal question is whether the slight decrease in performance on the largest graphs is due to an insufﬁcient time budget, or the learnt policy being less suitable for these problems.

Original: To test this,
Revised: To test this,

Original: ECORD was allowed to continue optimising G70 for 1 h .
Revised: ECORD was allowed to continue optimising G70 for 1 h .

Original: Whilst the obtained solution improved to have an approximation ratio of 0 .
Revised: Whilst the obtained solution improved to have an approximation ratio of 0 .

Original: 978 (obtained in only 233 s ), the optimal solution was not found.
Revised: 978 (obtained in only 233 s ), the optimal solution was not found.

Original: Ultimately, whilst previous RL-GNN methods were limited by the scalability of GNNs, now we ﬁnd ourselves limited by the ability of the agent to reason about larger systems – opening the door to, and deﬁning a challenge for, future research.
Revised: Ultimately, whilst previous RL-GNN methods were limited by the scalability of GNNs, now we ﬁnd ourselves limited by the ability of the agent to reason about larger systems – opening the door to, and deﬁning a challenge for, future research.

Original: [Table]
Revised: [Table]

Original: Ablations .
Revised: Ablations .

Original: ECORD’s key components are: (i) the use of a single (GNN) encoding step to embed the problem structure, (ii) rapid decoding steps where per-vertex actions are conditioned on only local observations and an (RNN) leant embedding of the optimisation trajectory, and (iii) an exploratory CO setting where actions can be reversed.
Revised: ECORD’s key components are: (i) the use of a single (GNN) encoding step to embed the problem structure, (ii) rapid decoding steps where per-vertex actions are conditioned on only local observations and an (RNN) leant embedding of the optimisation trajectory, and (iii) an exploratory CO setting where actions can be reversed.

Original: Ablations (detailed in appendix A.5) of the
Revised: Ablations (detailed in appendix A.5) of the

Original: GNN and RNN show that both are necessary for strong performance.
Revised: GNN and RNN show that both are necessary for strong performance.

Original: ECORD’s contribution to the exploratory CO setting is evidencing that a suitably stochastic policy outperforms a deterministic one.
Revised: ECORD’s contribution to the exploratory CO setting is evidencing that a suitably stochastic policy outperforms a deterministic one.

Original: To emphasise that this is not simply because of algorithmic improvements in M-DQN compared to DQN, a deterministic-acting ECORD trained with DQN is shown to also not match the performance reported in table 2.
Revised: To emphasise that this is not simply because of algorithmic improvements in M-DQN compared to DQN, a deterministic-acting ECORD trained with DQN is shown to also not match the performance reported in table 2.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We present ECORD, a new SOTA RL algorithm for the Max-Cut problem in term of both performance and scalability.
Revised: We present ECORD, a new SOTA RL algorithm for the Max-Cut problem in term of both performance and scalability.

Original: ECORD’s demonstrated efﬁcacy on graphs with up 10 k vertices, and highly favourable computational complexity, suggests even larger problems could be tackled.
Revised: ECORD’s demonstrated efﬁcacy on graphs with up 10 k vertices, and highly favourable computational complexity, suggests even larger problems could be tackled.

Original: By replacing multiple expensive GNN operations with a single embedding stage and rapid action-selection directed by a recurrent unit, this work highlights the importance of, and a method to achieve, efﬁcient exploration when solving CO problems, all within the broader pursuit of scalable geometric learning.
Revised: By replacing multiple expensive GNN operations with a single embedding stage and rapid action-selection directed by a recurrent unit, this work highlights the importance of, and a method to achieve, efﬁcient exploration when solving CO problems, all within the broader pursuit of scalable geometric learning.

Original: Algorithmic improvements are a possible direction for further research.
Revised: Algorithmic improvements are a possible direction for further research.

Original: An adaptive (or learnt) temperature schedule could better trade-off stochastic exploration and deterministic solution improvement.
Revised: An adaptive (or learnt) temperature schedule could better trade-off stochastic exploration and deterministic solution improvement.

Original: ECORD also runs multiple optimisation trajectories in parallel, and so utilising information from other trajectories to either better inform future decisions or ensure sufﬁcient diversity between them, would be another approach for improving performance.
Revised: ECORD also runs multiple optimisation trajectories in parallel, and so utilising information from other trajectories to either better inform future decisions or ensure sufﬁcient diversity between them, would be another approach for improving performance.

Original: With regards to further scaling improvements, any algorithm that labels vertices sequentially will at best have O ( | V | ) complexity.
Revised: With regards to further scaling improvements, any algorithm that labels vertices sequentially will at best have O ( | V | ) complexity.

Original: An interesting prospect for future work would be to note that
Revised: An interesting prospect for future work would be to note that

Original: ECORD does not rely on observing the current state of the neighbourhood of a vertex to evaluate the action quality – therefore one could envisage ﬂipping multiple vertices in a single step, with a single centralised, or decentralised multi-agent, system.
Revised: ECORD does not rely on observing the current state of the neighbourhood of a vertex to evaluate the action quality – therefore one could envisage ﬂipping multiple vertices in a single step, with a single centralised, or decentralised multi-agent, system.

Original: An alternative direction would be to apply our algorithm to other complex problems and, in principle,
Revised: An alternative direction would be to apply our algorithm to other complex problems and, in principle,

Original: ECORD could be applied to any vertex-labelling CO problem deﬁned on a graph.
Revised: ECORD could be applied to any vertex-labelling CO problem deﬁned on a graph.

Original: However, its use of ‘peeks’ (one-step look aheads for each action), does not directly translate to problems where the quality of intermediate actions are not naturally evaluated (e.g. those where an arbitrary node labelling may be invalid such as maximum clique).
Revised: However, its use of ‘peeks’ (one-step look aheads for each action), does not directly translate to problems where the quality of intermediate actions are not naturally evaluated (e.g. those where an arbitrary node labelling may be invalid such as maximum clique).

Original: Considering only valid actions or utilising indirect metrics of solution quality are possible solutions, but this remains a topic for future research.
Revised: Considering only valid actions or utilising indirect metrics of solution quality are possible solutions, but this remains a topic for future research.

Original: A A PPENDIX A. A RCHITECTURE DETAILS
Revised: A A PPENDIX A. A RCHITECTURE DETAILS

Original: Here we provide further details on the network architecture described in section 3.2 and ﬁgure 1 of the main text.
Revised: Here we provide further details on the network architecture described in section 3.2 and ﬁgure 1 of the main text.

Original: For simplicity, we implicitly drop any bias terms in the below equations.
Revised: For simplicity, we implicitly drop any bias terms in the below equations.

Original: Graph Neural Network .
Revised: Graph Neural Network .

Original: The gated graph convolution network of Li et al. (2015) is modiﬁed to include layer normalisation, LN( · ) , where the 16-dimensional embedding of node i at layer l + 1 is given by,
Revised: The gated graph convolution network of Li et al. (2015) is modiﬁed to include layer normalisation, LN( · ) , where the 16-dimensional embedding of node i at layer l + 1 is given by,

Original: [Equation]
Revised: [Equation]

Original: where W g ∈ R 16 × 16 .
Revised: where W g ∈ R 16 × 16 .

Original: The ﬁnal embeddings after 4 rounds of message-passing are linearly projected to per-vertex embeddings, x i =
Revised: The ﬁnal embeddings after 4 rounds of message-passing are linearly projected to per-vertex embeddings, x i =

Original: W p
Revised: W p

Original: x (4)
Revised: x (4)

Original: i with W p ∈ R 16 × 16 , that remain ﬁxed for the remainder of the episode.
Revised: i with W p ∈ R 16 × 16 , that remain ﬁxed for the remainder of the episode.

Original: Value network .
Revised: Value network .

Original: Recall from the main text that Q-values at time t are predicted as
Revised: Recall from the main text that Q-values at time t are predicted as

Original: [Equation]
Revised: [Equation]

Original: where h ( t ) ∈ R 1024 is the hidden state of the RNN and v ( t )
Revised: where h ( t ) ∈ R 1024 is the hidden state of the RNN and v ( t )

Original: i =
Revised: i =

Original: [ x i , W o o ( t ) i ]
Revised: [ x i , W o o ( t ) i ]

Original: ∈ R 32 are the per-node embeddings with x i ∈ R 16 , W o ∈ R 16 × 3 and W h ∈ R 32 × 1014 .
Revised: ∈ R 32 are the per-node embeddings with x i ∈ R 16 , W o ∈ R 16 × 3 and W h ∈ R 32 × 1014 .

Original: The value head, MLP V ( · ) : R 1024 → R , is a 2-layer network that applies a tanh activation to the input and leaky ReLU to the intermediate activations.
Revised: The value head, MLP V ( · ) : R 1024 → R , is a 2-layer network that applies a tanh activation to the input and leaky ReLU to the intermediate activations.

Original: The advantage head, MLP A ( · ) : R 

Revised: The advantage head, MLP A ( · ) : R

Original: → R , is a 2-layer network that applies layer norm and leaky
Revised: → R , is a 2-layer network that applies layer norm and leaky

Original: ReLU to the intermediate activations.
Revised: ReLU to the intermediate activations.

Original: Recurrent network .
Revised: Recurrent network .

Original: The RNN hidden state is 1024-dimensional and initialised to zeros.
Revised: The RNN hidden state is 1024-dimensional and initialised to zeros.

Original: It is updated using the embedding of the selected action, v ( t ) ∗ , and our global observation from the next step, h ( t +1) =
Revised: It is updated using the embedding of the selected action, v ( t ) ∗ , and our global observation from the next step, h ( t +1) =

Original: GRU ( h ( t ) , m ( t +1) ) , where m ( t +1) =
Revised: GRU ( h ( t ) , m ( t +1) ) , where m ( t +1) =

Original: LeakyReLU
Revised: LeakyReLU

Original: ( W m ([ v ( t ) ∗ , o ( t +1)G ])) , (6) where W m ∈ R 64 × 34 .
Revised: ( W m ([ v ( t ) ∗ , o ( t +1)G ])) , (6) where W m ∈ R 64 × 34 .

Original: A. T RAINING DETAILS
Revised: A. T RAINING DETAILS

Original: The ECORD pseudocode is given in algorithm 1.
Revised: The ECORD pseudocode is given in algorithm 1.

Original: Note that to update the network at time t we make use of truncated backpropagation through time (BPTT)
Revised: Note that to update the network at time t we make use of truncated backpropagation through time (BPTT)

Original: (Werbos, 1990; Sutskever, 2013) using the previous t BPTT experiences.
Revised: (Werbos, 1990; Sutskever, 2013) using the previous t BPTT experiences.

Original: We perform network updates every f upd time steps, where we update the online network parameters θ using stochastic gradient descent (SGD) on the M-DQN loss and the target network parameters θ using a soft update (Lillicrap et al., 2015) with update parameter τ upd .
Revised: We perform network updates every f upd time steps, where we update the online network parameters θ using stochastic gradient descent (SGD) on the M-DQN loss and the target network parameters θ using a soft update (Lillicrap et al., 2015) with update parameter τ upd .

Original: The hyperparameters used are summarised in table 3.
Revised: The hyperparameters used are summarised in table 3.

Original: [Table]
Revised: [Table]

Original: A. E FFICIENT RE - CALCULATION OF CUT VALUE AND PEEKS
Revised: A. E FFICIENT RE - CALCULATION OF CUT VALUE AND PEEKS

Original: Given a graph, G ( V, E ) , with edge weights w ij ≡ w ( e ij ) for e ij ∈ E , and a node labelling represented as a binary vector, z ∈ { 0 , 1 } | V | , the cut value is given by,
Revised: Given a graph, G ( V, E ) , with edge weights w ij ≡ w ( e ij ) for e ij ∈ E , and a node labelling represented as a binary vector, z ∈ { 0 , 1 } | V | , the cut value is given by,

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to decompose this into the sum of ‘local’ cuts
Revised: It is straightforward to decompose this into the sum of ‘local’ cuts

Original: [Equation]
Revised: [Equation]

Original: where C i , the cut value of the sub-graph containing node only i and its neighbours, N ( i ) .
Revised: where C i , the cut value of the sub-graph containing node only i and its neighbours, N ( i ) .

Original: Similarly, we can deﬁne the total weight of un-cut edges connected to each vertex as
Revised: Similarly, we can deﬁne the total weight of un-cut edges connected to each vertex as

Original: [Equation]
Revised: [Equation]

Original: Initialise do
Revised: Initialise do

Original: Calculate
Revised: Calculate

Original: Initialise do a ( t ) ∼ (cid:40) softmax( Q θ ( s ( t )
Revised: Initialise do a ( t ) ∼ (cid:40) softmax( Q θ ( s ( t )

Original: · ) τ ) with prob.
Revised: · ) τ ) with prob.

Original: 1 − ε
Revised: 1 − ε

Original: S ( t +1) :
Revised: S ( t +1) :

Original: = (cid:40) S ( t ) ∪ { a ( t ) } , if a ( t ) / ∈ S ( t )
Revised: = (cid:40) S ( t ) ∪ { a ( t ) } , if a ( t ) / ∈ S ( t )

Original: S ( t ) \ { a ( t ) } , if a
Revised: S ( t ) \ { a ( t ) } , if a

Original: ) ∈ S t a ( t − a θ ← θτ upd + θ (1 − τ upd ) end end end endreturn θ
Revised: ) ∈ S t a ( t − a θ ← θτ upd + θ (1 − τ upd ) end end end endreturn θ

Original: The change in cut value (referred to as the ‘peek’ feature in the main text) if the label of vertex i is ﬂipped is then then given by ∆ C i
Revised: The change in cut value (referred to as the ‘peek’ feature in the main text) if the label of vertex i is ﬂipped is then then given by ∆ C i

Original: =
Revised: =

Original: C
Revised: C

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: C i , = (cid:88)
Revised: C i , = (cid:88)

Original: j ∈N ( i ) w ij (4 z i z
Revised: j ∈N ( i ) w ij (4 z i z

Original: j − 2( z i + z j ) + 1) , = (cid:88)
Revised: j − 2( z i + z j ) + 1) , = (cid:88)

Original: j ∈N ( i ) w ij (2 z i − 1)(

Revised: j ∈N ( i ) w ij (2 z i − 1)(

Original: z
Revised: z

Original: j − 1) .
Revised: j − 1) .

Original: Calculating these one-step look-aheads to the change in cut for each action clearly has the same complexity as calculating the cut-value itself (equation (7)).
Revised: Calculating these one-step look-aheads to the change in cut for each action clearly has the same complexity as calculating the cut-value itself (equation (7)).

Original: Moreover, they only have to be calculated once at the start of each episode, as when vertex i is ﬂipped from z i to z i , only the ‘peeks’ of vertrices i and j ∈ N ( i ) need to be updated.
Revised: Moreover, they only have to be calculated once at the start of each episode, as when vertex i is ﬂipped from z i to z i , only the ‘peeks’ of vertrices i and j ∈ N ( i ) need to be updated.

Original: These updates follow directly from the above and are given by
Revised: These updates follow directly from the above and are given by

Original: [Equation]
Revised: [Equation]

Original: A. MCA- SOFT
Revised: A. MCA- SOFT

Original: MCA-soft attempts to upper bound the performance simple policies that condition actions based solely on the provided ‘peeks’ for each action.
Revised: MCA-soft attempts to upper bound the performance simple policies that condition actions based solely on the provided ‘peeks’ for each action.

Original: Denoting the known change in cut value if vertex i was to be ﬂipped as ∆ C i (see eq.
Revised: Denoting the known change in cut value if vertex i was to be ﬂipped as ∆ C i (see eq.

Original: , MCA-soft follows a stochastic policy given by
Revised: , MCA-soft follows a stochastic policy given by

Original: [Equation]
Revised: [Equation]

Original: To maximise the performance of MCA-soft, the temperature, τ mca ∈ R , is independently tuned to maximise performance on every set of graphs considered.
Revised: To maximise the performance of MCA-soft, the temperature, τ mca ∈ R , is independently tuned to maximise performance on every set of graphs considered.

Original: In practice, this process is a grid search over τ mca ∈ { 0 , 0 .
Revised: In practice, this process is a grid search over τ mca ∈ { 0 , 0 .

Original: 3 , 1 , 3 } for results in table 1 and 4, and τ mca ∈ { 0 , 0 . 0001 , 0 .
Revised: 3 , 1 , 3 } for results in table 1 and 4, and τ mca ∈ { 0 , 0 . 0001 , 0 .

Original: 001 , 0 . 01 , 0 . 1 , 1 } for results in table 2.
Revised: 001 , 0 . 01 , 0 . 1 , 1 } for results in table 2.

Original: A. E XTENDED RESULTS Training on BA graphs .
Revised: A. E XTENDED RESULTS Training on BA graphs .

Original: For completeness, we repeat the experiments shown in the main manuscript but now training on 40 -vertex BA graphs and evaluating on both ER and BA graphs with up to 500 vertices, with the results shown in Table 4.
Revised: For completeness, we repeat the experiments shown in the main manuscript but now training on 40 -vertex BA graphs and evaluating on both ER and BA graphs with up to 500 vertices, with the results shown in Table 4.

Original: [Table]
Revised: [Table]

Original: Baseline comparison .
Revised: Baseline comparison .

Original: Here we conduct a thorough solver comparison by training and inferring on both ER and BA graphs of up to 500 vertices.
Revised: Here we conduct a thorough solver comparison by training and inferring on both ER and BA graphs of up to 500 vertices.

Original: We compare ECORD to seven baselines; six taken directly from Barrett et al.
Revised: We compare ECORD to seven baselines; six taken directly from Barrett et al.

Original: (2020) (ECO-DQN, S2V-DQN, MCA, CPLEX, SimCIM, and Leleu et al.
Revised: (2020) (ECO-DQN, S2V-DQN, MCA, CPLEX, SimCIM, and Leleu et al.

Original: (2019); refer to Barrett et al.
Revised: (2019); refer to Barrett et al.

Original: (2020) for implementation details), and one an extension of
Revised: (2020) for implementation details), and one an extension of

Original: MCA (MCA-soft, as described in this manuscript).
Revised: MCA (MCA-soft, as described in this manuscript).

Original: The results are summarised in Table 5, with the approximation ratios shown taken from averaging the solvers’ performances across 100 BA and ER graphs of up to 500 vertices.
Revised: The results are summarised in Table 5, with the approximation ratios shown taken from averaging the solvers’ performances across 100 BA and ER graphs of up to 500 vertices.

Original: [Table]
Revised: [Table]

Original: Semideﬁnite programming comparison .
Revised: Semideﬁnite programming comparison .

Original: In addition to heuristics, another important branch of
Revised: In addition to heuristics, another important branch of

Original: Max-Cut solver research is that of approximation algorithms.
Revised: Max-Cut solver research is that of approximation algorithms.

Original: Such algorithms can offer a theoretical guarantee on the approximation ratio while still solving a relaxed formulation of the original problem in polynomial time.
Revised: Such algorithms can offer a theoretical guarantee on the approximation ratio while still solving a relaxed formulation of the original problem in polynomial time.

Original: One such approximation method is the canonical semideﬁnite programming (SDP) approach of Goemans & Williamson (1995).
Revised: One such approximation method is the canonical semideﬁnite programming (SDP) approach of Goemans & Williamson (1995).

Original: Goemans & Williamson (1995) ﬁrst formulate
Revised: Goemans & Williamson (1995) ﬁrst formulate

Original: Max-Cut as an SDP by framing the objective as a linear function of a symmetric matrix subject to linear equality constraints (as in a linear programme) but with the additional constraint that the matrix must be positive semideﬁnite (whereby, for an n × n matrix A , ∀ x ∈ R n , x t Ax ≥ 0 ).
Revised: Max-Cut as an SDP by framing the objective as a linear function of a symmetric matrix subject to linear equality constraints (as in a linear programme) but with the additional constraint that the matrix must be positive semideﬁnite (whereby, for an n × n matrix A , ∀ x ∈ R n , x t Ax ≥ 0 ).

Original: This relaxed Max-Cut SDP formulation can be solved efﬁciently using algorithms such as
Revised: This relaxed Max-Cut SDP formulation can be solved efﬁciently using algorithms such as

Original: a generalised
Revised: a generalised

Original: Simplex method (Pólik & Terlaky, 2010).
Revised: Simplex method (Pólik & Terlaky, 2010).

Original: The insight of Goemans & Williamson (1995) was to then apply a geometric randomised rounding technique to convert the SDP solution into a feasible
Revised: The insight of Goemans & Williamson (1995) was to then apply a geometric randomised rounding technique to convert the SDP solution into a feasible

Original: Max-Cut solution.
Revised: Max-Cut solution.

Original: Crucially, the randomised rounding method gives a guarantee to be within at least 0 .
Revised: Crucially, the randomised rounding method gives a guarantee to be within at least 0 .

Original: 87856 times the optimal Max-Cut value.
Revised: 87856 times the optimal Max-Cut value.

Original: To the best of our knowledge, no open-access Goemans & Williamson (1995) solver exists which can handle Max-Cut problems with negatively weighted edges (Hong, 2008).
Revised: To the best of our knowledge, no open-access Goemans & Williamson (1995) solver exists which can handle Max-Cut problems with negatively weighted edges (Hong, 2008).

Original: However, G1-5 of the GSet graphs used in Table 2 of this manuscript have all-positive edge weights, therefore we ran the opensource cvx solver ( https://github.com/hermish/cvx-graph-algorithms ), which implements Goemans & Williamson (1995), on these 5 problems to obtain approximation ratios of 0 . 971 , 0 .
Revised: However, G1-5 of the GSet graphs used in Table 2 of this manuscript have all-positive edge weights, therefore we ran the opensource cvx solver ( https://github.com/hermish/cvx-graph-algorithms ), which implements Goemans & Williamson (1995), on these 5 problems to obtain approximation ratios of 0 . 971 , 0 .

Original: 970 , 0 . 975 , 0 . 970 , and 0 .
Revised: 970 , 0 . 975 , 0 . 970 , and 0 .

Original: 967 in 449 , 484 , 496 , 521 , and 599 s respectively.
Revised: 967 in 449 , 484 , 496 , 521 , and 599 s respectively.

Original: In addition to
Revised: In addition to

Original: ECORD outperforming Goemans & Williamson (1995) on these 5 GSet graphs (see Table 2) in both solving time and optimality, we note that ECORD also exceeds the 0 .
Revised: ECORD outperforming Goemans & Williamson (1995) on these 5 GSet graphs (see Table 2) in both solving time and optimality, we note that ECORD also exceeds the 0 .

Original: 87856 approximation ratio guarantee across all the GSet, ER, and BA graphs examined in our work (see Tables 1, 2, 4, and 5).
Revised: 87856 approximation ratio guarantee across all the GSet, ER, and BA graphs examined in our work (see Tables 1, 2, 4, and 5).

Original: Ablations .
Revised: Ablations .

Original: We provide ablations to further investigate the key components of ECORD, highlighted in the main text as
Revised: We provide ablations to further investigate the key components of ECORD, highlighted in the main text as

Original: a GNN to encode the spatial structure of the problem, 2.
Revised: a GNN to encode the spatial structure of the problem, 2.

Original: a rapid decoding conditioned on local per-node observations and an RNN’s internal state that represents the optimisation trajectory, 3.
Revised: a rapid decoding conditioned on local per-node observations and an RNN’s internal state that represents the optimisation trajectory, 3.

Original: an exploratory CO setting.
Revised: an exploratory CO setting.

Original: We ablate the GNN and RNN by using ﬁxed zero-vectors in-place of the static per-node embeddings, x i , and the RNN hidden state, h ( t ) , respectively.
Revised: We ablate the GNN and RNN by using ﬁxed zero-vectors in-place of the static per-node embeddings, x i , and the RNN hidden state, h ( t ) , respectively.

Original: As the advantage of exploratory CO has already been demonstrated in prior works (Barrett et al., 2020), we instead ablate the stochastic policies we ﬁnd provide further improved exploration at test time (see section 4.3 of the main text).
Revised: As the advantage of exploratory CO has already been demonstrated in prior works (Barrett et al., 2020), we instead ablate the stochastic policies we ﬁnd provide further improved exploration at test time (see section 4.3 of the main text).

Original: Speciﬁcally, we train the agent using DQN instead of M-DQN, as this optimises for a deterministic policy.
Revised: Speciﬁcally, we train the agent using DQN instead of M-DQN, as this optimises for a deterministic policy.

Original: For all ablations, we otherwise use the same procedure as the full ECORD agent from section 4.3 of the main text, including tuning temperature of the agents policy with a grid-search.
Revised: For all ablations, we otherwise use the same procedure as the full ECORD agent from section 4.3 of the main text, including tuning temperature of the agents policy with a grid-search.

Original: DQN is, unsurprisingly, found to be best at τ = 0 , as is the agent with the GNN ablated.
Revised: DQN is, unsurprisingly, found to be best at τ = 0 , as is the agent with the GNN ablated.

Original: The agent with the RNN ablated uses τ of 1e − 4 , 8 .
Revised: The agent with the RNN ablated uses τ of 1e − 4 , 8 .

Original: 5e − 5 and 5e − 5 for For G55, G60 and G70, respectively, with τ =5e −for all other graphs.
Revised: 5e − 5 and 5e − 5 for For G55, G60 and G70, respectively, with τ =5e −for all other graphs.

Original: Results are presented in table 6 with ECORD signiﬁcantly outperforming all ablations on larger graphs.
Revised: Results are presented in table 6 with ECORD signiﬁcantly outperforming all ablations on larger graphs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Graph network timing .
Revised: Graph network timing .

Original: As stated in the main text, the single pass of the graph neural network is negligible compared to the overall run time of ECORD.
Revised: As stated in the main text, the single pass of the graph neural network is negligible compared to the overall run time of ECORD.

Original: To be concrete, on the largest graph for which results are reported (G70 with | V | = 10 k nodes), our embedding stage takes (1 . 96 ± 0 . 09) ms , compared to the tens of seconds of exploratory decoding.
Revised: To be concrete, on the largest graph for which results are reported (G70 with | V | = 10 k nodes), our embedding stage takes (1 . 96 ± 0 . 09) ms , compared to the tens of seconds of exploratory decoding.