Original: A
Revised: A

Original: F RAMEWORK
Revised: F RAMEWORK

Original: F OR
Revised: F OR

Original: D IFFERENTIABLE
Revised: D IFFERENTIABLE

Original: D ISCOVERY
Revised: D ISCOVERY

Original: O F G RAPH A LGORITHMS
Revised: O F G RAPH A LGORITHMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently there is a surge of interests in using graph neural networks (GNNs) to learn algorithms.
Revised: Recently there is a surge of interests in using graph neural networks (GNNs) to learn algorithms.

Original: However, these works focus more on imitating existing algorithms, and are limited in two important aspects: the search space for algorithms is too small and the learned GNN models are not interpretable.
Revised: However, these works focus more on imitating existing algorithms, and are limited in two important aspects: the search space for algorithms is too small and the learned GNN models are not interpretable.

Original: To address these issues, we propose a novel framework which enlarge the search space using cheap global information from tree decomposition of the graphs, and can explain the structures of the graph leading to the decision of learned algorithms.
Revised: To address these issues, we propose a novel framework which enlarge the search space using cheap global information from tree decomposition of the graphs, and can explain the structures of the graph leading to the decision of learned algorithms.

Original: We apply our framework to three NP-complete problems on graphs and show that the framework is able to discover effective and explainable algorithms
Revised: We apply our framework to three NP-complete problems on graphs and show that the framework is able to discover effective and explainable algorithms

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Many graph problems such as maximum cut and minimum vertex cover are NP-hard.
Revised: Many graph problems such as maximum cut and minimum vertex cover are NP-hard.

Original: The classical algorithm design paradigm often requires signiﬁcant efforts from domain experts to understand and exploit problem structures, in order to come up with effective procedures.
Revised: The classical algorithm design paradigm often requires signiﬁcant efforts from domain experts to understand and exploit problem structures, in order to come up with effective procedures.

Original: However, for more complex problems and in the presence of a family of problem instances, it is becoming increasingly challenging for human to identify the problem structures and tailor algorithms.
Revised: However, for more complex problems and in the presence of a family of problem instances, it is becoming increasingly challenging for human to identify the problem structures and tailor algorithms.

Original: Thus there is a surge of interests in recent years to use learning and differentiable search to discover graph algorithms.
Revised: Thus there is a surge of interests in recent years to use learning and differentiable search to discover graph algorithms.

Original: In this context, GNNs have been widely used for representing and learning graph algorithms (Dai et al., 2018; Li et al., 2018).
Revised: In this context, GNNs have been widely used for representing and learning graph algorithms (Dai et al., 2018; Li et al., 2018).

Original: However, directly using a GNN model to deﬁne the algorithm search space may not be enough for discovering an algorithm better than existing greedy ones.
Revised: However, directly using a GNN model to deﬁne the algorithm search space may not be enough for discovering an algorithm better than existing greedy ones.

Original: Hella et al.
Revised: Hella et al.

Original: ; Sato et al.
Revised: ; Sato et al.

Original: (2019) have theoretically discussed the limitations of GNNs for expressing more powerful algorithms, by bridging the connection between GNNs and distributed local algorithms.
Revised: (2019) have theoretically discussed the limitations of GNNs for expressing more powerful algorithms, by bridging the connection between GNNs and distributed local algorithms.

Original: Sato et al.
Revised: Sato et al.

Original: (2019) derived the approximation ratios of the algorithms that can be learned by GNNs, which are much worse than those of some simple algorithms (Johnson, 1974; Chleb ´ ık & Chleb ´ ıkov ´ a, 2008).
Revised: (2019) derived the approximation ratios of the algorithms that can be learned by GNNs, which are much worse than those of some simple algorithms (Johnson, 1974; Chleb ´ ık & Chleb ´ ıkov ´ a, 2008).

Original: Intuitively, GNNs can only capture local graph patterns, but miss out the global information, which fundamentally restricts their expressiveness power.
Revised: Intuitively, GNNs can only capture local graph patterns, but miss out the global information, which fundamentally restricts their expressiveness power.

Original: To enhance the capacity of GNNs and allow for a larger search space, we incorporate global information about the graph as additional features, and augment them with other node/edge features.
Revised: To enhance the capacity of GNNs and allow for a larger search space, we incorporate global information about the graph as additional features, and augment them with other node/edge features.

Original: The idea of incorporating additional features to improve the expressiveness of GNNs has been deployed in serval existing models, by adding either unique node identiﬁers (Donnat et al., 2018; Seo et al., 2019;
Revised: The idea of incorporating additional features to improve the expressiveness of GNNs has been deployed in serval existing models, by adding either unique node identiﬁers (Donnat et al., 2018; Seo et al., 2019;

Original: Zhang et al., 2020), the information of port numbering (Sato et al., 2019), or randomness (Sato et al., 2020).
Revised: Zhang et al., 2020), the information of port numbering (Sato et al., 2019), or randomness (Sato et al., 2020).

Original: However, these features are added mainly to break the local symmetry of similar graph patterns, but do not add much information about the global graph properties.
Revised: However, these features are added mainly to break the local symmetry of similar graph patterns, but do not add much information about the global graph properties.

Original: Another important aspect which have largely been ignored in previous work is explaining the learned algorithm encoded in GNN.
Revised: Another important aspect which have largely been ignored in previous work is explaining the learned algorithm encoded in GNN.

Original: Many previous works focus on the ability of GNNs to imitate existing graph algorithms, without showing new algorithms are being learned.
Revised: Many previous works focus on the ability of GNNs to imitate existing graph algorithms, without showing new algorithms are being learned.

Original: One exception is that Khalil et al.
Revised: One exception is that Khalil et al.

Original: (2017) experimentally showed that GNN has discovered a new algorithm for minimum vertex cover problem where the node selection policy balances between node degree and the connectivity of the graph.
Revised: (2017) experimentally showed that GNN has discovered a new algorithm for minimum vertex cover problem where the node selection policy balances between node degree and the connectivity of the graph.

Original: However, this phenomenon was just mentioned in passing, and a systematic explanation of the graph patterns leading to the algorithm decision is missing.
Revised: However, this phenomenon was just mentioned in passing, and a systematic explanation of the graph patterns leading to the algorithm decision is missing.

Original: Therefore, there is an urgent need to develop explainable graph models to understand the learned algorithm.
Revised: Therefore, there is an urgent need to develop explainable graph models to understand the learned algorithm.

Original: In this paper, we propose a new framework for differentiable graph algorithm discovery (DAD), focusing on two important aspects of the discovery process: designing a larger search space, and an effective explainer model.
Revised: In this paper, we propose a new framework for differentiable graph algorithm discovery (DAD), focusing on two important aspects of the discovery process: designing a larger search space, and an effective explainer model.

Original: More speciﬁcally, we design a search space for graph algorithms by
Revised: More speciﬁcally, we design a search space for graph algorithms by

Original: [Figure]
Revised: [Figure]

Original: augmenting GNNs with cheap global graph features .
Revised: augmenting GNNs with cheap global graph features .

Original: Our proposed global features are specially designed for learning graph algorithms.
Revised: Our proposed global features are specially designed for learning graph algorithms.

Original: It reveals to be a simple yet effective way of enhancing the capacity of GNNs.
Revised: It reveals to be a simple yet effective way of enhancing the capacity of GNNs.

Original: After the GNN is sufﬁciently trained, we employ a novel graph explainer model to ﬁgure out the structural property of the graph leading to the algorithm decision.
Revised: After the GNN is sufﬁciently trained, we employ a novel graph explainer model to ﬁgure out the structural property of the graph leading to the algorithm decision.

Original: The graph explainer model allows us to transform the algorithm represented in continuous embedding space back to the interpretable discrete space, so that it can assist human experts in understanding the discovery.
Revised: The graph explainer model allows us to transform the algorithm represented in continuous embedding space back to the interpretable discrete space, so that it can assist human experts in understanding the discovery.

Original: We demonstrate our framework on three NP-hard graph algorithms: minimum vertex cover (MVC), maximum cut (Max-Cut), and minimum dominating set (MDS).
Revised: We demonstrate our framework on three NP-hard graph algorithms: minimum vertex cover (MVC), maximum cut (Max-Cut), and minimum dominating set (MDS).

Original: Our DAD framework is able to discover graph algorithms that achieve better approximation ratios.
Revised: Our DAD framework is able to discover graph algorithms that achieve better approximation ratios.

Original: Besides, we apply our explainer model to the learned algorithms and demonstrate some interesting and explainable patterns.
Revised: Besides, we apply our explainer model to the learned algorithms and demonstrate some interesting and explainable patterns.

Original: Except for methodology, we also contribute in generating a new dataset.
Revised: Except for methodology, we also contribute in generating a new dataset.

Original: To conduct experiments on these three problems, we need to generate the solutions for the purpose of evaluation or supervised learning.
Revised: To conduct experiments on these three problems, we need to generate the solutions for the purpose of evaluation or supervised learning.

Original: We run Gurobi for around 12,000,000 core hours to generate a reasonably large set of (graph problem, solution) pairs, with varying graph sizes.
Revised: We run Gurobi for around 12,000,000 core hours to generate a reasonably large set of (graph problem, solution) pairs, with varying graph sizes.

Original: It could be a very useful benchmark dataset for graph algorithm learning, which will help with future researches in this area.
Revised: It could be a very useful benchmark dataset for graph algorithm learning, which will help with future researches in this area.

Original:  O VERVIEW
Revised:  O VERVIEW

Original: Our framework will innovate on three important aspects of the graph algorithm discovery process.
Revised: Our framework will innovate on three important aspects of the graph algorithm discovery process.

Original: That is, how to design a larger search space, how to learn the GNN algorithms, and how to effectively explain the GNN after it is learned.
Revised: That is, how to design a larger search space, how to learn the GNN algorithms, and how to effectively explain the GNN after it is learned.

Original: An overview of our framework is illustrated in Figure 1.
Revised: An overview of our framework is illustrated in Figure 1.

Original: Search space design.
Revised: Search space design.

Original: To allow GNNs to represent a more powerful space of algorithms, we make use of cheap global information which can be obtained in subquadratic time in the number of nodes and edges of the graph.
Revised: To allow GNNs to represent a more powerful space of algorithms, we make use of cheap global information which can be obtained in subquadratic time in the number of nodes and edges of the graph.

Original: In particular, we will ﬁnd spanning trees of the original graph, and use the solutions on these trees as augmented node/edge features.
Revised: In particular, we will ﬁnd spanning trees of the original graph, and use the solutions on these trees as augmented node/edge features.

Original: In this way, the capacity of GNNs can be enhanced, and a larger space will be searched over to discovery new and effective algorithms.
Revised: In this way, the capacity of GNNs can be enhanced, and a larger space will be searched over to discovery new and effective algorithms.

Original: Learning method.
Revised: Learning method.

Original: We can use either supervised or unsupervised approaches to train the GNNs.
Revised: We can use either supervised or unsupervised approaches to train the GNNs.

Original: In supervised setting, the goal is to learn a GNN algorithm that can imitate the results of expensive solvers but run much faster.
Revised: In supervised setting, the goal is to learn a GNN algorithm that can imitate the results of expensive solvers but run much faster.

Original: In unsupervised setting, a larger unlabeled dataset can be used for training, and it is favorable for large graph problems where supervisions are hard to obtain.
Revised: In unsupervised setting, a larger unlabeled dataset can be used for training, and it is favorable for large graph problems where supervisions are hard to obtain.

Original: Explainer model.
Revised: Explainer model.

Original: We design a novel graph explainer model based on an information theoretic formulation, where we train a subgraph pattern selection model such that the selected subgraph patterns are more inﬂuential to the decisions of the original GNN model in an information theoretic sense.
Revised: We design a novel graph explainer model based on an information theoretic formulation, where we train a subgraph pattern selection model such that the selected subgraph patterns are more inﬂuential to the decisions of the original GNN model in an information theoretic sense.

Original: Furthermore, the explainer model is trained by differentiating through a blackbox combinatorial solver.
Revised: Furthermore, the explainer model is trained by differentiating through a blackbox combinatorial solver.

Original: Since our selection model produces discrete and interpretable local graph structures as explanations, it provides human intuition on what the kind of new algorithm is learned.
Revised: Since our selection model produces discrete and interpretable local graph structures as explanations, it provides human intuition on what the kind of new algorithm is learned.

Original:  D IFFERENTIABLE G RAPH A LGORITHM D ISCOVERY (DAD) F RAMEWORK 3. B ACKGROUND
Revised:  D IFFERENTIABLE G RAPH A LGORITHM D ISCOVERY (DAD) F RAMEWORK 3. B ACKGROUND

Original: We focus on three NP-hard problems in this paper: MVC, Max-Cut, and MDS.
Revised: We focus on three NP-hard problems in this paper: MVC, Max-Cut, and MDS.

Original: Let G =
Revised: Let G =

Original: ( V, E, w ) be a weighted graph, where V denotes the set of nodes, E the set of edges, and w : E (cid:55)→ R the edge weight function.
Revised: ( V, E, w ) be a weighted graph, where V denotes the set of nodes, E the set of edges, and w : E (cid:55)→ R the edge weight function.

Original: Furthermore, let A and L w be the adjacent and weighted Laplacian matrices of G , and B ∈ { 0 , 1 } | E |×| V | be the incidence matrix, where B ei = 

Revised: Furthermore, let A and L w be the adjacent and weighted Laplacian matrices of G , and B ∈ { 0 , 1 } | E |×| V | be the incidence matrix, where B ei =

Original: and B ej
Revised: and B ej

Original: = 1 if e =
Revised: = 1 if e =

Original: ( i, j ) ∈ E and otherwise 0.
Revised: ( i, j ) ∈ E and otherwise 0.

Original: Then these problems can be formulated as integer programming (IP) problems: min y f ( y ; G ) subject to g ( y ; G ) ≤ 0 and y ∈ { 0 , 1 } | V | , (1) where the objective and the set of constraints g =
Revised: Then these problems can be formulated as integer programming (IP) problems: min y f ( y ; G ) subject to g ( y ; G ) ≤ 0 and y ∈ { 0 , 1 } | V | , (1) where the objective and the set of constraints g =

Original: [ g 1 , · · · , g l ] (cid:62) are listed respectively in the table: f ( y ;
Revised: [ g 1 , · · · , g l ] (cid:62) are listed respectively in the table: f ( y ;

Original: G ) g
Revised: G ) g

Original: ( y ; G )
Revised: ( y ; G )

Original: MVC 1 (cid:62) y 1 − By
Revised: MVC 1 (cid:62) y 1 − By

Original: Max-Cut y (cid:62) L
Revised: Max-Cut y (cid:62) L

Original: w y
Revised: w y

Original: MDS 1 (cid:62) y 1 − y − Ay
Revised: MDS 1 (cid:62) y 1 − y − Ay

Original: The optimal solutions y ∗ ∈ { 0 , 1 } | V | to the above problems can be viewed as binary labels on each node.
Revised: The optimal solutions y ∗ ∈ { 0 , 1 } | V | to the above problems can be viewed as binary labels on each node.

Original: Therefore, learning these solutions can be modeled as binary node classiﬁcation tasks.
Revised: Therefore, learning these solutions can be modeled as binary node classiﬁcation tasks.

Original: Furthermore, we will denote Y =
Revised: Furthermore, we will denote Y =

Original: [ Y 1 , · · · , Y | V | ] (cid:62) ∈ { 0 , 1 } | V | as the random variable representing the labels on the nodes, and y =
Revised: [ Y 1 , · · · , Y | V | ] (cid:62) ∈ { 0 , 1 } | V | as the random variable representing the labels on the nodes, and y =

Original: [ y 1 , · · · , y | V | ] (cid:62) as the values of the random variable Y .
Revised: [ y 1 , · · · , y | V | ] (cid:62) as the values of the random variable Y .

Original: 3. C HEAP S OLUTIONS A S G LOBAL F EATURES
Revised: 3. C HEAP S OLUTIONS A S G LOBAL F EATURES

Original: What global features could improve the expressive power of GNNs for solving graph problems?
Revised: What global features could improve the expressive power of GNNs for solving graph problems?

Original: The following example motivates our design of global features.
Revised: The following example motivates our design of global features.

Original: A motivating example.
Revised: A motivating example.

Original: While the best known algorithm for solving a general linear system takes time O ( n 2 . 373 ) (Williams, 2012), Kelner et al.
Revised: While the best known algorithm for solving a general linear system takes time O ( n 2 . 373 ) (Williams, 2012), Kelner et al.

Original: (2013) proposed a simple algorithm for solving the system L w x = b where L w is the Laplacian matrix of a graph G , in nearly-linear time.
Revised: (2013) proposed a simple algorithm for solving the system L w x = b where L w is the Laplacian matrix of a graph G , in nearly-linear time.

Original: The algorithm can be summarized by 2 major steps: (i) Find a low-stretch spanning tree T ⊆ G , and solve the problem on the spanning tree L T x 

Revised: The algorithm can be summarized by 2 major steps: (i) Find a low-stretch spanning tree T ⊆ G , and solve the problem on the spanning tree L T x

Original: = b to obtain an initial solution x 0 ; (ii) Reﬁne the initialized solution by iteratively operating on local cycles in the original graph G .
Revised: = b to obtain an initial solution x 0 ; (ii) Reﬁne the initialized solution by iteratively operating on local cycles in the original graph G .

Original: The authors also formally proved the convergence and run-time guarantees for the algorithm.
Revised: The authors also formally proved the convergence and run-time guarantees for the algorithm.

Original: Inspired by the above successful algorithm with theoretical guarantees, we generalize the idea to solve a broader class of graph problems.
Revised: Inspired by the above successful algorithm with theoretical guarantees, we generalize the idea to solve a broader class of graph problems.

Original: In many graph problems, global optimal solutions can be found efﬁciently by algorithms such as dynamic programming if the graph is a tree.
Revised: In many graph problems, global optimal solutions can be found efﬁciently by algorithms such as dynamic programming if the graph is a tree.

Original: Therefore, we propose to use (i) solutions on spanning trees T ⊆ G and (ii) approximate solutions of greedy algorithms on G as cheap global features to augment the original graphs.
Revised: Therefore, we propose to use (i) solutions on spanning trees T ⊆ G and (ii) approximate solutions of greedy algorithms on G as cheap global features to augment the original graphs.

Original: More speciﬁcally, (i)
Revised: More speciﬁcally, (i)

Original: Given a graph G , we ﬁrst ﬁnd its spanning tree T ⊆ G .
Revised: Given a graph G , we ﬁrst ﬁnd its spanning tree T ⊆ G .

Original: There are multiple ways of constructing the trees, and we can obtain n spanning trees T (1) , · · · , T ( n ) for each graph, using the same set of n algorithms.
Revised: There are multiple ways of constructing the trees, and we can obtain n spanning trees T (1) , · · · , T ( n ) for each graph, using the same set of n algorithms.

Original: On each tree T ( k ) , we can apply dynamic programming (DP) based algorithms to quickly ﬁnd an optimal solution on the tree, denoted by a binary vector y T ( k ) ∈ { 0 , 1 } | V | .
Revised: On each tree T ( k ) , we can apply dynamic programming (DP) based algorithms to quickly ﬁnd an optimal solution on the tree, denoted by a binary vector y T ( k ) ∈ { 0 , 1 } | V | .

Original: The tree solutions, together with the trees themselves, can be used as global features for the graph G .
Revised: The tree solutions, together with the trees themselves, can be used as global features for the graph G .

Original: [Equation]
Revised: [Equation]

Original: (ii
Revised: (ii

Original: ) We can also operate on the original graph G to quickly obtain an approximate solution by using some greedy algorithms.
Revised: ) We can also operate on the original graph G to quickly obtain an approximate solution by using some greedy algorithms.

Original: With a set of m greedy algorithms denoted by Gd (1) , · · · , Gd ( m ) , we can obtain a set of approximate solutions.
Revised: With a set of m greedy algorithms denoted by Gd (1) , · · · , Gd ( m ) , we can obtain a set of approximate solutions.

Original: [Equation]
Revised: [Equation]

Original: The tree solutions and greedy solutions will be concatenated and used as node features.
Revised: The tree solutions and greedy solutions will be concatenated and used as node features.

Original: Besides, each spanning tree can be represented by a binary matrix T ( k ) ∈ { 0 , 1 } | V |×| V | , where the ( i, j ) -th entry indicates whether the edge ( i, j ) is in the tree.
Revised: Besides, each spanning tree can be represented by a binary matrix T ( k ) ∈ { 0 , 1 } | V |×| V | , where the ( i, j ) -th entry indicates whether the edge ( i, j ) is in the tree.

Original: These spanning trees will also be concatenated and used as edge features.
Revised: These spanning trees will also be concatenated and used as edge features.

Original: Then the overall features can be denoted by
Revised: Then the overall features can be denoted by

Original: [Equation]
Revised: [Equation]

Original: In the rest of this paper, we denote X i as the i -th row vector of X which corresponds to the features of node i , and Z ij as the vector Z [ i, j, :] which corresponds to the features of the edge ( i, j ) .
Revised: In the rest of this paper, we denote X i as the i -th row vector of X which corresponds to the features of node i , and Z ij as the vector Z [ i, j, :] which corresponds to the features of the edge ( i, j ) .

Original: We note that these tree and greedy solution features can typically be computed in time linear in the number of nodes and edges, which will not change substantially the computational complexity of the GNN algorithms learned later.
Revised: We note that these tree and greedy solution features can typically be computed in time linear in the number of nodes and edges, which will not change substantially the computational complexity of the GNN algorithms learned later.

Original: Furthermore, in our later experiments, we also show that the proposed global features can be generally applied to different graph problems and performs better than other trivially augmented features.
Revised: Furthermore, in our later experiments, we also show that the proposed global features can be generally applied to different graph problems and performs better than other trivially augmented features.

Original: 3. L EARNING D ISTRIBUTED L OCAL G RAPH A LGORITHM
Revised: 3. L EARNING D ISTRIBUTED L OCAL G RAPH A LGORITHM

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V, E, w ) and its associated global features X and Z , a GNN computes the node embeddings by iteratively aggregating the messages from neighbors:
Revised: ( V, E, w ) and its associated global features X and Z , a GNN computes the node embeddings by iteratively aggregating the messages from neighbors:

Original: [Equation]
Revised: [Equation]

Original: where h ( t ) i is the d -dimensional feature embedding for node i ∈ V at t -th iteration, θ ∈ Θ are the parameters in the GNN, F θ is a nonlinear propagation function, and N ( i ) denotes the neighbors of node i .
Revised: where h ( t ) i is the d -dimensional feature embedding for node i ∈ V at t -th iteration, θ ∈ Θ are the parameters in the GNN, F θ is a nonlinear propagation function, and N ( i ) denotes the neighbors of node i .

Original: Note that for a weighted graph we could also use the edge weights as edge features.
Revised: Note that for a weighted graph we could also use the edge weights as edge features.

Original: In the rest of this paper, we overload the notation Z
Revised: In the rest of this paper, we overload the notation Z

Original: and assume it is the concatenation of the global feature
Revised: and assume it is the concatenation of the global feature

Original: Z and the weights of the graph if provided.
Revised: Z and the weights of the graph if provided.

Original: After T iterations, each node embedding h ( T ) i will contain information about its T -hop neighborhood.
Revised: After T iterations, each node embedding h ( T ) i will contain information about its T -hop neighborhood.

Original: An output layer is applied to compute the probability distribution of Y i :
Revised: An output layer is applied to compute the probability distribution of Y i :

Original: [Equation]
Revised: [Equation]

Original: ∼ Bernoulli
Revised: ∼ Bernoulli

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Given the predictive distribution { p i } i ∈ V , we will retrieve the binary solution { y i ∈ { 0 , 1 }} i ∈ V by a method of conditional expectation described in (Raghavan, 1988; Karalias & Loukas, 2020).
Revised: Given the predictive distribution { p i } i ∈ V , we will retrieve the binary solution { y i ∈ { 0 , 1 }} i ∈ V by a method of conditional expectation described in (Raghavan, 1988; Karalias & Loukas, 2020).

Original: To learn the parameters θ in the GNN, we consider both the supervised and unsupervised setting:
Revised: To learn the parameters θ in the GNN, we consider both the supervised and unsupervised setting:

Original: Supervised learning with ILP solvers.
Revised: Supervised learning with ILP solvers.

Original: In the supervised setting, for each graph G , a solution y ∗ is given by running an expensive solver, and the goal is to learn a GNN-based distributed local algorithm that can imitate the results of the expensive solver but runs much faster.
Revised: In the supervised setting, for each graph G , a solution y ∗ is given by running an expensive solver, and the goal is to learn a GNN-based distributed local algorithm that can imitate the results of the expensive solver but runs much faster.

Original: However, one challenge is that there could exist symmetry in the solution y ∗ .
Revised: However, one challenge is that there could exist symmetry in the solution y ∗ .

Original: For example, given a feasible solution y ∗ to a Max-Cut problem, the vector 1 − y ∗ is an equivalent solution that gives the same cost and the same cut-set, so there could be multiple modes in the labels which will cause the ‘modeaveraging’ problem (Hinton et al., 1995).
Revised: For example, given a feasible solution y ∗ to a Max-Cut problem, the vector 1 − y ∗ is an equivalent solution that gives the same cost and the same cut-set, so there could be multiple modes in the labels which will cause the ‘modeaveraging’ problem (Hinton et al., 1995).

Original: To overcome this problem, instead of predicting only one probability p i as in Eq.
Revised: To overcome this problem, instead of predicting only one probability p i as in Eq.

Original: 7, we generate K many probability distributions, and employ the hindsight loss for each node prediction (Guzman-Rivera et al., 2012; Li et al., 2018).
Revised: 7, we generate K many probability distributions, and employ the hindsight loss for each node prediction (Guzman-Rivera et al., 2012; Li et al., 2018).

Original: That is
Revised: That is

Original: Hindsight loss: min k ∈{ 1 , ··· ,K } (cid:96) ( y ∗ i , p i,k ) where p i,k =
Revised: Hindsight loss: min k ∈{ 1 , ··· ,K } (cid:96) ( y ∗ i , p i,k ) where p i,k =

Original: OutputLayer k ( h ( T ) i ) , (8)
Revised: OutputLayer k ( h ( T ) i ) , (8)

Original: K could be the number of modes, and (cid:96) is the binary cross entropy loss.
Revised: K could be the number of modes, and (cid:96) is the binary cross entropy loss.

Original: Unsupervised learning with continuous relaxation.
Revised: Unsupervised learning with continuous relaxation.

Original: Generating optimal solutions
Revised: Generating optimal solutions

Original: for large graphs
Revised: for large graphs

Original: can be very expensive.
Revised: can be very expensive.

Original: Therefore, we also consider the unsupervised setting which allows the use of a large unlabeled training dataset.
Revised: Therefore, we also consider the unsupervised setting which allows the use of a large unlabeled training dataset.

Original: We will construct the unsupervised training loss based on the optimization objective f and constraints g
Revised: We will construct the unsupervised training loss based on the optimization objective f and constraints g

Original: =
Revised: =

Original: [ g 1 , · · · , g l ] of the graph problems described in
Revised: [ g 1 , · · · , g l ] of the graph problems described in

Original: Sec 3.1.
Revised: Sec 3.1.

Original: The design of the loss mainly follows (Karalias & Loukas, 2020) and more details and principles can be found there.
Revised: The design of the loss mainly follows (Karalias & Loukas, 2020) and more details and principles can be found there.

Original: To summarize, let p = { p i } i ∈ V be the predictive probability given by the GNN, our unsupervised loss is deﬁned as the following probabilistic penalty loss:
Revised: To summarize, let p = { p i } i ∈ V be the predictive probability given by the GNN, our unsupervised loss is deﬁned as the following probabilistic penalty loss:

Original: [Equation]
Revised: [Equation]

Original: where Y ∼ Bernoulli ( p
Revised: where Y ∼ Bernoulli ( p

Original: [Equation]
Revised: [Equation]

Original: , (9) and β is a penalty coefﬁcient.
Revised: , (9) and β is a penalty coefﬁcient.

Original: Note that it is a relaxed version of the loss proposed by Karalias &
Revised: Note that it is a relaxed version of the loss proposed by Karalias &

Original: Loukas (2020), who considers the probability of the satisfaction of all constraints jointly.
Revised: Loukas (2020), who considers the probability of the satisfaction of all constraints jointly.

Original: Here we treat the l constraints separately for efﬁcient computations.
Revised: Here we treat the l constraints separately for efﬁcient computations.

Original: See Appendix B for more details.
Revised: See Appendix B for more details.

Original: Approximation ratio: As GNN is a type of distributed local algorithm, it is easy to see that it admits the identify function in the function space.
Revised: Approximation ratio: As GNN is a type of distributed local algorithm, it is easy to see that it admits the identify function in the function space.

Original: Given that we can use the greedy algorithmic features, it is easy to see that our design space admits an algorithm that enjoys the same approximation ratio as the greedy algorithm, like O (log n ) for MDS, or 2 for MVC and Max-Cut.
Revised: Given that we can use the greedy algorithmic features, it is easy to see that our design space admits an algorithm that enjoys the same approximation ratio as the greedy algorithm, like O (log n ) for MDS, or 2 for MVC and Max-Cut.

Original: This naturally achieves or improves the ratio obtained by Sato et al.
Revised: This naturally achieves or improves the ratio obtained by Sato et al.

Original: 3. E XPLAIN T HE L EARNED GNN
Revised: 3. E XPLAIN T HE L EARNED GNN

Original: We simply denoted the learned GNN as F θ .
Revised: We simply denoted the learned GNN as F θ .

Original: To understand what algorithm is discovered in F θ , we propose a novel method to explain its predictions { p i }
Revised: To understand what algorithm is discovered in F θ , we propose a novel method to explain its predictions { p i }

Original: i ∈ V .
Revised: i ∈ V .

Original: We want to ﬁnd that, for each node i , which subset of nodes V iS ⊆ V and edges E iS ⊆
Revised: We want to ﬁnd that, for each node i , which subset of nodes V iS ⊆ V and edges E iS ⊆

Original: E are most inﬂuential to the prediction p i .
Revised: E are most inﬂuential to the prediction p i .

Original: Besides, the global features X
Revised: Besides, the global features X

Original: and Z are induced by the solutions of n spanning trees and m greedy algorithms, which we denote as S :
Revised: and Z are induced by the solutions of n spanning trees and m greedy algorithms, which we denote as S :

Original: = { T (1) , · · · , T ( n ) , Gd (1) , · · · , Gd ( m ) } .
Revised: = { T (1) , · · · , T ( n ) , Gd (1) , · · · , Gd ( m ) } .

Original: We are also interested in ﬁnding which trees or algorithms provide more useful features for the prediction.
Revised: We are also interested in ﬁnding which trees or algorithms provide more useful features for the prediction.

Original: Since we focus on node-level predictions, our explainer is deﬁned as a mapping that ﬁnds nodedependent subsets
Revised: Since we focus on node-level predictions, our explainer is deﬁned as a mapping that ﬁnds nodedependent subsets

Original: [Equation]
Revised: [Equation]

Original: where V iS and E iS are selected subsets of k V nodes and k E edges respectively, and S iS ⊆
Revised: where V iS and E iS are selected subsets of k V nodes and k E edges respectively, and S iS ⊆

Original: S is a subset of k S trees/algorithms.
Revised: S is a subset of k S trees/algorithms.

Original: More precisely, we parameterize E as a second GNN along with a top- K operation, where the GNN is applied to score the set of structural elements in ( V, E, S ) , and the top- K operation selects the ( k V , k E , k S ) highest scoring elements.
Revised: More precisely, we parameterize E as a second GNN along with a top- K operation, where the GNN is applied to score the set of structural elements in ( V, E, S ) , and the top- K operation selects the ( k V , k E , k S ) highest scoring elements.

Original: Details are provided below.
Revised: Details are provided below.

Original: Parameterization of E .
Revised: Parameterization of E .

Original: It is notable the prediction p
Revised: It is notable the prediction p

Original: i is computed by F θ based on the T -hop subgraph of the node i and is irrelevant to nodes of distance larger than T .
Revised: i is computed by F θ based on the T -hop subgraph of the node i and is irrelevant to nodes of distance larger than T .

Original: Therefore, when working on node i , the explainer only needs to select the inﬂuential nodes/edges from its T -hop subgraph, denoted by G ( i, T ) :
Revised: Therefore, when working on node i , the explainer only needs to select the inﬂuential nodes/edges from its T -hop subgraph, denoted by G ( i, T ) :

Original: = ( V ( i, T ) , E ( i, T ) , w ) .
Revised: = ( V ( i, T ) , E ( i, T ) , w ) .

Original: Therefore, we deﬁne explainer GNN as
Revised: Therefore, we deﬁne explainer GNN as

Original: Explainer
Revised: Explainer

Original: [Equation]
Revised: [Equation]

Original: where X ( i, T ) :
Revised: where X ( i, T ) :

Original: =
Revised: =

Original: [ X j ] j ∈ V ( i,T ) and Z
Revised: [ X j ] j ∈ V ( i,T ) and Z

Original: ( i, T ) :=
Revised: ( i, T ) :=

Original: [ Z jk ] ( j,k ) ∈ E ( i,T ) are features induced by the subgraph.
Revised: [ Z jk ] ( j,k ) ∈ E ( i,T ) are features induced by the subgraph.

Original: Then multi-layer perceptrons (MLP) are applied to deﬁne the scoring functions:
Revised: Then multi-layer perceptrons (MLP) are applied to deﬁne the scoring functions:

Original: (I) Node scores: c j = MLP Vφ ( h E j ) for each node j ∈ V ( i, T ) .
Revised: (I) Node scores: c j = MLP Vφ ( h E j ) for each node j ∈ V ( i, T ) .

Original: (II) Edge scores: c j,k = MLP Eφ ([ h E j , h E k , E j,k ]) for each edge (
Revised: (II) Edge scores: c j,k = MLP Eφ ([ h E j , h E k , E j,k ]) for each edge (

Original: j, k ) ∈ E ( i, T ) .
Revised: j, k ) ∈ E ( i, T ) .

Original: (III) Feature scores: c = MLP S φ (cid:0) Pooling (cid:0) { h E j } j ∈ V ( i,T ) (cid:1)(cid:1) , where the output c is a ( n + m ) -dimensional vector that gives the score for features induced by the n spanning trees and the m greedy algorithms.
Revised: (III) Feature scores: c = MLP S φ (cid:0) Pooling (cid:0) { h E j } j ∈ V ( i,T ) (cid:1)(cid:1) , where the output c is a ( n + m ) -dimensional vector that gives the score for features induced by the n spanning trees and the m greedy algorithms.

Original: With the scores, the subsets ( V iS , E iS , S iS ) can be selected as choosing the elements with scores in top- K , for K = k V , k E , and k S , respectively.
Revised: With the scores, the subsets ( V iS , E iS , S iS ) can be selected as choosing the elements with scores in top- K , for K = k V , k E , and k S , respectively.

Original: In summary, the explainer model can be written as the composition of three operations
Revised: In summary, the explainer model can be written as the composition of three operations

Original: Parameterized
Revised: Parameterized

Original: [Equation]
Revised: [Equation]

Original: Variational lower bound of mutual information.
Revised: Variational lower bound of mutual information.

Original: To learn the explainer E φ , we employ the learning to explain (L2X) framework in (Chen et al., 2018).
Revised: To learn the explainer E φ , we employ the learning to explain (L2X) framework in (Chen et al., 2018).

Original: In L2X, E φ is learned by maximizing the mutual information between the predicted label Y i and the selected subsets ( V iS , E iS , S iS ) .
Revised: In L2X, E φ is learned by maximizing the mutual information between the predicted label Y i and the selected subsets ( V iS , E iS , S iS ) .

Original: However, due to intractability, a variational distribution parameterized by a neural network Q ψ will be learned jointly with the explainer E φ to maximize the variational lower bound of the mutual information
Revised: However, due to intractability, a variational distribution parameterized by a neural network Q ψ will be learned jointly with the explainer E φ to maximize the variational lower bound of the mutual information

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the predictive probability p i given by the learned GNN, over all node i in graph G and all graph G in the training set G train .
Revised: where the expectation is taken over the predictive probability p i given by the learned GNN, over all node i in graph G and all graph G in the training set G train .

Original: Parameterization of Q .
Revised: Parameterization of Q .

Original: Again we will parametrize the variational distribution Q ψ by a third GNN
Revised: Again we will parametrize the variational distribution Q ψ by a third GNN

Original: F Qψ : ( V iS , E iS , X iS , Z iS ) (cid:55)→ { h Qj } j ∈ V ( i,T ) followed by pooling and MLP:
Revised: F Qψ : ( V iS , E iS , X iS , Z iS ) (cid:55)→ { h Qj } j ∈ V ( i,T ) followed by pooling and MLP:

Original: Variational Distribution: Q ψ =
Revised: Variational Distribution: Q ψ =

Original: MLP ψ (cid:16)
Revised: MLP ψ (cid:16)

Original: [Equation]
Revised: [Equation]

Original: where the features ( X iS , Z iS )
Revised: where the features ( X iS , Z iS )

Original: =
Revised: =

Original: H ( V iS , E iS , S iS , X, Z ) are induced by the selected subsets via a differentiable mapping H .
Revised: H ( V iS , E iS , S iS , X, Z ) are induced by the selected subsets via a differentiable mapping H .

Original: Differentiable top- K .
Revised: Differentiable top- K .

Original: The additional technical challenges for optimizing the variational objective in Eq.
Revised: The additional technical challenges for optimizing the variational objective in Eq.

Original: 13 is that the top- K operation in the parameterized explainer (Eq. 12) is not differentiable.
Revised: 13 is that the top- K operation in the parameterized explainer (Eq. 12) is not differentiable.

Original: More speciﬁcally, a top- K operation over the scores c 1 , · · · , c M for some M is equivalent to solving the following integer linear programming (ILP) problem:
Revised: More speciﬁcally, a top- K operation over the scores c 1 , · · · , c M for some M is equivalent to solving the following integer linear programming (ILP) problem:

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that the optimal solution x ∗ j = 1 only if c j is among the top- K scores.
Revised: It is easy to see that the optimal solution x ∗ j = 1 only if c j is among the top- K scores.

Original: Given this equivalence, we can employ the blackbox differentiation techniques introduced by Vlastelica et al.
Revised: Given this equivalence, we can employ the blackbox differentiation techniques introduced by Vlastelica et al.

Original: (2019) to compute the gradient of the ILP solver, and optimize the parameters φ in the explainer.
Revised: (2019) to compute the gradient of the ILP solver, and optimize the parameters φ in the explainer.

Original: Discussion.
Revised: Discussion.

Original: Our explainer model is very different from GNNExplainer
Revised: Our explainer model is very different from GNNExplainer

Original: (Ying et al., 2019), which may be of independent interests in term of explainer model for GNNs.
Revised: (Ying et al., 2019), which may be of independent interests in term of explainer model for GNNs.

Original: For instance, GNNExplainer optimizes the selection mask (i.e., the explainer) for an individual graph, while we parameterize the explainer as another GNN which can be applied to explain the predictions on a class of different graphs.
Revised: For instance, GNNExplainer optimizes the selection mask (i.e., the explainer) for an individual graph, while we parameterize the explainer as another GNN which can be applied to explain the predictions on a class of different graphs.

Original: Besides, the variational Q in GNNExplainer is deﬁned to be the original GNN F θ which is ﬁxed, while we optimize Q over a parametric family, so that our method is more principled and optimal.
Revised: Besides, the variational Q in GNNExplainer is deﬁned to be the original GNN F θ which is ﬁxed, while we optimize Q over a parametric family, so that our method is more principled and optimal.

Original: Furthermore, we formulate the top- K selection as a solution to an ILP and apply a blackbox differentiation technique to optimize the explainer model, which has not been attempted in the space of explainer models before.
Revised: Furthermore, we formulate the top- K selection as a solution to an ILP and apply a blackbox differentiation technique to optimize the explainer model, which has not been attempted in the space of explainer models before.

Original: We provide some empirical justiﬁcations in Appendix G. Discussion with other explainer models can be found in Appendix A.
Revised: We provide some empirical justiﬁcations in Appendix G. Discussion with other explainer models can be found in Appendix A.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Expressiveness of GNNs for representing algorithms.
Revised: Expressiveness of GNNs for representing algorithms.

Original: GNNs have been used to learn graph algorithms.
Revised: GNNs have been used to learn graph algorithms.

Original: Some recent works have discussed the representation power of GNNs for representing graph algorithms.
Revised: Some recent works have discussed the representation power of GNNs for representing graph algorithms.

Original: Xu et al.
Revised: Xu et al.

Original: (2018) showed that GNNs are at most as powerful as the Weisfeiler-Lehman (WL) test in distinguishing graph structures.
Revised: (2018) showed that GNNs are at most as powerful as the Weisfeiler-Lehman (WL) test in distinguishing graph structures.

Original: Maron et al.
Revised: Maron et al.

Original: (2019; 2018) proposed k -GNN which can represent the results of k -WL test.
Revised: (2019; 2018) proposed k -GNN which can represent the results of k -WL test.

Original: Barcel´o et al.
Revised: Barcel´o et al.

Original: (2019) showed that GNNs are too weak to represent logical Boolean classiﬁers - FOC 2 , and proposed to add a global read-out function break this limitation.
Revised: (2019) showed that GNNs are too weak to represent logical Boolean classiﬁers - FOC 2 , and proposed to add a global read-out function break this limitation.

Original: Focusing on combinatorial problems including MDS and MVC, Sato et al.
Revised: Focusing on combinatorial problems including MDS and MVC, Sato et al.

Original: (2019) derived the approximation ratios of algorithms that GNNs can learn, and theoretically showed that adding the information of port numbering and weak-coloring can improve those approximation ratios.
Revised: (2019) derived the approximation ratios of algorithms that GNNs can learn, and theoretically showed that adding the information of port numbering and weak-coloring can improve those approximation ratios.

Original: Learning based graph algorithms.
Revised: Learning based graph algorithms.

Original: Empirically, several works have demonstrated the ability of
Revised: Empirically, several works have demonstrated the ability of

Original: GNNs for imitating existing graph algorithms.
Revised: GNNs for imitating existing graph algorithms.

Original: For instance, GNN is used to learn the convergent states of graph algorithms (Dai et al., 2018), imitate DP algorithms (Xu et al., 2019), and imitate individual steps and intermediate outputs of classical graph algorithms (Veli ˇ ckovi
Revised: For instance, GNN is used to learn the convergent states of graph algorithms (Dai et al., 2018), imitate DP algorithms (Xu et al., 2019), and imitate individual steps and intermediate outputs of classical graph algorithms (Veli ˇ ckovi

Original: ´ c et al., 2019).
Revised: ´ c et al., 2019).

Original: To learn a new and more effective graph algorithm, Khalil et al.
Revised: To learn a new and more effective graph algorithm, Khalil et al.

Original: (2017) combined reinforcement learning with GNNs to learn greedy policies that can solve graph problems such as MVC and Maxcut.
Revised: (2017) combined reinforcement learning with GNNs to learn greedy policies that can solve graph problems such as MVC and Maxcut.

Original: Karalias & Loukas (2020) proposed an unsupervised framework for learning graph algorithms and demonstrated the results on maximum clique and local partitioning problems.
Revised: Karalias & Loukas (2020) proposed an unsupervised framework for learning graph algorithms and demonstrated the results on maximum clique and local partitioning problems.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In Section 5.1, we ﬁrst introduce problems we are going to tackle, and the baselines we are going to compare with.
Revised: In Section 5.1, we ﬁrst introduce problems we are going to tackle, and the baselines we are going to compare with.

Original: Then in Section 5.2 and Section 5.3, we show the empirical approximation ratio of
Revised: Then in Section 5.2 and Section 5.3, we show the empirical approximation ratio of

Original: DAD on several synthetic and real-world datasets, and compare against several known polynomial algorithms, as well as other relevant graph neural network variants.
Revised: DAD on several synthetic and real-world datasets, and compare against several known polynomial algorithms, as well as other relevant graph neural network variants.

Original: Finally in Section 5.4, we leverage our proposed explainer to understand the learned algorithm.
Revised: Finally in Section 5.4, we leverage our proposed explainer to understand the learned algorithm.

Original: 5. E XPERIMENT SETUP
Revised: 5. E XPERIMENT SETUP

Original: We leave the details of greedy algorithms and dynamic programming on trees in Appendix C.
Revised: We leave the details of greedy algorithms and dynamic programming on trees in Appendix C.

Original: We ﬁrst compare against some of the cheap algorithms that are also served as features.
Revised: We ﬁrst compare against some of the cheap algorithms that are also served as features.

Original: As we use the continuous relaxation in Eq.
Revised: As we use the continuous relaxation in Eq.

Original: 9, we also compare with the linear/semi-deﬁnite programming (LP/SDP) with randomized rounding for these problems:
Revised: 9, we also compare with the linear/semi-deﬁnite programming (LP/SDP) with randomized rounding for these problems:

Original: • MVC: LogN-Approx (node-degree greedy), 2Approx-Greedy (edge selection with degree greedy), 2Approx (edge selection) and LP relaxation are included.
Revised: • MVC: LogN-Approx (node-degree greedy), 2Approx-Greedy (edge selection with degree greedy), 2Approx (edge selection) and LP relaxation are included.

Original: • MDS: Greedy (sequential coverage based greedy selection) and LP relaxation are included.
Revised: • MDS: Greedy (sequential coverage based greedy selection) and LP relaxation are included.

Original: • Max-CUT: 2Approx-Greedy (greedy local adjustment), SDP and MaxSpanning (solution directly obtained on max spanning tree, as any node assignment is a valid cut) are included.
Revised: • Max-CUT: 2Approx-Greedy (greedy local adjustment), SDP and MaxSpanning (solution directly obtained on max spanning tree, as any node assignment is a valid cut) are included.

Original: We here include several recent works that extends GNNs with different forms of global information:
Revised: We here include several recent works that extends GNNs with different forms of global information:

Original: • Onehot (Karalias & Loukas, 2020; Seo et al., 2019): adds one-hot encoding to a random node.
Revised: • Onehot (Karalias & Loukas, 2020; Seo et al., 2019): adds one-hot encoding to a random node.

Original: • RandFeat (Sato et al., 2020): uses random node features to break the tie.
Revised: • RandFeat (Sato et al., 2020): uses random node features to break the tie.

Original: • CPNGNNs (Sato et al., 2019): leverages port numbering in GNN’s message passing, together with weak 2-coloring node features.
Revised: • CPNGNNs (Sato et al., 2019): leverages port numbering in GNN’s message passing, together with weak 2-coloring node features.

Original: As CPNGNNs only works for graphs with bounded degree, we use the position encoding (Vaswani et al., 2017) for port number representation instead.
Revised: As CPNGNNs only works for graphs with bounded degree, we use the position encoding (Vaswani et al., 2017) for port number representation instead.

Original: • Raw : we simply use constant (or edge weight, when applicable) for node/edge features, which serves as the baseline for all the global information augmented GNNs.
Revised: • Raw : we simply use constant (or edge weight, when applicable) for node/edge features, which serves as the baseline for all the global information augmented GNNs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. Q UANTITATIVE RESULTS ON RANDOM GRAPHS
Revised: 5. Q UANTITATIVE RESULTS ON RANDOM GRAPHS

Original: Data preparation We use two random graph models, namely Erdos Renyi (ER) with edge probability 0 . 15 , and Barabasi Albert (BA) with edge attachment 4 to generate random graphs for training and test.
Revised: Data preparation We use two random graph models, namely Erdos Renyi (ER) with edge probability 0 . 15 , and Barabasi Albert (BA) with edge attachment 4 to generate random graphs for training and test.

Original: We vary the graph sizes in { 16-20 , 32-40 , 64-75 , 128-150 , 256-300 , 512-600 , 1024-1100 } .
Revised: We vary the graph sizes in { 16-20 , 32-40 , 64-75 , 128-150 , 256-300 , 512-600 , 1024-1100 } .

Original: In each { problem , graph type , graph size } conﬁguration, we generation { 10 k, 1 k, 1 k } graph instances for training, validation and test, respectively.
Revised: In each { problem , graph type , graph size } conﬁguration, we generation { 10 k, 1 k, 1 k } graph instances for training, validation and test, respectively.

Original: We run Gurobi on each instance to get the solution for evaluation or supervised training.
Revised: We run Gurobi on each instance to get the solution for evaluation or supervised training.

Original: See Appendix D for more details on setup and
Revised: See Appendix D for more details on setup and

Original: Gurobi .
Revised: Gurobi .

Original: Main results: Figure 2 shows the comparison between classical algorithms.
Revised: Main results: Figure 2 shows the comparison between classical algorithms.

Original: We can see DAD leverages but goes beyond the cheap algorithms, with both supervised and unsupervised (DAD-U) learning strategies.
Revised: We can see DAD leverages but goes beyond the cheap algorithms, with both supervised and unsupervised (DAD-U) learning strategies.

Original: The gain is signiﬁcant regarding both the average and maximum test ratio.
Revised: The gain is signiﬁcant regarding both the average and maximum test ratio.

Original: While the maximum ratio is similar in MDS, DAD still gains advantages in average case, which is expected as we do empirical risk minimization during training.
Revised: While the maximum ratio is similar in MDS, DAD still gains advantages in average case, which is expected as we do empirical risk minimization during training.

Original: Figure 3 compares against other GNNs with global information, where our design consistently outperforms alternatives.
Revised: Figure 3 compares against other GNNs with global information, where our design consistently outperforms alternatives.

Original: We also tried to pad the baseline GNN features with either random or constant values to match the number of parameters in GNN.
Revised: We also tried to pad the baseline GNN features with either random or constant values to match the number of parameters in GNN.

Original: This doesn’t help the baselines much, which veriﬁes that the gain of DAD is not simply coming from the input size.
Revised: This doesn’t help the baselines much, which veriﬁes that the gain of DAD is not simply coming from the input size.

Original: See Appendix F.3.1 for more details.
Revised: See Appendix F.3.1 for more details.

Original: Extrapolation: We train the models on graphs up to 300 nodes, and evaluate them up to 1100 nodes to see their extrapolation ability.
Revised: Extrapolation: We train the models on graphs up to 300 nodes, and evaluate them up to 1100 nodes to see their extrapolation ability.

Original: We show the details in Appendix F.2, where the extrapolation results are still consistently better than the best classical baseline algorithm.
Revised: We show the details in Appendix F.2, where the extrapolation results are still consistently better than the best classical baseline algorithm.

Original: Ablation study:
Revised: Ablation study:

Original: Here we try to understand the contributions of tree solution versus greedy solution.
Revised: Here we try to understand the contributions of tree solution versus greedy solution.

Original: We present such study in Figure 12 in appendix on the three tasks.
Revised: We present such study in Figure 12 in appendix on the three tasks.

Original: Our main takeaways are: 1) both of the two feature families can boost GNN in most cases, comparing to not using any features.
Revised: Our main takeaways are: 1) both of the two feature families can boost GNN in most cases, comparing to not using any features.

Original: 2) having multiple trees can also be helpful for MVC and MDS, as the spanning tree is not unique. 3) using both types of the features is better than either of them.
Revised: 2) having multiple trees can also be helpful for MVC and MDS, as the spanning tree is not unique. 3) using both types of the features is better than either of them.

Original: This validates our design principle.
Revised: This validates our design principle.

Original: Time-Solution trade-off: As exhaustive algorithms can already achieve optimal solution in ﬁnite time, we want to study the trade-off between time and the approximation ratio obtained by different algorithms.
Revised: Time-Solution trade-off: As exhaustive algorithms can already achieve optimal solution in ﬁnite time, we want to study the trade-off between time and the approximation ratio obtained by different algorithms.

Original: We show the results in ﬁgure 4 on BA graphs with minimum 1024 nodes.
Revised: We show the results in ﬁgure 4 on BA graphs with minimum 1024 nodes.

Original: For each single graph, we record down the time spent for solving and the approximation ratio, and obtain a dot in the scatter plot.
Revised: For each single graph, we record down the time spent for solving and the approximation ratio, and obtain a dot in the scatter plot.

Original: As Gurobi solver is an iterative process, we plot the data points every time it improves the solution on the graph, and truncate.
Revised: As Gurobi solver is an iterative process, we plot the data points every time it improves the solution on the graph, and truncate.

Original: We can see DAD can achieve similar or better solution quality than what can be achieved by Gurobi within 2 magnitudes more time.
Revised: We can see DAD can achieve similar or better solution quality than what can be achieved by Gurobi, with 2 magnitudes less time.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. R ESULTS ON REAL WORLD GRAPHS
Revised: 5. R ESULTS ON REAL WORLD GRAPHS

Original: Data preparation We ﬁrst collect several social network datasets Twitter and Github from
Revised: Data preparation We ﬁrst collect several social network datasets Twitter and Github from

Original: Morris et al.
Revised: Morris et al.

Original: We also collect Memetracker , Optsicom and follow Khalil et al.
Revised: We also collect Memetracker , Optsicom and follow Khalil et al.

Original: (2017) to generate training/evaluation graphs.
Revised: (2017) to generate training/evaluation graphs.

Original: We run MVC and MDS tasks on the social networks, and run MAXCUT on the physics( Optsicom ) graphs.
Revised: We run MVC and MDS tasks on the social networks, and run MAXCUT on the physics( Optsicom ) graphs.

Original: See Appendix E for more information about the dataset statistics and real-world experiment results.
Revised: See Appendix E for more information about the dataset statistics and real-world experiment results.

Original: Results: We show the relative error compared to the optimal solution obtained by Gurobi in
Revised: Results: We show the relative error compared to the optimal solution obtained by Gurobi in

Original: Table 1 and Table 2.
Revised: Table 1 and Table 2.

Original: We report DAD’s performance with supervised learning setting.
Revised: We report DAD’s performance with supervised learning setting.

Original: It still achieves the best performance on both the social networks and physics graphs, where the relative error can be several magnitudes smaller than the second best classical algorithms in some cases.
Revised: It still achieves the best performance on both the social networks and physics graphs, where the relative error can be several magnitudes smaller than the second best classical algorithms in some cases.

Original: 5. E XPLAINING LEARNED ALGORITHMS .
Revised: 5. E XPLAINING LEARNED ALGORITHMS .

Original: In this section, we use our proposed explainer to explain the learned algorithm on BA graphs.
Revised: In this section, we use our proposed explainer to explain the learned algorithm on BA graphs.

Original: By zooming in into the important discrete substructures of the graph, we are able to ﬁnd some interesting algorithmic patterns that aligns with our intuition.
Revised: By zooming in into the important discrete substructures of the graph, we are able to ﬁnd some interesting algorithmic patterns that aligns with our intuition.

Original: Explanation with edge selection: Here we limit the number of nodes to be 5, and edges to be 10 when explaining each target node of a graph, and parameterize E φ and Q ψ using GNN with the same number of layers as the target GNN.
Revised: Explanation with edge selection: Here we limit the number of nodes to be 5, and edges to be 10 when explaining each target node of a graph, and parameterize E φ and Q ψ using GNN with the same number of layers as the target GNN.

Original: We found in ﬁgure 5 that some nodes depict the greedy behavior, where the budget for edge explanations is mostly allocated to adjacent edges.
Revised: We found in ﬁgure 5 that some nodes depict the greedy behavior, where the budget for edge explanations is mostly allocated to adjacent edges.

Original: Note that although we use greedy algorithm solution as features, the GNN doesn’t know about the underlying physical meaning of them.
Revised: Note that although we use greedy algorithm solution as features, the GNN doesn’t know about the underlying physical meaning of them.

Original: This shows that our explainer is able to re-discover the greedy algorithm.
Revised: This shows that our explainer is able to re-discover the greedy algorithm.

Original: Explanation with node selection: When running the explainer on a single graph with different target nodes, we found that there are several nodes appear frequently in the selected subset.
Revised: Explanation with node selection: When running the explainer on a single graph with different target nodes, we found that there are several nodes appear frequently in the selected subset.

Original: We call these nodes as anchor nodes .
Revised: We call these nodes as anchor nodes .

Original: We try to understand the correlation between anchor nodes and the node predictive probabilities in ﬁgure 6.
Revised: We try to understand the correlation between anchor nodes and the node predictive probabilities in ﬁgure 6.

Original: We ﬁrst run explainer on every node in a graph, and count how many times each node has appeared in the explanation of other nodes.
Revised: We ﬁrst run explainer on every node in a graph, and count how many times each node has appeared in the explanation of other nodes.

Original: The node color indicates such frequency, where the dark red has highest frequency.
Revised: The node color indicates such frequency, where the dark red has highest frequency.

Original: We can clearly see the existence of such anchors.
Revised: We can clearly see the existence of such anchors.

Original: Then we show the predicted node probability as the boundary color of each node, where the dark blue has probability close to 1 . 0 .
Revised: Then we show the predicted node probability as the boundary color of each node, where the dark blue has probability close to 1 . 0 .

Original: We can see no matter how many nodes with dark blue boundaries, the anchor nodes tend to distribute evenly among both high probability and low probability nodes.
Revised: We can see no matter how many nodes with dark blue boundaries, the anchor nodes tend to distribute evenly among both high probability and low probability nodes.

Original: Our hypothesis is that GNN leverage the similarities between target node and the diverse anchor nodes to make a proper decision, where the anchors serve like landmarks in kernel methods.
Revised: Our hypothesis is that GNN leverage the similarities between target node and the diverse anchor nodes to make a proper decision, where the anchors serve like landmarks in kernel methods.

Original: Such anchor node behavior is also consistent to some recent distance based designs (You et al., 2019; Li et al., 2020) of GNNs.
Revised: Such anchor node behavior is also consistent to some recent distance based designs (You et al., 2019; Li et al., 2020) of GNNs.

Original: [Figure]
Revised: [Figure]

Original: Explanation with feature selection: We further look at the contribution of different cheap global features to the GNN prediction.
Revised: Explanation with feature selection: We further look at the contribution of different cheap global features to the GNN prediction.

Original: We take the Max-Cut problem, where we use 5 global features, including Greedy and { Min, AKPW, Max, Max } -Spanning trees.
Revised: We take the Max-Cut problem, where we use 5 global features, including Greedy and { Min, AKPW, Max, Max } -Spanning trees.

Original: We include MaxSpanning twice as the cut solutions are symmetric and come in pairs.
Revised: We include MaxSpanning twice as the cut solutions are symmetric and come in pairs.

Original: We can see from ﬁgure 7 when budget is set to 3, all the explanations select { Greedy , AKPW , MaxSpanning } , despite that MaxSpanning solution has higher quality than AKPW .
Revised: We can see from ﬁgure 7 when budget is set to 3, all the explanations select { Greedy , AKPW , MaxSpanning } , despite that MaxSpanning solution has higher quality than AKPW .

Original: This shows that the explainer learns that two MaxSpaning are duplicated in identifying global information.
Revised: This shows that the explainer learns that two MaxSpaning are duplicated in identifying global information.

Original: When the budget is limited to 2, our explainer still consistently identiﬁes the best performing algorithm, i.e. , Greedy .
Revised: When the budget is limited to 2, our explainer still consistently identiﬁes the best performing algorithm, i.e. , Greedy .

Original: From this we can learn two principles in instantiating our framework: 1) better algorithm solution generally introduces better global information; 2) having diverse algorithmic solutions is better than multiple similar ones.
Revised: From this we can learn two principles in instantiating our framework: 1) better algorithm solution generally introduces better global information; 2) having diverse algorithmic solutions is better than multiple similar ones.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel framework, called DAD, for differentiable discovery of graph algorithms.
Revised: In this paper, we propose a novel framework, called DAD, for differentiable discovery of graph algorithms.

Original: We have demonstrated that our framework can search over a large space of graph algorithms using GNNs and global features, and can interpret the learned algorithm by a novel explainer model.
Revised: We have demonstrated that our framework can search over a large space of graph algorithms using GNNs and global features, and can interpret the learned algorithm by a novel explainer model.

Original: It can potentially serve as a powerful tool to improve the existing graph-based algorithms, as well as providing explanations for other downstream tasks.
Revised: It can potentially serve as a powerful tool to improve the existing graph-based algorithms, as well as providing explanations for other downstream tasks.

Original: Appendix A O THER RELATED WORKS
Revised: Appendix A O THER RELATED WORKS

Original: Explainer models.
Revised: Explainer models.

Original: Existing work on explainer models approach the problems from three directions.
Revised: Existing work on explainer models approach the problems from three directions.

Original: The ﬁrst line of work use gradients of the outputs with respect to inputs to identify the salient features in the inputs (Simonyan et al., 2013; Springenberg et al., 2014); The second approximates the model with simple interpretable models, such as locally additive models (Bach et al., 2015; Kindermans et al., 2016; Ribeiro et al., 2016; Lundberg & Lee, 2017); the third deﬁnes input pattern selection operators, such that the outputs of the model based on the selected input patterns has large mutual informaton with the original model outputs (Chen et al., 2018;
Revised: The ﬁrst line of work use gradients of the outputs with respect to inputs to identify the salient features in the inputs (Simonyan et al., 2013; Springenberg et al., 2014); The second approximates the model with simple interpretable models, such as locally additive models (Bach et al., 2015; Kindermans et al., 2016; Ribeiro et al., 2016; Lundberg & Lee, 2017); the third deﬁnes input pattern selection operators, such that the outputs of the model based on the selected input patterns has large mutual informaton with the original model outputs (Chen et al., 2018;

Original: Ying et al., 2019).
Revised: Ying et al., 2019).

Original: The difference of our explainer model compared to GNNExplainer (Ying et al., 2019) is discussed in Sec 3.4.
Revised: The difference of our explainer model compared to GNNExplainer (Ying et al., 2019) is discussed in Sec 3.4.

Original: The explainer we used also share the similar merits as a concurrent work published by Luo et al.
Revised: The explainer we used also shares the similar merits as a concurrent work published by Luo et al.

Original: The main difference lies in the optimization technique, where the stacked Gumbel-softmax is used in Luo et al.
Revised: The main difference lies in the optimization technique, where the stacked Gumbel-softmax is used in Luo et al.

Original: (2020), while we use a differentiable top-k operator that is more stable and easier to train in general.
Revised: (2020), while we use a differentiable top-k operator that is more stable and easier to train in general.

Original: B P ROBABILISITC RELAXATION OF CONSTRAINTS
Revised: B P ROBABILISITC RELAXATION OF CONSTRAINTS

Original: In main paper we mentioned that we are solving the following probabilistic relaxation of the integer programming:
Revised: In main paper we mentioned that we are solving the following probabilistic relaxation of the integer programming:

Original: [Equation]
Revised: [Equation]

Original: where Y ∼ Bernoulli ( p ) ,
Revised: where Y ∼ Bernoulli ( p ) ,

Original: This can be more general for many constrained optimization problems.
Revised: This can be more general for many constrained optimization problems.

Original: For example:
Revised: For example:

Original: MVC: MVC constraint requires that each edge e
Revised: MVC: MVC constraint requires that each edge e

Original: = ( i, j ) has p i + p j ≥ 1 .
Revised: = ( i, j ) has p i + p j ≥ 1 .

Original: In the probabilistic sense, the probability of violating this constraint is (1 − p
Revised: In the probabilistic sense, the probability of violating this constraint is (1 − p

Original: i ) ∗ (1 − p j ) .
Revised: i ) ∗ (1 − p j ) .

Original: Thus the expected penalty would be:
Revised: Thus the expected penalty would be:

Original: [Equation]
Revised: [Equation]

Original: MDS: MDS constraints are
Revised: MDS: MDS constraints are

Original: placed on nodes.
Revised: placed on nodes.

Original: The ILP constraint says p i + (cid:80) j ∈N ( i ) p
Revised: The ILP constraint says p i + (cid:80) j ∈N ( i ) p

Original: j ≥ 1 .
Revised: j ≥ 1 .

Original: Again, we write it in the probability sense, with the expected penalty be:
Revised: Again, we write it in the probability sense, with the expected penalty be:

Original: [Equation]
Revised: [Equation]

Original: In our experiment, we use above expected penalty to handle the hard constraints in unsupervised learning setting.
Revised: In our experiment, we use above expected penalty to handle the hard constraints in unsupervised learning setting.

Original: C D ETAILS OF CHEAP ALGORITHMIC FEATURES
Revised: C D ETAILS OF CHEAP ALGORITHMIC FEATURES

Original: The cheap algorithm conﬁgurations are described in Table 3.
Revised: The cheap algorithm conﬁgurations are described in Table 3.

Original: For unweighted graphs the spanning
Revised: For unweighted graphs the spanning

Original: [Table]
Revised: [Table]

Original: trees are usually not unique, so we can use multiple tree solutions as global features for DAD.
Revised: trees are usually not unique, so we can use multiple tree solutions as global features for DAD.

Original: We use AKPW (Alon et al., 1995) for computing the low-stretch spanning tree for Max-Cut problems.
Revised: We use AKPW (Alon et al., 1995) for computing the low-stretch spanning tree for Max-Cut problems.

Original: C. O BTAINING TREE SOLUTIONS
Revised: C. O BTAINING TREE SOLUTIONS

Original: Note that although these combinatorial optimization on trees are easier than NP-hard, but they are still nontrivial.
Revised: Note that although these combinatorial optimization on trees are easier than NP-hard, but they are still nontrivial.

Original: Here we cover the recursion of dynamic programming for these problems:
Revised: Here we cover the recursion of dynamic programming for these problems:

Original: MVC:
Revised: MVC:

Original: Firstly we pick a random node as root, and traverse the tree in DFS order.
Revised: Firstly we pick a random node as root, and traverse the tree in DFS order.

Original: Suppose we use f [ n, 0] to denote the cover size of the subtree rooted at node n , when the node n is not chosen, and f [ n, 1] to denote the cover size when node n is chosen, then we have:
Revised: Suppose we use f [ n, 0] to denote the cover size of the subtree rooted at node n , when the node n is not chosen, and f [ n, 1] to denote the cover size when node n is chosen, then we have:

Original: [Equation]
Revised: [Equation]

Original: And the ﬁnal cover size is obtained by min { f [ root , 0] , f [ root , 1] } .
Revised: And the ﬁnal cover size is obtained by min { f [ root , 0] , f [ root , 1] } .

Original: MDS:
Revised: MDS:

Original: Firstly we pick a random node as root, and traverse the tree in DFS order.
Revised: Firstly we pick a random node as root, and traverse the tree in DFS order.

Original: Suppose we use f [ n, 0] to denote the dominant set size of the subtree rooted at node n , when the node n is chosen; use f [ n, 1] to denote the situation when n is not picked, but it has been dominated by other nodes; use f [ n, 2] to denote that we don’t care about node n ’s situation, but its subtree should satisfy the constraint.
Revised: Suppose we use f [ n, 0] to denote the dominant set size of the subtree rooted at node n , when the node n is chosen; use f [ n, 1] to denote the situation when n is not picked, but it has been dominated by other nodes; use f [ n, 2] to denote that we don’t care about node n ’s situation, but its subtree should satisfy the constraint.

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: min { f [ c
Revised: min { f [ c

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f [ c
Revised: f [ c

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f
Revised: f

Original: [ n
Revised: [ n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:26) ∞ , child(n)
Revised: (cid:26) ∞ , child(n)

Original: [Equation]
Revised: [Equation]

Original: ∅ f [ n, 2] + min c
Revised: ∅ f [ n, 2] + min c

Original: [Equation]
Revised: [Equation]

Original: child(n)
Revised: child(n)

Original: [ f [ c, 0] − min { f [ c, 0] , f [ c, 1] } ] , otherwise (19) and ﬁnally the dominant set size equals to min { f [ root , 0] , f [ root , 1] } .
Revised: [ f [ c, 0] − min { f [ c, 0] , f [ c, 1] } ] , otherwise (19) and ﬁnally the dominant set size equals to min { f [ root , 0] , f [ root , 1] } .

Original: Max-Cut: Max-Cut on tree is easy when all the edge weights are non-negative, as every tree is a bipartite graph.
Revised: Max-Cut: Max-Cut on tree is easy when all the edge weights are non-negative, as every tree is a bipartite graph.

Original: We can perform two-coloring on this tree to obtain the solution.
Revised: We can perform two-coloring on this tree to obtain the solution.

Original: By ﬂipping the colors afterwards, we can get another equivalent solution.
Revised: By ﬂipping the colors afterwards, we can get another equivalent solution.

Original: Although this solution is simple, it still has nontrivial effect on obtaining the global information, as the 2-coloring solution on tree is the weak 2-coloring on the original graph.
Revised: Although this solution is simple, it still has nontrivial effect on obtaining the global information, as the 2-coloring solution on tree is the weak 2-coloring on the original graph.

Original: It is known that for some problems, such information can help boost the approximation ratio of distributed local algorithms (Sato et al., 2019).
Revised: It is known that for some problems, such information can help boost the approximation ratio of distributed local algorithms (Sato et al., 2019).

Original: D E XPERIMENTAL DETAILS
Revised: D E XPERIMENTAL DETAILS

Original: We run all the experiments on a cluster with 6149 nodes.
Revised: We run all the experiments on a cluster with 6149 nodes.

Original: In each node, there are 64 cores.
Revised: In each node, there are 64 cores.

Original: To obtain the solution for evaluation or supervised training, for each graph per task, we run Gurobi on one node with the time limit as 1 hour.
Revised: To obtain the solution for evaluation or supervised training, for each graph per task, we run Gurobi on one node with the time limit as 1 hour.

Original: As we have three tasks, two graph models, seven graph sizes, and 12 k graph instances in each category, the computational budget is 32 million core hours.
Revised: As we have three tasks, two graph models, seven graph sizes, and 12 k graph instances in each category, the computational budget is 32 million core hours.

Original: In reality, we used around 12 million core hours because only the larger graph tasks can run up to 1 hour.
Revised: In reality, we used around 12 million core hours because only the larger graph tasks can run up to 1 hour.

Original: Compared to that of the exact solution, the core hour consumption of the relaxed LP is neglectable.
Revised: Compared to that of the exact solution, the core hour consumption of the relaxed LP is neglectable.

Original: However, the SDP for Max-CUT is computationally intensive.
Revised: However, the SDP for Max-CUT is computationally intensive.

Original: We used cvxpy to realize it.
Revised: We used cvxpy to realize it.

Original: For the larger graphs, it can take up to 40 mins to resolve one graph with 64 cores.
Revised: For the larger graphs, it can take up to 40 mins to resolve one graph with 64 cores.

Original: It takes around 0.
Revised: It takes around 0.

Original: million core hours to obtain the SDP solutions.
Revised: million core hours to obtain the SDP solutions.

Original: We tuned the hyperparameters of GNN models for each graph category, such as the number of layers and the number of spanning trees, using grid search.
Revised: We tuned the hyperparameters of GNN models for each graph category, such as the number of layers and the number of spanning trees, using grid search.

Original: We run each job up to 24 hours on a 64-core node.
Revised: We run each job up to 24 hours on a 64-core node.

Original: It takes around 5 million core hours to run the jobs on the synthetic datasets and 0.3 million core hours for the real datasets.
Revised: It takes around 5 million core hours to run the jobs on the synthetic datasets and 0.3 million core hours for the real datasets.

Original: In total, it takes us around 17.5 million core hours to run all the jobs.
Revised: In total, it takes us around 17.5 million core hours to run all the jobs.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E M ORE REAL WORLD DATA INFORMATION
Revised: E M ORE REAL WORLD DATA INFORMATION

Original: We collect a set of social network dataset Collab, Twitter, Github and Facebook from Morris et al.
Revised: We collect a set of social network dataset Collab, Twitter, Github and Facebook from Morris et al.

Original: (2020), following Karalias & Loukas (2020).
Revised: (2020), following Karalias & Loukas (2020).

Original: Each dataset comes with a set of graphs, and we split them into training/validation/test with ratio 6:2:2.
Revised: Each dataset comes with a set of graphs, and we split them into training/validation/test with ratio 6:2:2.

Original: Additionally, we collect Memetracker graph from
Revised: Additionally, we collect Memetracker graph from

Original: SNAP (Leskovec & Krevl, 2014) and Optsicom used in Khalil et al. (2017).
Revised: SNAP (Leskovec & Krevl, 2014) and Optsicom used in Khalil et al. (2017).

Original: As these only have a few graphs, we use the same strategy from Khalil et al.
Revised: As these only have a few graphs, we use the same strategy from Khalil et al.

Original: (2017) to generate graphs.
Revised: (2017) to generate graphs.

Original: Speciﬁcally, for
Revised: Speciﬁcally, for

Original: Memetracker, we use widely-adopted Independent Cascade model and sample a diffusion cascade from the full graph with constant set to 7, and consider the largest connected component in the graph as a single graph instance; for Optsicom we keep the graph structures, but perturb the edges weights using Bernoulli distribution (denoted as Optsicom-B) or Gaussian distribution with zero mean and 0.1 standard deviation (denoted as Optsicom-G).
Revised: Memetracker, we use widely-adopted Independent Cascade model and sample a diffusion cascade from the full graph with constant set to 7, and consider the largest connected component in the graph as a single graph instance; for Optsicom we keep the graph structures, but perturb the edges weights using Bernoulli distribution (denoted as Optsicom-B) or Gaussian distribution with zero mean and 0.1 standard deviation (denoted as Optsicom-G).

Original: See Table 4 for the ﬁnal statistics about the datasets.
Revised: See Table 4 for the ﬁnal statistics about the datasets.

Original: For the completeness, we additionally include MVC and MDS results on Collab and Facebook datasets in Table 5.
Revised: For the completeness, we additionally include MVC and MDS results on Collab and Facebook datasets in Table 5.

Original: These datasets are used in Karalias & Loukas (2020) but not very suitable for evaluating MVC or MDS tasks, as it is easy to get exact solution with simple algorithms on thest two datasets.
Revised: These datasets are used in Karalias & Loukas (2020) but not very suitable for evaluating MVC or MDS tasks, as it is easy to get exact solution with simple algorithms on thest two datasets.

Original: F M ORE EXPERIMENTAL RESULTS F. M ORE IN - DISTRIBUTION GENERALIZATION RESULTS
Revised: F M ORE EXPERIMENTAL RESULTS F. M ORE IN - DISTRIBUTION GENERALIZATION RESULTS

Original: [Figure]
Revised: [Figure]

Original: r o x i m a t i o n r a t
Revised: r o x i m a t i o n r a t

Original: i o
Revised: i o

Original: [Figure]
Revised: [Figure]

Original: In main paper we covered the comparison between DAD and other classical algorithms, as well as other types of global GNNs on Barabasi Albert graphs.
Revised: In main paper we covered the comparison between DAD and other classical algorithms, as well as other types of global GNNs on Barabasi Albert graphs.

Original: Here we present the results on Erdos Renyi
Revised: Here we present the results on Erdos Renyi

Original: [Figure]
Revised: [Figure]

Original: MVC-ER Supervised
Revised: MVC-ER Supervised

Original: MDS-ER Supervised
Revised: MDS-ER Supervised

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Supervised
Revised: Supervised

Original: [Figure]
Revised: [Figure]

Original: t i o
Revised: t i o

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: t G u r o b
Revised: t G u r o b

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: graphs, as well as the results with unsupervised learning in ﬁgure 8 and ﬁgure 9.
Revised: graphs, as well as the results with unsupervised learning in ﬁgure 8 and ﬁgure 9.

Original: We can see the proposed DAD consistently beats all other alternatives in all these settings.
Revised: We can see the proposed DAD consistently beats all other alternatives in all these settings.

Original: F. M ORE EXTRAPOLATION RESULTS
Revised: F. M ORE EXTRAPOLATION RESULTS

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Here we show the extrapolation results on Erdos Renyi and Barabasi Albert graphs
Revised: Here we show the extrapolation results on Erdos Renyi and Barabasi Albert graphs

Original: [Figure]
Revised: [Figure]

Original: where DAD is trained with supervion on graphs with up to 300 nodes.
Revised: where DAD is trained with supervion on graphs with up to 300 nodes.

Original: We can see from ﬁgure 10, DAD still extrapolates well to large graphs.
Revised: We can see from ﬁgure 10, DAD still extrapolates well to large graphs.

Original: The unsupervised learning extrapolates in a similar way, as seen in ﬁgure 11.
Revised: The unsupervised learning extrapolates in a similar way, as seen in ﬁgure 11.

Original: One interesting ﬁnding is that, it achieves approximation ratio that is less than 1 on MDS large graphs.
Revised: One interesting ﬁnding is that, it achieves approximation ratio that is less than 1 on MDS large graphs.

Original: This indicates that the Gurobi is not capable of generating high quality solution on large graphs.
Revised: This indicates that the Gurobi is not capable of generating high quality solution on large graphs.

Original: Also for ER graphs, it is harder to extraplate to smaller graphs.
Revised: Also for ER graphs, it is harder to extraplate to smaller graphs.

Original: As we ﬁx the edge probability for different sizes of the graphs, the node degree distribution would shift when graph size changes.
Revised: As we ﬁx the edge probability for different sizes of the graphs, the node degree distribution would shift when graph size changes.

Original: This may prevent GNNs from extrapolating well.
Revised: This may prevent GNNs from extrapolating well.

Original: F. M ORE ABLATION RESULTS
Revised: F. M ORE ABLATION RESULTS

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Here we include more ablation results, on both real world datasets (see Table 7 and Table 6) and synthetic datasets (see ﬁgure 12).
Revised: Here we include more ablation results, on both real world datasets (see Table 7 and Table 6) and synthetic datasets (see ﬁgure 12).

Original: We can see the conclusion is still consistent with our claim in the main paper.
Revised: We can see the conclusion is still consistent with our claim in the main paper.

Original: [Figure]
Revised: [Figure]

Original: MVC-ER Supervised
Revised: MVC-ER Supervised

Original: MDS-ER Supervised
Revised: MDS-ER Supervised

Original: MaxCut-
Revised: MaxCut-

Original: [Figure]
Revised: [Figure]

Original: Supervised
Revised: Supervised

Original: [Figure]
Revised: [Figure]

Original: t i o w .
Revised: t i o w .

Original: r .
Revised: r .

Original: t G
Revised: t G

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: b i
Revised: b i

Original: [Figure]
Revised: [Figure]

Original: MVC-BA Supervised
Revised: MVC-BA Supervised

Original: MDS-BA Supervised
Revised: MDS-BA Supervised

Original: MaxCut-BA Supervised
Revised: MaxCut-BA Supervised

Original: [Figure]
Revised: [Figure]

Original: a t o w . r .
Revised: a t o w . r .

Original: t G u r o b i
Revised: t G u r o b i

Original: DADDAD
Revised: DADDAD

Original: [Figure]
Revised: [Figure]

Original: F.3. A BLATION ON FEATURE DIMENSION
Revised: F.3. A BLATION ON FEATURE DIMENSION

Original: As our algorithmic feature augmented GNNs have more dimensions in the node and edge features, the input layer thus have more parameters than some of the baseline features.
Revised: As our algorithmic feature augmented GNNs have more dimensions in the node and edge features, the input layer thus has more parameters than some of the baseline features.

Original: For a more rigorous comparison, we have padded the baseline features to the same dimension, and thus all the methods would have same neural network capacity.
Revised: For a more rigorous comparison, we have padded the baseline features to the same dimension, and thus all the methods would have same neural network capacity.

Original: Speciﬁcally, for the Raw baseline, we simply pad constant 1 (or edge weights, when applicable) to the same dimension as DAD.
Revised: Speciﬁcally, for the Raw baseline, we simply pad constant 1 (or edge weights, when applicable) to the same dimension as DAD.

Original: For RandFeat baseline, we augment additional random features to match the dimensionality.
Revised: For RandFeat baseline, we augment additional random features to match the dimensionality.

Original: The two variants are denoted as Raw-Multi and RandFeat-Multi .
Revised: The two variants are denoted as Raw-Multi and RandFeat-Multi .

Original: Figure 13 shows the results across three tasks, with either supervised or unsupervised setting, on two types of random graphs.
Revised: Figure 13 shows the results across three tasks, with either supervised or unsupervised setting, on two types of random graphs.

Original: We can see for some cases the RandFeat-Multi would improve a bit over RandFeat , but overall the -Multi variant is comparable as its original counterpart, while being worse than DAD in all cases.
Revised: We can see for some cases the RandFeat-Multi would improve a bit over RandFeat , but overall the -Multi variant is comparable as its original counterpart, while being worse than DAD in all cases.

Original: This conﬁrms that DAD’s gain of performance is not mainly coming from larger networks, but indeed the algorithmic features improve the global information representation quality.
Revised: This conﬁrms that DAD’s gain of performance is not mainly coming from larger networks, but indeed the algorithmic features improve the global information representation quality.

Original: [Figure]
Revised: [Figure]

Original: MVC-ER Supervised
Revised: MVC-ER Supervised

Original: MDS-ER Supervised
Revised: MDS-ER Supervised

Original: Maxcut-ER Supervised
Revised: Maxcut-ER Supervised

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: MVC-BA Supervised
Revised: MVC-BA Supervised

Original: MDS-BA Supervised
Revised: MDS-BA Supervised

Original: Maxcut-BA Supervised
Revised: Maxcut-BA Supervised

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F.3. M ORE TIME - SOLUTION TRADE - OFF PLOTS
Revised: F.3. M ORE TIME - SOLUTION TRADE - OFF PLOTS

Original: We here include the time-solution trade-off plots for all 3 tasks on 2 types of graphs.
Revised: We here include the time-solution trade-off plots for all 3 tasks on 2 types of graphs.

Original: We vary the average size of graphs in { 256 , 512 , 1024 } .
Revised: We vary the average size of graphs in { 256 , 512 , 1024 } .

Original: All the evaluation is done using 16 threads, where the baselines are run using embarrassingly parallel setup ( i.e. , the average runtime is divided by 16), and DAD runs 16 CPU + 1 GPU mixed parallelism.
Revised: All the evaluation is done using 16 threads, where the baselines are run using embarrassingly parallel setup ( i.e. , the average runtime is divided by 16), and DAD runs 16 CPU + 1 GPU mixed parallelism.

Original: From Figure 14 we can see DAD consistently achieves good trade-off between the time and solution quality.
Revised: From Figure 14, we can see DAD consistently achieves good trade-off between the time and solution quality.

Original: G C OMPARISON WITH GNNE XPLAINER
Revised: G C OMPARISON WITH GNNE XPLAINER

Original: In this section, we compare the explanation quality against the GNNExplainer.
Revised: In this section, we compare the explanation quality against the GNNExplainer.

Original: With out loss of generality, we carry out the explanation experiments on the GNN that is supervised trained for BA graphs with 32-40 nodes, for MVC task.
Revised: Without loss of generality, we carry out the explanation experiments on the GNN that is supervised trained for BA graphs with 32-40 nodes, for MVC task.

Original: For GNNExplainer, we follow the implementation from pytorch geometric.
Revised: For GNNExplainer, we follow the implementation from pytorch geometric.

Original: We set the entropy regularizer coefﬁcient to be 1, and tune the coefﬁcient for the sparsity regularizer in { 0 . 1 , 1 . 0 , 10 . 0 } .
Revised: We set the entropy regularizer coefﬁcient to be 1, and tune the coefﬁcient for the sparsity regularizer in { 0 . 1 , 1 . 0 , 10 . 0 } .

Original: We learn the node and edge masks using Adam optimizer, for at most 3000 gradient update steps.
Revised: We learn the node and edge masks using Adam optimizer, for at most 3000 gradient update steps.

Original: We visualize the explanation provided by DAD and different conﬁgurations of GNNExplainer in
Revised: We visualize the explanation provided by DAD and different conﬁgurations of GNNExplainer in

Original: Figure 15.
Revised: Figure 15.

Original: In the ﬁgure, the node with red boundary color is the target node to be explained.
Revised: In the ﬁgure, the node with red boundary color is the target node to be explained.

Original: For
Revised: For

Original: DAD, we use blue color to indicate the selected node or edge, and gray color to indicate the unselected ones.
Revised: DAD, we use blue color to indicate the selected nodes or edges, and gray color to indicate the un-selected ones.

Original: We ﬁx the explanation budget to be 5 nodes and 10 edges for DAD.
Revised: We ﬁx the explanation budget to be 5 nodes and 10 edges for DAD.

Original: For GNNExplainer, the darkness of the color indicates the score of the soft selection.
Revised: For GNNExplainer, the darkness of the color indicates the score of the soft selection.

Original: We can see all  explainers can at least select the target node as  important node when explaining the target node.
Revised: We can see all the explainers can at least select the target node as the important node when explaining the target node.

Original: This at least shows that the explainers are doing reasonable jobs.
Revised: This at least shows that the explainers are doing reasonable jobs.

Original: We can also see from the ﬁgure that, DAD can offer some node degree based explanations for the ﬁrst and second rows, while GNNExplainer failed to do so, as it either has only one edge with outstanding score, or has a hard time balancing the number of edges with the explanation ﬁtness.
Revised: We can also see from the ﬁgure that, DAD can offer some node degree-based explanations for the ﬁrst and second rows, while GNNExplainer failed to do so, as it either has only one edge with outstanding score, or has a hard time balancing the number of edges with the explanation ﬁtness.

Original: Also from the last row we can see DAD is able to select some disconnected anchor nodes (see section 5.4) for explanation, while GNNExplainer needs either too large or too small chunk of connected subgraph.
Revised: Also, from the last row, we can see DAD is able to select some disconnected anchor nodes (see section 5.4) for explanation while GNNExplainer needs either too large or too small chunk of connected subgraphs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]