Original: R ETHINKING G RAPH L OTTERY T ICKETS : G RAPH S PARSITY M ATTERS
Revised: R ETHINKING G RAPH L OTTERY T ICKETS : G RAPH S PARSITY M ATTERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Lottery Ticket Hypothesis (LTH) claims the existence of a winning ticket (i.e., a properly pruned sub-network together with original weight initialization) that can achieve competitive performance to the original dense network.
Revised: Lottery Ticket Hypothesis (LTH) claims the existence of a winning ticket (i.e., a properly pruned sub-network together with original weight initialization) that can achieve competitive performance to the original dense network.

Original: A recent work, called UGS, extended LTH to prune graph neural networks (GNNs) for effectively accelerating GNN inference.
Revised: A recent work, called UGS, extended LTH to prune graph neural networks (GNNs) for effectively accelerating GNN inference.

Original: UGS simultaneously prunes the graph adjacency matrix and the model weights using the same masking mechanism, but since the roles of the graph adjacency matrix and the weight matrices are very different, we find that their sparsifications lead to different performance characteristics.
Revised: UGS simultaneously prunes the graph adjacency matrix and the model weights using the same masking mechanism, but since the roles of the graph adjacency matrix and the weight matrices are very different, we find that their sparsifications lead to different performance characteristics.

Original: Specifically, we find that the performance of a sparsified GNN degrades significantly when the graph sparsity goes beyond a certain extent.
Revised: Specifically, we find that the performance of a sparsified GNN degrades significantly when the graph sparsity goes beyond a certain extent.

Original: Therefore, we propose two techniques to improve GNN performance when the graph sparsity is high.
Revised: Therefore, we propose two techniques to improve GNN performance when the graph sparsity is high.

Original: First,
Revised: First,

Original: UGS prunes the adjacency matrix using a loss formulation which, however, does not properly involve all elements of the adjacency matrix; in contrast, we add a new auxiliary loss head to better guide the edge pruning by involving the entire adjacency matrix.
Revised: UGS prunes the adjacency matrix using a loss formulation which, however, does not properly involve all elements of the adjacency matrix; in contrast, we add a new auxiliary loss head to better guide the edge pruning by involving the entire adjacency matrix.

Original: Second, by regarding unfavorable graph sparsification as adversarial data perturbations, we formulate the pruning process as a min-max optimization problem to gain the robustness of lottery tickets when the graph sparsity is high.
Revised: Second, by regarding unfavorable graph sparsification as adversarial data perturbations, we formulate the pruning process as a min-max optimization problem to gain the robustness of lottery tickets when the graph sparsity is high.

Original: We further investigate the question: Can the “retrainable” winning ticket of a GNN be also effective for graph transferring learning?
Revised: We further investigate the question: Can the “retrainable” winning ticket of a GNN be also effective for graph transferring learning?

Original: We call it the transferable graph lottery ticket (GLT) hypothesis.
Revised: We call it the transferable graph lottery ticket (GLT) hypothesis.

Original: Extensive experiments were conducted which demonstrate the superiority of our proposed sparsification method over UGS, and which empirically verified our transferable GLT hypothesis.
Revised: Extensive experiments were conducted which demonstrate the superiority of our proposed sparsification method over UGS, and which empirically verified our transferable GLT hypothesis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) Kipf & Welling (2017); Hamilton et al.
Revised: Graph Neural Networks (GNNs) Kipf & Welling (2017); Hamilton et al.

Original: (2017) have demonstrated state-of-the-art performance on various graph-based learning tasks.
Revised: (2017) have demonstrated state-of-the-art performance on various graph-based learning tasks.

Original: However, large graph size and over-parameterized network layers are factors that limit the scalability of GNNs, causing high training cost, slow inference speed, and large memory consumption.
Revised: However, large graph size and over-parameterized network layers are factors that limit the scalability of GNNs, causing high training cost, slow inference speed, and large memory consumption.

Original: Recently, Lottery Ticket Hypothesis (LTH) Frankle & Carbin (2019) claims that there exists properly pruned sub-networks together with original weight initialization that can be retrained to achieve comparable performance to the original large deep neural networks.
Revised: Recently, Lottery Ticket Hypothesis (LTH) Frankle & Carbin (2019) claims that there exists properly pruned sub-networks together with original weight initialization that can be retrained to achieve comparable performance to the original large deep neural networks.

Original: LTH has recently been extended to GNNs by Chen et al.
Revised: LTH has recently been extended to GNNs by Chen et al.

Original: (2021b), which proposes a unified GNN sparsification (UGS) framework that simultaneously prunes the graph adjacency matrix and the model weights to accelerate GNN inference on large graphs.
Revised: (2021b), which proposes a unified GNN sparsification (UGS) framework that simultaneously prunes the graph adjacency matrix and the model weights to accelerate GNN inference on large graphs.

Original: Specifically, two differentiable masks m g and m θ are applied to the adjacency matrix A and the model weights Θ , respectively, during end-to-end training by element-wise product.
Revised: Specifically, two differentiable masks m g and m θ are applied to the adjacency matrix A and the model weights Θ , respectively, during end-to-end training by element-wise product.

Original: After training, lowest-magnitude elements in m g and m θ are set to zero w.r.t.
Revised: After training, lowest-magnitude elements in m g and m θ are set to zero w.r.t.

Original: pre-defined ratios p g and p θ , which basically eliminates low-scored edges and weights, respectively.
Revised: pre-defined ratios p g and p θ , which basically eliminates low-scored edges and weights, respectively.

Original: The weight parameters are then rewound to their original initialization, and this pruning process is repeated until pre-defined sparsity levels are reached, i.e., graph sparsity
Revised: The weight parameters are then rewound to their original initialization, and this pruning process is repeated until pre-defined sparsity levels are reached, i.e., graph sparsity

Original: [Equation]
Revised: [Equation]

Original: and weight sparsity

Original: [Equation]

Original: where ∥ .
Revised: where ∥ .

Original: ∥ 0 is the L 0 norm counting the number of non-zero elements.
Revised: ∥ 0 is the L 0 norm counting the number of non-zero elements.

Revised: Intuitively, UGS simply extends the basic parameter-masking algorithm of Frankle & Carbin (2019) for identifying winning tickets to also mask and remove graph edges.

Revised: However, our empirical study

Original: [Figure]
Revised: [Figure]

Original: Intuitively, UGS simply extends the basic parameter-masking algorithm of Frankle &
Revised: (a) Performance Study of UGS Variants

Original: Carbin (2019) for identifying winning tickets to also mask and remove graph edges.
Revised: [Figure]

Original: However, our empirical study finds that the performance of a sparsified GNN degrades significantly when the graph sparsity goes beyond a certain level, while it is relatively insensitive to weight sparsification.
Revised: finds that the performance of a sparsified GNN degrades significantly when the graph sparsity goes beyond a certain level, while it is relatively insensitive to weight sparsification.

Original: Specifically, we compare UGS with its two variants: (1) “Only weight,” which does not conduct graph sparsification, and (2) “80% edges,” which stops pruning edges as soon as the graph sparsity is increased to 20% or above.
Revised: Specifically, we compare UGS with its two variants: (1) “Only weight,” which does not conduct graph sparsification, and (2) “80% edges,” which stops pruning edges as soon as the graph sparsity is increased to 20% or above.

Original: Figure 1 shows the performance comparison of UGS and the two variants on the Cora dataset Chen et al.
Revised: Figure 1(a) shows the performance comparison of UGS and the two variants on the Cora dataset Chen et al.

Original: (2021b), where we can see that the accuracy of UGS (the red line) collapses when the graph sparsity becomes larger than 25%, while the two variants do not suffer from such a significant performance degradation since neither of them sparsifies the edges beyond 20%.
Revised: (2021b), where we can see that the accuracy of UGS (the red line) collapses when the graph sparsity becomes larger than 25%, while the two variants do not suffer from such a significant performance degradation since neither of them sparsifies the edges beyond 20%.

Original: Clearly, the performance of GNNs is vulnerable to graph sparsification: removing certain edges tends to undermine the underlying structure of the graph, hampering message passing along edges.
Revised: Clearly, the performance of GNNs is vulnerable to graph sparsification: removing certain edges tends to undermine the underlying structure of the graph, hampering message passing along edges.

Original: [Figure]

Original: In this paper, we propose two techniques to improve GNN performance when the graph sparsity is high.
Revised: In this paper, we propose two techniques to improve GNN performance when the graph sparsity is high.

Original: The first technique is based on the observation that in UGS, only a fraction of the adjacency matrix elements (i.e., graph edges) are involved in loss calculation.
Revised: The first technique is based on the observation that in UGS, only a fraction of the adjacency matrix elements (i.e., graph edges) are involved in loss calculation.

Original: As an illustration, consider the semi-supervised node classification task shown in Figure 2 where the nodes in yellow are labeled, and we assume that a GNN with 2 graph convolution layers is used so only nodes within two hops from the yellow nodes are involved in the training process, which are highlighted in green.
Revised: As an illustration, consider the semi-supervised node classification task shown in Figure 1(b) where the nodes in yellow are labeled, and we assume that a GNN with 2 graph convolution layers is used so only nodes within two hops from the yellow nodes are involved in the training process, which are highlighted in green.

Original: Note that the gray edges in Figure 2 are not involved in loss calculation, i.e., no message passing happens along the dashed edges so their corresponding mask elements get zero gradients during backpropagation.
Revised: Note that the gray edges in Figure 1(b) are not involved in loss calculation, i.e., no message passing happens along the dashed edges so their corresponding mask elements get zero gradients during backpropagation.

Original: On the Cora dataset, we find that around 50% edges are in such a situation, leaving the values of their corresponding mask elements unchanged throughout the entire training process.
Revised: On the Cora dataset, we find that around 50% edges are in such a situation, leaving the values of their corresponding mask elements unchanged throughout the entire training process.

Original: After checking the source code of UGS, we find that it initializes these mask elements by adding a random noise, so the ordering of the edge mask-scores is totally determined by this initial mask randomization rather than the graph topology.
Revised: After checking the source code of UGS, we find that it initializes these mask elements by adding a random noise, so the ordering of the edge mask-scores is totally determined by this initial mask randomization rather than the graph topology.

Original: As a result, the removal of low-scored edges tends to be random in later iterations of UGS, causing performance collapse as some important “dashed” edges are removed.
Revised: As a result, the removal of low-scored edges tends to be random in later iterations of UGS, causing performance collapse as some important “dashed” edges are removed.

Original: To address this problem, we add a new auxiliary loss head to better guide the edge pruning by involving the entire adjacency matrix.
Revised: To address this problem, we add a new auxiliary loss head to better guide the edge pruning by involving the entire adjacency matrix.

Original: Specifically, this loss head uses a novel loss function that measures the inter-class separateness of nodes with Wasserstein distance (WD).
Revised: Specifically, this loss head uses a novel loss function that measures the inter-class separateness of nodes with Wasserstein distance (WD).

Original: For each class, we calculate the WD between (1) the set of nodes that are predicted to be in the class and (2) the set of other nodes.
Revised: For each class, we calculate the WD between (1) the set of nodes that are predicted to be in the class and (2) the set of other nodes.

Original: By minimizing WD for all classes, we maximize the difference between the extracted node features of different classes.
Revised: By minimizing WD for all classes, we maximize the difference between the extracted node features of different classes.

Original: Now that this loss function involves all nodes, all elements in the graph mask m g will now have gradient during backpropagation.
Revised: Now that this loss function involves all nodes, all elements in the graph mask m g will now have gradient during backpropagation.

Original: Our second technique is based on adversarial perturbation, which is widely used to improve the robustness of deep neural networks Wong et al.
Revised: Our second technique is based on adversarial perturbation, which is widely used to improve the robustness of deep neural networks Wong et al.

Original: To improve the robustness of the graph lottery tickets (i.e., the pruned GNN subnetworks) when graph sparsity is high, we regard unfavorable graph sparsification as an adversarial data perturbation and formulate the pruning process as a min-max optimization problem.
Revised: To improve the robustness of the graph lottery tickets (i.e., the pruned GNN subnetworks) when graph sparsity is high, we regard unfavorable graph sparsification as an adversarial data perturbation and formulate the pruning process as a min-max optimization problem.

Original: Specifically, a minimizer seeks to update both the weight parameters Θ and its mask m θ against a maximizer that aims to perturb the graph mask m g .
Revised: Specifically, a minimizer seeks to update both the weight parameters Θ and its mask m θ against a maximizer that aims to perturb the graph mask m g .

Original: By performing projected gradient ascent on the graph mask, we are essentially using adversarial perturbations to significantly improve the robustness of our graph lottery tickets against the graph sparsity.
Revised: By performing projected gradient ascent on the graph mask, we are essentially using adversarial perturbations to significantly improve the robustness of our graph lottery tickets against the graph sparsity.

Original: We further investigate the question: Can we use the obtained winning ticket of a GNN for graph transfer learning?
Revised: We further investigate the question: Can we use the obtained winning ticket of a GNN for graph transfer learning?

Original: Studying this problem is particularly interesting since the “retrainability” of a winning ticket (i.e., a pruned sub-network together with original weight initialization) on the same task is the most distinctive property of Lottery Ticket Hypothesis (LTH): many works Liu et al.
Revised: Studying this problem is particularly interesting since the “retrainability” of a winning ticket (i.e., a pruned sub-network together with original weight initialization) on the same task is the most distinctive property of Lottery Ticket Hypothesis (LTH): many works Liu et al.

Original: ; Frankle & Carbin (2019) have verified the importance of winning ticket initialization and stated that the winning ticket initialization might land in a region of the loss landscape that is particularly amenable to optimization.
Revised: ; Frankle & Carbin (2019) have verified the importance of winning ticket initialization and stated that the winning ticket initialization might land in a region of the loss landscape that is particularly amenable to optimization.

Original: Here, we move one step further in the transfer learning setting, to investigate if the winning ticket identified on a source task also works on a target task.
Revised: Here, we move one step further in the transfer learning setting, to investigate if the winning ticket identified on a source task also works on a target task.

Original: The answer is affirmative through our empirical study, and we call it the transferable graph LTH.
Revised: The answer is affirmative through our empirical study, and we call it the transferable graph LTH.

Original: The contributions of this paper are summarized as follows:
Revised: The contributions of this paper are summarized as follows:

Original: • We design a new auxiliary loss function to guide the edges pruning for identifying graph lottery tickets.
Revised: • We design a new auxiliary loss function to guide the edges pruning for identifying graph lottery tickets.

Original: Our new auxiliary loss is able to address the issue of random edge pruning in UGS.
Revised: Our new auxiliary loss is able to address the issue of random edge pruning in UGS.

Original: • We formalize our sparsification algorithm as a min-max optimization process to gain the robustness of lottery tickets to the graph sparsity.
Revised: • We formalize our sparsification algorithm as a min-max optimization process to gain the robustness of lottery tickets to the graph sparsity.

Original: • We empirically investigated and verified the transferable graph lottery ticket hypotheses.
Revised: • We empirically investigated and verified the transferable graph lottery ticket hypotheses.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: This section defines our notations, reviews the concept of GNNs, defines graph lottery tickets, and reports our empirical study of UGS’s random edge pruning issue.
Revised: This section defines our notations, reviews the concept of GNNs, defines graph lottery tickets, and reports our empirical study of UGS’s random edge pruning issue.

Original: Notations and GNN Formulation.
Revised: Notations and GNN Formulation.

Original: We consider an undirected graph G =
Revised: We consider an undirected graph G =

Original: ( V , E ) where V and E are the sets of nodes and edges of G , respectively.
Revised: ( V , E ) where V and E are the sets of nodes and edges of G , respectively.

Original: Each node v i ∈ V has a feature vector x i ∈ R F , where F is the number of node features.
Revised: Each node v i ∈ V has a feature vector x i ∈ R F , where F is the number of node features.

Original: We use X ∈ R N × F to denote the feature matrix of the whole graph G , where N = |V| is the number of nodes, and X stacks x i of all nodes v i as its rows.
Revised: We use X ∈ R N × F to denote the feature matrix of the whole graph G , where N = |V| is the number of nodes, and X stacks x i of all nodes v i as its rows.

Original: Let A ∈ R N × N be the adjacency matrix of G , i.e., A i,j = 1 if there is an edge e i,j ∈ E between nodes v i and v j , and A i,j = 0 otherwise.
Revised: Let A ∈ R N × N be the adjacency matrix of G , i.e., A i,j = 1 if there is an edge e i,j ∈ E between nodes v i and v j , and A i,j = 0 otherwise.

Original: Graph Convolutional Networks (GCN) Kipf & Welling (2017) is the most popular and widely adopted GNN model.
Revised: Graph Convolutional Networks (GCN) Kipf & Welling (2017) is the most popular and widely adopted GNN model.

Original: Without loss of generality, we consider a GCN with two graph convolution layers, which is formulated as follows:
Revised: Without loss of generality, we consider a GCN with two graph convolution layers, which is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, W (0) is an input-to-hidden weight matrix and W (1) is a hidden-to-output weight matrix.
Revised: Here, W (0) is an input-to-hidden weight matrix and W (1) is a hidden-to-output weight matrix.

Original: The softmax activation function is applied row-wise and σ ( · ) = max(0 , · ) is the ReLU activation function.
Revised: The softmax activation function is applied row-wise and σ ( · ) = max(0 , · ) is the ReLU activation function.

Original: The total number of classes is C .
Revised: The total number of classes is C .

Original: The adjacency matrix with self-connections ˜
Revised: The adjacency matrix with self-connections ˜

Original: A =
Revised: A =

Original: A + I N is normalized by ˆ A =
Revised: A + I N is normalized by ˆ A =

Original: ˜ D − 12 ˜
Revised: ˜ D − 12 ˜

Original: A ˜ D − 12 where ˜ D is the degree matrix of ˜ A .
Revised: A ˜ D − 12 where ˜ D is the degree matrix of ˜ A .

Original: For the semi-supervised node classification tasks, the cross-entropy error over labeled nodes is given by:
Revised: For the semi-supervised node classification tasks, the cross-entropy error over labeled nodes is given by:

Original: [Equation]
Revised: [Equation]

Original: where Y L is the set of node indices for those nodes with labels, Y l is the one-hot encoding (row) vector of node v l ’s label.
Revised: where Y L is the set of node indices for those nodes with labels, Y l is the one-hot encoding (row) vector of node v l ’s label.

Original: Graph Lottery Tickets.
Revised: Graph Lottery Tickets.

Original: If a sub-network of a GNN with the original initialization trained on a sparsified graph has a comparable performance to the original GNN trained on the full graph in terms of test accuracy, then the GNN subnetwork along with the sparsified graph is defined as a graph lottery ticket (GLT)
Revised: If a sub-network of a GNN with the original initialization trained on a sparsified graph has a comparable performance to the original GNN trained on the full graph in terms of test accuracy, then the GNN subnetwork along with the sparsified graph is defined as a graph lottery ticket (GLT)

Original: Chen et al.
Revised: Chen et al.

Original: (2021b).
Revised: (2021b).

Original: Different from the general LTH literature, GLT consists of three elements: (1) a sparsified graph obtained by pruning some edges in G , (2) a GNN sub-network and (3) the initialization of its learnable parameters.
Revised: Different from the general LTH literature, GLT consists of three elements: (1) a sparsified graph obtained by pruning some edges in G , (2) a GNN sub-network and (3) the initialization of its learnable parameters.

Original: Without loss of generality, let us consider a 2-layer GCN denoted by f ( X , Θ 0 ) where Θ 0 = { W (0)0 , W (1)0 } is the initial weight parameters.
Revised: Without loss of generality, let us consider a 2-layer GCN denoted by f ( X , Θ 0 ) where Θ 0 = { W (0)0 , W (1)0 } is the initial weight parameters.

Original: Then, the task is to find two masks m g and m θ such that the sub-network f
Revised: Then, the task is to find two masks m g and m θ such that the sub-network f

Original: ( X , m θ ⊙ Θ 0 ) along with the sparsified graph { m g ⊙ A , X } can be trained to a comparable accuracy as f ( X , Θ 0 ) .
Revised: ( X , m θ ⊙ Θ 0 ) along with the sparsified graph { m g ⊙ A , X } can be trained to a comparable accuracy as f ( X , Θ 0 ) .

Original: UGS Analysis.
Revised: UGS Analysis.

Original: LTH algorithms including UGS Chen et al.
Revised: LTH algorithms including UGS Chen et al.

Original: (2021b) sort the elements in a mask by magnitude and “zero out” the smallest ones for sparsification.
Revised: (2021b) sort the elements in a mask by magnitude and “zero out” the smallest ones for sparsification.

Original: To find and remove the insignificant connections and weights in the graph and the GNN model, UGS updates two masks m g and m θ by backpropagation.
Revised: To find and remove the insignificant connections and weights in the graph and the GNN model, UGS updates two masks m g and m θ by backpropagation.

Original: However, recall from Figure 2 that UGS suffers from the issue that only a fraction of edges in G are related to the training loss as given by Eq (2) for semi-supervised node
Revised: However, recall from Figure 1(b) that UGS suffers from the issue that only a fraction of edges in G are related to the training loss as given by Eq (2) for semi-supervised node

Original: [Table]
Revised: [Table]

Original: classification.
Revised: classification.

Original: Table 1 shows (1) the number of nodes in the training v.s. entire dataset and (2) the percentage of edges with gradients during backpropagation for three graph datasets:
Revised: Table 1 shows (1) the number of nodes in the training v.s. entire dataset and (2) the percentage of edges with gradients during backpropagation for three graph datasets:

Original: Cora, Citeseer and PubMed.
Revised: Cora, Citeseer and PubMed.

Original: We can see that a significant percentage of edges (up to 91% as in
Revised: We can see that a significant percentage of edges (up to 91% as in

Original: PubMed) are not related to the training loss.
Revised: PubMed) are not related to the training loss.

Original: We carefully examined the source code of UGS and found that the pruning process for these edges tends to be random since each element in the graph mask is initialized by adding a random noise to its magnitude.
Revised: We carefully examined the source code of UGS and found that the pruning process for these edges tends to be random since each element in the graph mask is initialized by adding a random noise to its magnitude.

Original: As we have seen in Figure 1, the accuracy of UGS collapses when the graph sparsity becomes high.
Revised: As we have seen in Figure 1(a), the accuracy of UGS collapses when the graph sparsity becomes high.

Original: It is crucial to properly guide the mask learning for those edges not related to the training loss in Eq (2) to alleviate the performance degradation.
Revised: It is crucial to properly guide the mask learning for those edges not related to the training loss in Eq (2) to alleviate the performance degradation.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Auxiliary Loss Function.
Revised: Auxiliary Loss Function.

Original: Since L 0 as given by Eq (2) does not involve all edges for backpropagation, we hereby design another auxiliary loss function L 1 to better guide the edge pruning together with L 0 .
Revised: Since L 0 as given by Eq (2) does not involve all edges for backpropagation, we hereby design another auxiliary loss function L 1 to better guide the edge pruning together with L 0 .

Original: Given GNN output Z and a class c , we separate potential nodes of class c and the other nodes by:
Revised: Given GNN output Z and a class c , we separate potential nodes of class c and the other nodes by:

Original: Z c = { z i ∈ rows ( Z ) | argmax ( z i )
Revised: Z c = { z i ∈ rows ( Z ) | argmax ( z i )

Original: =
Revised: =

Original: c } and Z ¯ c = { z i ∈ rows ( Z ) | argmax ( z i )
Revised: c } and Z ¯ c = { z i ∈ rows ( Z ) | argmax ( z i )

Original: ̸ = c } , (3) for c ∈ { 1 , 2 , · · · , C } .
Revised: ̸ = c } , (3) for c ∈ { 1 , 2 , · · · , C } .

Original: Since z i of those i ∈ Y L are properly guided by L 0 , we can use argmax ( z i ) =
Revised: Since z i of those i ∈ Y L are properly guided by L 0 , we can use argmax ( z i ) =

Original: c to filter out the potential nodes that belong to class c .
Revised: c to filter out the potential nodes that belong to class c .

Original: For each class c , we then calculate the
Revised: For each class c , we then calculate the

Original: Wasserstein distance (WD) between Z c and Z ¯ c :
Revised: Wasserstein distance (WD) between Z c and Z ¯ c :

Original: [Equation]
Revised: [Equation]

Original: where Π( Z c , Z ¯ c ) denotes the set of all joint distributions
Revised: where Π( Z c , Z ¯ c ) denotes the set of all joint distributions

Original: π ( z i , z j ) whose marginals are, respectively,
Revised: π ( z i , z j ) whose marginals are, respectively,

Original: Z c and Z ¯ c :
Revised: Z c and Z ¯ c :

Original: [Equation]
Revised: [Equation]

Original: Wasserstein distance (WD) Villani (2009) is widely used to measure the distance between two distributions, especially under the vanishing overlap of two distributions.
Revised: Wasserstein distance (WD) Villani (2009) is widely used to measure the distance between two distributions, especially under the vanishing overlap of two distributions.

Original: Intuitively, it is the cost of the optimal plan for transporting one distribution to match another distribution: in Eq (5), Π( Z c , Z ¯ c ) is the set of valid transport plans, where P is a coupling matrix with each element P ij representing how much probability mass from point z
Revised: Intuitively, it is the cost of the optimal plan for transporting one distribution to match another distribution: in Eq (5), Π( Z c , Z ¯ c ) is the set of valid transport plans, where P is a coupling matrix with each element P ij representing how much probability mass from point z

Original: i ∈
Revised: i ∈

Original: Z c is assigned to a point z j ∈ Z ¯ c ; and 1 is an all-one vector.
Revised: Z c is assigned to a point z j ∈ Z ¯ c ; and 1 is an all-one vector.

Original: We consider Z c and Z ¯ c as the distribution of class c and other classes, respectively.
Revised: We consider Z c and Z ¯ c as the distribution of class c and other classes, respectively.

Original: Equation (4) can be approximated by a smooth convex optimization problem with an entropic regularization term:
Revised: Equation (4) can be approximated by a smooth convex optimization problem with an entropic regularization term:

Original: [Equation]
Revised: [Equation]

Original: where D is the pairwise Euclidean distance matrix between points of Z c and Z ¯ c , and ⟨· , ·⟩ is the
Revised: where D is the pairwise Euclidean distance matrix between points of Z c and Z ¯ c , and ⟨· , ·⟩ is the

Original: Frobenius inner product.
Revised: Frobenius inner product.

Original: Minimizing Eq (6) is solved using Sinkhorn iterations Cuturi (2013), which form a sequence of linear operations straightforward for backpropagation.
Revised: Minimizing Eq (6) is solved using Sinkhorn iterations Cuturi (2013), which form a sequence of linear operations straightforward for backpropagation.

Original: Ideally, we want the node representations in one class to be very different from those in other classes so that the classes can be easily distinguished.
Revised: Ideally, we want the node representations in one class to be very different from those in other classes so that the classes can be easily distinguished.

Original: By maximizing WD ( Z c , Z ¯ c ) for all classes c , we maximize the difference between GNN outputs for different classes.
Revised: By maximizing WD ( Z c , Z ¯ c ) for all classes c , we maximize the difference between GNN outputs for different classes.

Original: Formally, the auxiliary loss function to minimize is given by:
Revised: Formally, the auxiliary loss function to minimize is given by:

Original: [Equation]
Revised: [Equation]

Original: To explore the relationship between unsupervised loss L 1 with the supervised loss L 0 (i.e., Eq (2)), we train a 2-layer GCN on Cora with L 0 as the only loss function, and Figure 3 plots the value of Las well as the corresponding value of (cid:80) c ∈C WD ( Z c , Z ¯ c ) (abbr.
Revised: To explore the relationship between unsupervised loss L 1 with the supervised loss L 0 (i.e., Eq (2)), we train a 2-layer GCN on Cora with L 0 as the only loss function, and Figure 2 plots the value of Las well as the corresponding value of (cid:80) c ∈C WD ( Z c , Z ¯ c ) (abbr.

Original: WD)
Revised: WD)

Original: during the training process.
Revised: during the training process.

Original: We can see that as training loss L 0 decreases, the value of WD increases.
Revised: We can see that as training loss L 0 decreases, the value of WD increases.

Original: The curves of both L 0 and WD
Revised: The curves of both L 0 and WD

Original: [Figure]
Revised: [Figure]

Original: start to bounce back after 100 epochs.
Revised: start to bounce back after 100 epochs.

Original: Therefore, maximizing WD is consistent with improving classification accuracy.
Revised: Therefore, maximizing WD is consistent with improving classification accuracy.

Original: Note that (cid:80) c ∈C WD ( Z c , Z ¯ c ) is unsupervised and related to all nodes in the training set.
Revised: Note that (cid:80) c ∈C WD ( Z c , Z ¯ c ) is unsupervised and related to all nodes in the training set.

Original: It is natural to use this new metric to guide the pruning of edges that are not directly related to L 0 : if masking an edge decreases
Revised: It is natural to use this new metric to guide the pruning of edges that are not directly related to L 0 : if masking an edge decreases

Original: WD, this edge tends to be important and its removal will decrease the test accuracy of node classification.
Revised: WD, this edge tends to be important and its removal will decrease the test accuracy of node classification.

Original: Our sparsification method uses a semi-supervised loss that combines L 0 and L 1 (or, WD):
Revised: Our sparsification method uses a semi-supervised loss that combines L 0 and L 1 (or, WD):

Original: L =
Revised: L =

Original: L 0 + λ L 1 .
Revised: L 0 + λ L 1 .

Original: Sparsification.
Revised: Sparsification.

Original: To prune the edges and weights while retaining robustness, we formulate GNN sparsification as a min-max problem given by:
Revised: To prune the edges and weights while retaining robustness, we formulate GNN sparsification as a min-max problem given by:

Original: [Equation]
Revised: [Equation]

Original: where the outer minimization seeks to train the model weights and their masks to minimize the classification error, and the inner maximization aims to adversarially perturb the graph mask.
Revised: where the outer minimization seeks to train the model weights and their masks to minimize the classification error, and the inner maximization aims to adversarially perturb the graph mask.

Original: Let us use L ( Θ , m θ , m g ) to denote ( L 0 + λ L 1 ) , then the optimization problem in Eq (8) can be solved by alternating the following two steps as specified by (i) Eq (9) and (ii) Eqs (10) & (11).
Revised: Let us use L ( Θ , m θ , m g ) to denote ( L 0 + λ L 1 ) , then the optimization problem in Eq (8) can be solved by alternating the following two steps as specified by (i) Eq (9) and (ii) Eqs (10) & (11).

Original: Specifically, the inner maximization perturbs the graph mask at ( t + 1) th iteration as follows:
Revised: Specifically, the inner maximization perturbs the graph mask at ( t + 1) th iteration as follows:

Original: [Equation]
Revised: [Equation]

Original: where P C [ m ] =
Revised: where P C [ m ] =

Original: arg min m ′ ∈C ∥ m ′ − m ∥ 2 F is a differentiable projection operation that projects the mask values onto [0 , 1] , and η 1 > 0 is a proper learning rate.
Revised: arg min m ′ ∈C ∥ m ′ − m ∥ 2 F is a differentiable projection operation that projects the mask values onto [0 , 1] , and η 1 > 0 is a proper learning rate.

Original: While the second step for outer minimization is given by:
Revised: While the second step for outer minimization is given by:

Original: [Equation]
Revised: [Equation]

Original: where the last term is an implicit gradient obtained by chain rule over m ( t +1) g as given by Eq (9).
Revised: where the last term is an implicit gradient obtained by chain rule over m ( t +1) g as given by Eq (9) [Equation] Algorithm [Equation]

Original: We follow UGS’s framework to repeatedly train, prune, and reset a GNN with initial weights after each round until the desired sparsity levels s g and s θ are reached.
Revised: m (0) θ [Equation] Θ (0) = { W (0)0 , W (

Revised: Compute m ( t +1) θ with Eq (10) 6:

Revised: Compute Θ ( t +1) with Eq (11) 7: end for 8:

Revised: [Equation]

Revised: Set p

Revised: g % of the lowest-scored values in m g to 0 and set the others to10:

Revised: Set p θ % of the lowest-scored values in m θ to 0 and set the others to11: end while

Revised: We follow UGS’s framework to repeatedly train, prune, and reset a GNN with initial weights after each round until the desired sparsity levels s g and s θ are reached.

Revised: Algorithm 1 illustrates our iterative pruning process.

Original: At each round once the model is trained and masks are computed, we set p g fraction of the lowest-scored values in m g to 0 and set the others to 1.
Revised: At each round once the model is trained and masks are computed, we set p g fraction of the lowest-scored values in m g to 0 and set the others to 1.

Original: Likewise, (1 − p θ ) fraction of the weights in Θ will survive to the next round.
Revised: Likewise, (1 − p θ ) fraction of the weights in Θ will survive to the next round.

Original: Finally, we retrain the obtained sub-network using only L 0 to report the test accuracy.
Revised: Finally, we retrain the obtained sub-network using only L 0 to report the test accuracy.

Original:  T RANSFER L EARNING WITH G RAPH L OTTERY T ICKETS
Revised:  T RANSFER L EARNING WITH G RAPH L OTTERY T ICKETS

Original: This section explores whether a graph lottery ticket (GLT) can be used for graph transfer learning.
Revised: This section explores whether a graph lottery ticket (GLT) can be used for graph transfer learning.

Original: Specifically, we ask two questions: (1) Can we use the graph winning ticket of a GNN for transfer learning?
Revised: Specifically, we ask two questions: (1) Can we use the graph winning ticket of a GNN for transfer learning?

Original: Here, the graph winning ticket is associated with the originally initialized weight parameters.
Revised: Here, the graph winning ticket is associated with the originally initialized weight parameters.

Original: (2) Can we use the post-trained winning ticket of a GNN for transfer learning?
Revised: (2) Can we use the post-trained winning ticket of a GNN for transfer learning?

Original: Here, “post-trained” means that the winning ticket directly uses the trained weight parameters on the source task to bootstrap the training on the target task. 
Revised: Here, “post-trained” means that the winning ticket directly uses the trained weight parameters on the source task to bootstrap the training on the target task. [Figure]

Original: To answer these two questions, we consider the node classification tasks on two citation networks arXiv and MAG Hu et al.
Revised: To answer these two questions, we consider the node classification tasks on two citation networks arXiv and MAG Hu et al.

Original: In both tasks, each node represents an author and is associated with a 128-d feature vector obtained by averaging the word embeddings of papers that are published by the author.
Revised: In both tasks, each node represents an author and is associated with a 128-d feature vector obtained by averaging the word embeddings of papers that are published by the author.

Original: We train a 3-layer GCN model on arXiv and transfer the model to MAG by replacing the last dense layer.
Revised: We train a 3-layer GCN model on arXiv and transfer the model to MAG by replacing the last dense layer.

Original: Since the graphs of arXiv and MAG are different, we only transfer m θ and Θ from arXiv, and use the original graph of MAG without sparsification.
Revised: Since the graphs of arXiv and MAG are different, we only transfer m θ and Θ from arXiv, and use the original graph of MAG without sparsification.

Original: Figure 4 shows the training accuracy curves on MAG with winning tickets obtained from arXiv.
Revised: Figure 3 shows the training accuracy curves on MAG with winning tickets obtained from arXiv.

Original: Here, “Post-trainedGLT” (red
Revised: Here, “Post-trained

Original: [Figure]
Revised: GLT” (red line) starts training with the pretrained winning ticket on arXiv, while "Re-initialized GLT" (green line) starts training with the winning ticket re-initialized using original initial weight parameters. We can see that the sub-network obtained from arXiv transfers well on MAG in both settings, so the answers to our two questions are affirmative. Also, the blue (resp. black) line starts training with the post-trained GNN from arXiv without any edge or weight pruning; while the yellow line starts training with the winning ticket but with randomly initialized weight parameters.

Original: We can observe that (1) the post-trained winning ticket (red) achieves the same performance as the post-trained GNN (blue), showing that the transferred performance is not sensitive to weight pruning; and that (2) the winning ticket identified on the source task (green) achieves even better performance on the target task than a randomly initialized GNN (black).
Revised: We can observe that (1) the post-trained winning ticket (red) achieves the same performance as the post-trained GNN (blue), showing that the transferred performance is not sensitive to weight pruning; and that (2) the winning ticket identified on the source task (green) achieves even better performance on the target task than a randomly initialized GNN (black).

Original: Note that gradient descent could get trapped in undesirable stationary points if it starts from arbitrary points, and our GLT identified from the source task allows the initialization to land in a region of the loss landscape that is particularly amenable to optimization Ganea et al.
Revised: Note that gradient descent could get trapped in undesirable stationary points if it starts from arbitrary points, and our GLT identified from the source task allows the initialization to land in a region of the loss landscape that is particularly amenable to optimization Ganea et al.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: Experimental Setups.
Revised: Experimental Setups.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2021b), we evaluate our sparsification method with three popular GNN models: GCN Kipf & Welling (2017), GAT Velickovic et al.
Revised: (2021b), we evaluate our sparsification method with three popular GNN models: GCN Kipf & Welling (2017), GAT Velickovic et al.

Original: (2018) and GIN Xu et al.
Revised: (2018) and GIN Xu et al.

Original: (2019), on three widely used graph datasets from Chen et al.
Revised: (2019), on three widely used graph datasets from Chen et al.

Original: (2021b)
Revised: (2021b)

Original: (Cora, Citeseer and
Revised: (Cora, Citeseer and

Original: PubMed) and two OGB datasets from Hu et al.
Revised: PubMed) and two OGB datasets from Hu et al.

Original: (2020) (Arxiv and MAG) for semi-supervised node classification.
Revised: (2020) (Arxiv and MAG) for semi-supervised node classification.

Original: We compare our method with two baselines: UGS Chen et al.
Revised: We compare our method with two baselines: UGS Chen et al.

Original: (2021b)
Revised: (2021b)

Original: and random pruning Chen et al.
Revised: and random pruning Chen et al.

Original: (2021b).
Revised: (2021b).

Original: For fair comparison, we follow UGS to use the default setting: p g =and p θ = 20 unless otherwise stated.
Revised: For fair comparison, we follow UGS to use the default setting: p g =and p θ = 20 unless otherwise stated.

Original: The value of λ is configured as 0 . 1 by default.
Revised: The value of λ is configured as 0 . 1 by default.

Original: More details on the dataset statistics and model configurations can be found in the appendix.
Revised: More details on the dataset statistics and model configurations can be found in the appendix.

Original: Evaluation of Our Sparsification Method.
Revised: Evaluation of Our Sparsification Method.

Original: Figures 5, 6 and 7 compare our method with UGS and random pruning in terms of the test accuracy on three 2-layer GNN models: GCN, GIN and GAT, respectively.
Revised: Figures 4, 5 and 6 compare our method with UGS and random pruning in terms of the test accuracy on three 2-layer GNN models: GCN, GIN and GAT, respectively.

Original: In each figure, the first (resp.
Revised: In each figure, the first (resp.

Original: second) row shows how the test accuracy changes with graph sparsity (resp.
Revised: second) row shows how the test accuracy changes with graph sparsity (resp.

Original: weight sparsity) during the iterative pruning process, on the three datasets: Cora,
Revised: weight sparsity) during the iterative pruning process, on the three datasets: Cora,

Original: Citeseer and PubMed.
Revised: Citeseer and PubMed.

Original: We can observe that the accuracy of our method (green line) is much higher than UGS (yellow line) when the graph sparsity is high and when the weight sparsity is high.[Figure]
Revised: We can observe that the accuracy of our method (green line) is much higher than UGS (yellow line) when the graph sparsity is high and when the weight sparsity is high.

Original: [Table]
Revised: [Table]

Original: To clearly see this accuracy difference
Revised: To clearly see this accuracy difference

Original: [Table]
Revised: [Table]

Original: we show the test accuracy, graph sparsity (GS) and weight sparsity (WS) after the 17 th pruning iteration in Table 2 for both UGS and our method.
Revised: we show the test accuracy, graph sparsity (GS) and weight sparsity (WS) after the 17 th pruning iteration in Table 2 for both UGS and our method.

Original: Note that each graph sparsity is paired with a weight sparsity.
Revised: Note that each graph sparsity is paired with a weight sparsity.

Original: We use • and ⋆ to represent UGS and our method, respectively.
Revised: We use • and ⋆ to represent UGS and our method, respectively.

Original: Given the same sparcification iteration number, the sub-networks found by our method have a higher accuracy than those found by UGS.
Revised: Given the same sparcification iteration number, the sub-networks found by our method have a higher accuracy than those found by UGS.

Original: For example, when GS = 56 .
Revised: For example, when GS = 56 .

Original: 28% and WS = 97 .
Revised: 28% and WS = 97 .

Original: 52% , the accuracy of GCN on Cora is 72.5%, whereas our method improves the accuracy to 75.1% with a comparable setting: GS = 57 .
Revised: 52% , the accuracy of GCN on Cora is 72.5%, whereas our method improves the accuracy to 75.1% with a comparable setting: GS = 57 .

Original: 95% and WS = 97 .
Revised: 95% and WS = 97 .

Original: Note that although we use the same p g and p θ for both our method and UGS, their sparsity values in the same pruning iteration are comparable but not exactly equal.
Revised: Note that although we use the same p g and p θ for both our method and UGS, their sparsity values in the same pruning iteration are comparable but not exactly equal.

Original: This is because multiple mask entries can have the same value, and they span across the pruning threshold p g or p θ ; we prune them all in an iteration such that slightly more than p g fraction of edges or p θ fraction of weights may be removed.
Revised: This is because multiple mask entries can have the same value, and they span across the pruning threshold p g or p θ ; we prune them all in an iteration such that slightly more than p g fraction of edges or p θ fraction of weights may be removed.

Original: The higher accuracy of our method over UGS demonstrates that our approach finds lottery tickets that are more robust against the sparsity, and better prevents a performance collapse when the graph sparsity is high.
Revised: The higher accuracy of our method over UGS demonstrates that our approach finds lottery tickets that are more robust against the sparsity, and better prevents a performance collapse when the graph sparsity is high.

Original: In addition, we use • and ⋆ to indicate the last GLT with an accuracy higher than the original model in the sparsification process of UGS and our method, respectively.
Revised: In addition, we use • and ⋆ to indicate the last GLT with an accuracy higher than the original model in the sparsification process of UGS and our method, respectively.

Original: We can observe that the GLTs identified by our method have higher sparsity than those located by UGS.
Revised: We can observe that the GLTs identified by our method have higher sparsity than those located by UGS.

Original: It further verifies that our method can improve the robustness of GLT against the sparsity.
Revised: It further verifies that our method can improve the robustness of GLT against the sparsity.

Original: Note that in Figure 5, the third subfigure, the accuracy of our method (green line) is very stable even when graph sparsity is high.
Revised: Note that in Figure 4, the third subfigure, the accuracy of our method (green line) is very stable even when graph sparsity is high.

Original: This is because PubMed is a dense graph where some edges can result from noise during data collection, so graph sparsification functions as denoising, which retains or even improves model accuracy by alleviating the oversmoothing problem of GCN. 
Revised: This is because PubMed is a dense graph where some edges can result from noise during data collection, so graph sparsification functions as denoising, which retains or even improves model accuracy by alleviating the oversmoothing problem of GCN. 0.6 Graph sparsity 0. [Table] A cc u r a c

Original: Also note from Figure 7 that the performance of GAT is less sensitive to the sparsity as compared with
Revised: y ( C o r a ) UGS Random Ours 0. [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Also note from Figure 6 that the performance of GAT is less sensitive to the sparsity as compared with

Original: GCN and GIN.
Revised: GCN and GIN.

Original: This is because the attention mechanism in GAT leads to many more parameters than the other two models, allowing for pruning more parameters; moreover, GAT itself is able to identify the importance of edges through the attention mechanism, which better guides the edge pruning.
Revised: This is because the attention mechanism in GAT leads to many more parameters than the other two models, allowing for pruning more parameters; moreover, GAT itself is able to identify the importance of edges through the attention mechanism, which better guides the edge pruning.

Original: [Figure]
Revised: [Figure]

Original: Ablation Study.
Revised: Ablation Study.

Original: To verify the effectiveness of each design component in our sparsification algorithm, we compare our method with several variants.
Revised: To verify the effectiveness of each design component in our sparsification algorithm, we compare our method with several variants.

Original: First, we remove the Wassersteindistance based auxiliary loss head by setting λ = 0 during sparsification.
Revised: First, we remove the Wassersteindistance based auxiliary loss head by setting λ = 0 during sparsification.

Original: As shown in Figure 8, removing the auxiliary loss function leads to performance degradation.
Revised: As shown in Figure 7, removing the auxiliary loss function leads to performance degradation.

Original: However, there exists a tradeoff to weight the importance of L 1 .
Revised: However, there exists a tradeoff to weight the importance of L 1 .

Original: For example, if we increase λ from 0.1 to 1, the performance backfires especially when the sparisty is high.
Revised: For example, if we increase λ from 0.1 to 1, the performance backfires especially when the sparisty is high.

Original: We further investigate if our adversarial perturbation strategy can improve the robustness of the pruned GNNs, by replacing our min-max optimization process with the algorithm in UGS.
Revised: We further investigate if our adversarial perturbation strategy can improve the robustness of the pruned GNNs, by replacing our min-max optimization process with the algorithm in UGS.

Original: Specifically, the variant “UGS+WD” uses the sparsification algorithm in UGS to minimize L ( θ, m θ , m g ) .
Revised: Specifically, the variant “UGS+WD” uses the sparsification algorithm in UGS to minimize L ( θ, m θ , m g ) .

Original: Compared with this variant, our method can alleviate the performance degradation when graph sparsity is high, which verifies that adversarial perturbation improves robustness.
Revised: Compared with this variant, our method can alleviate the performance degradation when graph sparsity is high, which verifies that adversarial perturbation improves robustness.

Original: [Figure] [Figure]
Revised: 0.25 p g 0.

Original: [Figure]
Revised: p 79.

Original: Pairwise Sparsity.
Revised: Pairwise Sparsity.

Original: In Figure 9, we investigate the accuracy of the pruned GNNs after 10 iterations with different combinations of p g and p θ .
Revised: In Figure 8, we investigate the accuracy of the pruned GNNs after 10 iterations with different combinations of p g and p θ .

Original: Note that larger p g and p θ result in higher sparsity of the adjacent matrix and the weight matrices, respectively.
Revised: Note that larger p g and p θ result in higher sparsity of the adjacent matrix and the weight matrices, respectively.

Original: We can see that the performance of the pruned GNN is more sensitive to the graph sparsity than the weight sparsity.
Revised: We can see that the performance of the pruned GNN is more sensitive to the graph sparsity than the weight sparsity.

Original: It further verifies the necessity of finding robust GLTs when the graph sparsity is high.
Revised: It further verifies the necessity of finding robust GLTs when the graph sparsity is high.

Original: Since the pruned GNN is less sensitive to the weight sparsity, we can obtain GLTs with higher overall sparsity by setting p θ to be larger p g .
Revised: Since the pruned GNN is less sensitive to the weight sparsity, we can obtain GLTs with higher overall sparsity by setting p θ to be larger p g .

Original: At the same time, a larger p g can result in the pruning of more noisy and unimportant edges, which may help prevent GCN oversmoothing.
Revised: At the same time, a larger p g can result in the pruning of more noisy and unimportant edges, which may help prevent GCN oversmoothing.

Original: Therefore, a tradeoff exists when choosing a proper combination of p g and p θ .
Revised: Therefore, a tradeoff exists when choosing a proper combination of p g and p θ .

Original: While users may try different combinations of p g and p θ to find a proper combination, our approach is superior to
Revised: While users may try different combinations of p g and p θ to find a proper combination, our approach is superior to

Original: UGS in all combinations of p g and p θ , so should be a better algorithm choice.
Revised: UGS in all combinations of p g and p θ , so should be a better algorithm choice.

Original: Transfer Learning with GLTs.

Original: To verify the transferable graph lottery ticket hypothesis, we identify the GLT on arXiv and investigate the transfer ratio Kooverjee et al.

Original: (2022) of this GLT on MAG.

Original: Since the number of classes on arXiv is different from that on MAG, we need to either add a new dense layer for class mapping or replace the last dense layer to match the output classes in MAG.

Original: Figure 10(a) shows the target-task test accuracy of 3-layer GLTs (or GNNs) where the last dense layer is replaced.

Original: We can see that the post-trained GLT and post-trained GNN identified on the source task have a similar transfer ratio on the target task.

Original: Both the post-trained GLT and post-trained GNN have

Original: [Figure]

Original: significant improvement over the randomly initialized GNNs.

Original: Also, we can observe improvement of re-initialized GLT over randomly initialized GLT.

Original: As shown in Figure 10(b), similar results can be observed when adding a new dense layer.

Original: Lastly, Figure 10(c) investigates the relation between sparsity and transfer ratio (recall that we only transfer WS rather than GS), where we can see that the performance of the target task decreases as the sparsity of GLT increases.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the last few years, a series of GNN models Kipf & Welling (2017); Velickovic et al.
Revised: In the last few years, a series of GNN models Kipf & Welling (2017); Velickovic et al.

Original: Xu et al.
Revised: Xu et al.

Original: Hamilton et al.
Revised: Hamilton et al.

Original: ; Ying et al.
Revised: ; Ying et al.

Original: Defferrard et al.
Revised: Defferrard et al.

Original: ; Dwivedi et al.
Revised: ; Dwivedi et al.

Original: ; Klicpera et al.
Revised: ; Klicpera et al.

Original: ; Tong et al.
Revised: ; Tong et al.

Original: Yun et al.
Revised: Yun et al.

Original: (2019) have been designed to utilize the graph structure in spatial domain and spectral domain for effective feature extraction.
Revised: (2019) have been designed to utilize the graph structure in spatial domain and spectral domain for effective feature extraction.

Original: However, the computational cost of GNNs on large-scale graphs can be very high especially when the graph is dense with many edges.
Revised: However, the computational cost of GNNs on large-scale graphs can be very high especially when the graph is dense with many edges.

Original: Many models have been proposed to reduce this computational cost, such as
Revised: Many models have been proposed to reduce this computational cost, such as

Original: FastGCN Chen et al.
Revised: FastGCN Chen et al.

Original: and SGC Wu et al.
Revised: and SGC Wu et al.

Original: However, their accuracy tends to be worse than GCN, while the speedup achieved is still limited.
Revised: However, their accuracy tends to be worse than GCN, while the speedup achieved is still limited.

Original: So, it is desirable to develop a pruning method to sparsify both the edges and the weight parameters for any given GNN model (including FastGCN and SGC) without accuracy compromise.
Revised: So, it is desirable to develop a pruning method to sparsify both the edges and the weight parameters for any given GNN model (including FastGCN and SGC) without accuracy compromise.

Original: The lottery ticket hypothesis is first proposed in Frankle & Carbin (2019) to find an identically initialized sub-network that can be trained to achieve a similar performance as the original neural network.
Revised: The lottery ticket hypothesis is first proposed in Frankle & Carbin (2019) to find an identically initialized sub-network that can be trained to achieve a similar performance as the original neural network.

Original: Many works Frankle et al.
Revised: Many works Frankle et al.

Original: (2019; 2020); Diffenderfer & Kailkhura (2021);
Revised: (2019; 2020); Diffenderfer & Kailkhura (2021);

Original: Malach et al.
Revised: Malach et al.

Original: ; Ma et al.
Revised: ; Ma et al.

Original: ; Savarese et al.
Revised: ; Savarese et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021a;b);
Revised: (2021a;b);

Original: Zhou et al.
Revised: Zhou et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021c)
Revised: (2021c)

Original: ; Su et al.
Revised: ; Su et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: Evci et al.
Revised: Evci et al.

Original: (2020) have verified that the identified winning tickets are retrainable to reduce the high memory cost and long inference time of the original neural networks.
Revised: (2020) have verified that the identified winning tickets are retrainable to reduce the high memory cost and long inference time of the original neural networks.

Original: LTH has been extended to different kinds of neural networks such as GANs Chen et al.
Revised: LTH has been extended to different kinds of neural networks such as GANs Chen et al.

Original: (2021e;a); Kalibhat et al.
Revised: (2021e;a); Kalibhat et al.

Original: Transformers Brix et al.
Revised: Transformers Brix et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020); Behnke & Heafield (2020) and even deep reinforcement learning models Vischer et al.
Revised: (2020); Behnke & Heafield (2020) and even deep reinforcement learning models Vischer et al.

Original: (2021); Yu et al.
Revised: (2021); Yu et al.

Original: It has also been studied on GNNs by UGS Chen et al.
Revised: It has also been studied on GNNs by UGS Chen et al.

Original: (2021b).
Revised: (2021b).

Original: Despite the competitive performance of winning tickets, the iterative train-prune-retrain process and even the subsequent fine-tuning Liu et al.
Revised: Despite the competitive performance of winning tickets, the iterative train-prune-retrain process and even the subsequent fine-tuning Liu et al.

Original: (2021) are costly to compute.
Revised: (2021) are costly to compute.

Original: To address this issue,
Revised: To address this issue,

Original: PrAC Zhang et al.
Revised: PrAC Zhang et al.

Original: (2021c) proposes to find lottery tickets more efficiently by using a specially selected subset of data rather than using the full training set.
Revised: (2021c) proposes to find lottery tickets more efficiently by using a specially selected subset of data rather than using the full training set.

Original: Other works Wang et al.
Revised: Other works Wang et al.

Original: Tanaka et al.
Revised: Tanaka et al.

Original: (2020) attempt to find winning tickets at initialization without training.
Revised: (2020) attempt to find winning tickets at initialization without training.

Original: Customized hardware architectures have also been used to accelerate sparse training Goli & Aamodt (2020);
Revised: Customized hardware architectures have also been used to accelerate sparse training Goli & Aamodt (2020);

Original: Raihan & Aamodt (2020).
Revised: Raihan & Aamodt (2020).

Original: The Elastic Lottery Ticket Hypothesis Chen et al.
Revised: The Elastic Lottery Ticket Hypothesis Chen et al.

Original: (2021d) studies the transferability of winning tickets between different network architectures within the same design family.
Revised: (2021d) studies the transferability of winning tickets between different network architectures within the same design family.

Original: For object recognition, Mehta (2019) shows that winning tickets from VGG19 do not transfer well to down-stream tasks.
Revised: For object recognition, Mehta (2019) shows that winning tickets from VGG19 do not transfer well to down-stream tasks.

Original: In the natural image domain, Morcos et al.
Revised: In the natural image domain, Morcos et al.

Original: (2019) finds that the winning ticket initializations generalize across a variety of vision benchmark datasets.
Revised: (2019) finds that the winning ticket initializations generalize across a variety of vision benchmark datasets.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we empirically observe that the performance of GLTs collapses when the graph sparsity is high.
Revised: In this paper, we empirically observe that the performance of GLTs collapses when the graph sparsity is high.

Original: To explore the reason, we find that UGS cannot guide the pruning of many edges that get zero gradients during training.
Revised: To explore the reason, we find that UGS cannot guide the pruning of many edges that get zero gradients during training.

Original: We design a new auxiliary loss to address this limitation.
Revised: We design a new auxiliary loss to address this limitation.

Original: Moreover, we formulate the GNN sparsification process as a min-max optimization problem which adopts adversarial perturbation to improve the robustness of graph lottery tickets.
Revised: Moreover, we formulate the GNN sparsification process as a min-max optimization problem which adopts adversarial perturbation to improve the robustness of graph lottery tickets.

Original: Our experiments verified the superiority of our method over UGS.
Revised: Our experiments verified the superiority of our method over UGS.

Original: We further empirically studied transfer learning with GLTs and confirmed the transferability that gives rise to the transferable graph lottery ticket hypotheses.
Revised: We further empirically studied transfer learning with GLTs and confirmed the transferability that gives rise to the transferable graph lottery ticket hypotheses.

Original: A A PPENDIX A. D ATASET AND C ONFIGURATION
Revised: A A PPENDIX A. D ATASET AND C ONFIGURATION

Original: The statistics of benchmark datasets in the experiment are summarized in Table 3.
Revised: The statistics of benchmark datasets in the experiment are summarized in Table 3.

Original: For three citation networks: Cora, CiteSeer and PubMed, the feature of each node corresponds to the bag-of-words representation of the document.
Revised: For three citation networks: Cora, CiteSeer and PubMed, the feature of each node corresponds to the bag-of-words representation of the document.

Original: For arXiv and MAG, each node is associated with a 128-dimensional feature vector obtained by averaging the word embeddings of papers.
Revised: For arXiv and MAG, each node is associated with a 128-dimensional feature vector obtained by averaging the word embeddings of papers.

Original: Three models (GCN, GIN and GAT) are configured with the default parameters in their corresponding source code, including number of layers, dimension of hidden state and learning rate.
Revised: Three models (GCN, GIN and GAT) are configured with the default parameters in their corresponding source code, including number of layers, dimension of hidden state and learning rate.

Original: In our sparsification process, the value of η 1 , η 2 and α are configured as 1e-2, 1e-2 and 1e-1 by default, respectively.
Revised: In our sparsification process, the value of η 1 , η 2 and α are configured as 1e-2, 1e-2 and 1e-1 by default, respectively.

Original: In each pruning round, the number of epochs to update masks is configured as the default number of epochs in the training process of the original GNN model.
Revised: In each pruning round, the number of epochs to update masks is configured as the default number of epochs in the training process of the original GNN model.

Original: [Table]
Revised: [Table]

Original: A. GLT COMPARISON
Revised: A. GLT COMPARISON

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we show the test accuracy, graph sparsity (GS) and weight sparsity (WS) of the last GLT that reaches higher accuracy than the original model in the sparsification process of UGS and our 
Revised: we show the test accuracy, graph sparsity (GS) and weight sparsity (WS) of the last GLT that reaches higher accuracy than the original model in the sparsification process of UGS and our method. Note that GLT of GIN cannot be found on Cora in the sparsification process. We can observe

Original: [Table]
Revised: [Table]

Original: method.
Revised: that the last GLT identified by our method either has higher sparsity or achieves higher accuracy with comparable sparsity.

Original: Note that GLT of GIN cannot be found on Cora in the sparsification process.
Revised: In Table 5, we evaluate the test accuracy of sparsified GCN on two large-scale graph datasets: arXiv and MAG.

Original: We can observe thatthe last GLT identified by our method either has higher sparsity or achieves higher accuracy  with comparable sparsity.
Revised: We can observe that our method can result in higher accuracy than UGS with comparable graph sparsity and weight sparsity after 5, 15, and 20 iterations. It further verifies the effectiveness of the proposed method on large-scale datasets. [Table]

Original: A. P ERFORMANCE STANDARD AND INFERENCE TIME
Revised: A. P ERFORMANCE STANDARD AND INFERENCE TIME

Original: To combat randomness
Revised: To combat randomness

Original: [Figure]
Revised: [Table]

Original: we visualize the standard deviation of accuracy on Cora with a shaded color.
Revised: we visualize the standard deviation of accuracy on Cora with a shaded color.

Original: Following UGS
Revised: Following UGS

Original: [Figure]
Revised: [Table]

Original: we also show the inference MACs (multiply–accumulate operations) of pruned models in the sparsification process.
Revised: we also show the inference MACs (multiply–accumulate operations) of pruned models in the sparsification process.

Original: We can see that our method can significantly reduce the inference cost of GNNs with less performance drop.
Revised: We can see that our method can significantly reduce the inference cost of GNNs with less performance drop [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Transfer Learning from Arxiv to MAG A. T RANSFER L EARNING WITH GLT S

Revised: To verify the transferable graph lottery ticket hypothesis, we identify the GLT on arXiv and investigate the transfer ratio Kooverjee et al.

Revised: (2022) of this GLT on MAG.

Revised: Since the number of classes on arXiv is different from that on MAG, we need to either add a new dense layer for class mapping or replace the last dense layer to match the output classes in MAG.

Revised: Figure 10(a) shows the target-task test accuracy of 3-layer GLTs (or GNNs) where the last dense layer is replaced.

Revised: We can see that the post-trained GLT and post-trained GNN identified on the source task have a similar transfer ratio on the target task.

Revised: Both the post-trained GLT and post-trained GNN have significant improvement over the randomly initialized GNNs.

Revised: Also, we can observe improvement of re-initialized GLT over randomly initialized GLT.

Revised: As shown in Figure 10(b), similar results can be observed when adding a new dense layer.

Revised: Lastly, Figure 10(c) investigates the relation between sparsity and transfer ratio (recall that we only transfer WS rather than GS), where we can see that the performance of the target task decreases as the sparsity of GLT increases.