I NDUCTIVE L OTTERY T ICKET L EARNING FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep graph neural networks (GNNs) have gained increasing popularity, while usually suffer from unaffordable computations for real-world large-scale applications.
Hence, pruning GNNs is of great need but largely unexplored.
A recent work (Chen et al., 2021) studies lottery ticket learning for GNNs, aiming to ﬁnd a subset of model parameters and graph structure that can best maintain the GNN performance.
However, it is tailed for the transductive setting, failing to generalize to unseen graphs, which are common in inductive tasks like graph classiﬁcation.
In this work, we propose a simple and effective learning paradigm,
Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity.
To prune the input graphs, we design a predictive model to predict importance scores for each edge based on the input; to prune the model parameters, it views the weight’s magnitude as their importance scores.
Then we design an iterative co-pruning strategy to trim the graph edges and GNN weights based on their importance scores.
Although it might be strikingly simple, ICPG surpasses the existing pruning method and can be universally applicable in both inductive and transductive learning settings.
On ten graph-classiﬁcation and two node-classiﬁcation benchmarks, ICPG achieves the same performance level with 14.26% ∼ 43.12% sparsity for graphs and 48.80% ∼ 91.41% sparsity for the model.
I NTRODUCTION
Graph neural networks (GNNs) (Kipf & Welling, 2016; Veli ˇ ckovi ´ c et al., 2017; Zhou et al., 2018) have become a prevalent solution for machine learning tasks on graph-structured data.
Such success is usually ascribed to the powerful representation learning of GNN, which incorporates the graph structure into the representations, such as aggregating neural messages from the neighboring nodes to update the ego node’s representation (Veli ˇ ckovi ´ c et al., 2017; Kipf & Welling, 2016).
As the ﬁeld grows, there is an increasing need of building deeper GNN architectures (Li et al., 2020a; 2021) on larger-scale graphs (Hu et al., 2020).
While deepening GNNs shows potentials on large-scale graphs, it also brings expensive computations due to the increased scale of graph data and model parameters, limiting their deployment in resource-constrained applications.
Taking fraud detection in a transaction network as an example, the scale of user nodes easily reaches millions or even larger, making a GNN-detector model prohibitive to stack deep layers and predict the malicious behaviors in real time.
Hence, pruning over-parameterized GNNs is of great need, which aims to answer the question: Can we co-sparsify the input graphs and model parameters, while preserving or even improving the performance?
Recently, the pruning approach for GNNs, UGS, (Chen et al., 2021), is proposed to ﬁnd graph lottery tickets (GLTs) — smaller subsets of model parameters and input graphs.
At its core is Lottery Ticket
Hypothesis (LTH) (Frankle & Carbin, 2018) speculating that any dense, randomly-initialized neural network contains a sparse subnetwork, which can be trained independently to achieve a matching performance as the dense network.
Speciﬁcally, UGS employs trainable masks on each edge in the input graph and each weight in the model parameters, to specify their importance.
When training the model with the masks, the strategy of iterative magnitude-based pruning (IMP) (Frankle & Carbin, 2018) is used to discard the edges and weights with the lowest mask values at each iteration.
Despite effectiveness, there exist the following limitations: (1) UGS focuses solely on providing the transductive graph masks by generating a painstakingly customized mask for a single edge individually and independently.
That is, the edge masks are limited to the given graph, making
UGS infeasible to be applied in the inductive setting, since the edge masks hardly generalize to unseen edges or entirely new graphs.
(2) Applying a mask for each edge alone only provides a local understanding of the edge, rather than the global view of the entire graph ( e.g., in node classiﬁcation) or multiple graphs ( e.g., in graph classiﬁcation).
Moreover, the way of creating trainable edge masks will double the parameters of GNNs, which violates the purpose of pruning somehow.
As a result, these edge masks could be suboptimal to guide the pruning.
The unsatisfactory graph pruning will negatively inﬂuence the pruning of model weights.
Worse still, low-quality weight pruning will amplify the misleading signal of edge masks in turn.
They inﬂuence each other and form a vicious circle.
We ascribe all these limitations of UGS to its transductive nature.
Hence, conducting the combinatorial pruning in the inductive setting is crucial to high-quality winning tickets.
In this work, we emphasize the inductive nature within the combinatorial pruning of input graphs and
GNN parameters and present our framework, Inductive Co-Pruning of GNNs ( ICPG ).
It is an extremely simple but effective pruning framework that is applicable to any GNN in both inductive and transductive settings.
Speciﬁcally, for the input graphs, we design a generative probabilistic model (termed AutoMasker), which learns to generate edge masks from the observed graphs.
It is parameterized with an additional GNN-based encoder, whose parameters are shared across the population of observed graphs.
As a consequence, AutoMasker is naturally capable to specify the signiﬁcance for each edge and extract core-subgraphs from a global view of the entire observations.
For the model parameters, we simply exploit the magnitude of a model weight to assess whether it should be pruned, rather than training an additional mask (Chen et al., 2021).
Having established the edge masks and weight magnitudes, we can obtain high-quality GLTs by pruning the lowest-mask edges and lowest-magnitude weights.
Experiments on ten graph-classiﬁcation and two node-classiﬁcation benchmarks consistently validate our framework ICPG by identifying high-quality GLTs.
The visualizations show that ICPG always retains decisive subgraphs, such as edges located on digital pixels in MNIST graphs, which further illustrates the effectiveness and rationality.
Moreover, we inspect the graph- and GNN-level transferability of GLTs, which promises for deploying our ICPG in the pre-training and ﬁne-tuning paradigm.
Our main contributions can be summarized as follows:
• We proposed a simple but effective pruning framework, ICPG, to prune the GNN model and input graphs simultaneously.
It can identify high-quality GLTs in diverse tasks of graph representation learning, under both inductive and transductive settings.
• For graph classiﬁcation tasks, ICPG can locate GLTs from small-scale (TUDataset), mediumscale (Superpixel graphs) to challenging large-scale datasets (OGB) with graph sparsity from 22.62% ∼ 43.12% and GNN sparsity from 67.23% ∼ 91.41% with no degradation on performance.
• For node classiﬁcation tasks, ICPG can effectively develop on both transductive learning (Cora dataset) and inductive learning (PPI dataset), which identiﬁes the GLTs with graph sparsity from 22.62% ∼ 26.49% and GNN sparsity from 67.23% ∼ 73.79% without sacriﬁcing performance.
• The proposed AutoMasker promises for both GNN-level and graph-level transferability, which can achieve comparable or even better performance as compared with the original full graphs.
In-depth analyses with visual inspections further demonstrate the effectiveness and rationality.
R ELATED W ORK
Graph Neural Networks (GNNs) (Kipf & Welling, 2016; Veliˇckovi´c et al., 2017; Xu et al., 2019;
Ying et al., 2018) have emerged as a powerful tool for learning the representation of graph-structured data.
The great success mainly comes from the structure-aware learning, which follows the iterative message-passing scheme (Veliˇckovi´c et al., 2017).
Speciﬁcally, we denote an undirected graph by
G =
( A , X ) with the node set V and edge set E .
A ∈ { 0 , 1 } |V|×|V| is the adjacency matrix, where
A [ i, j ] = 1 denotes the edge between node v i and node v j , otherwise A [ i, j ] = 0 .
X ∈ R |V|× d is the matrix of node features, where x i =
X
[ i, :] is the d -dimensional feature of the node v i ∈ V .
Given a K -layer GNN, its k -th layer generates the representation of node v i as:
[Equation]
where h ( k ) i and a ( k ) i are the representation of node v i and the message aggregated from its neighbor nodes set N ( i ) , respectively; the AGGREGATION and COMBINE operators are the message passing and update functions, respectively.
After propagating through K layers, we get the ﬁnal representations of nodes, which facilitate downstream node-level tasks, such as node classiﬁcation and link prediction.
As for graph-level tasks like graph classiﬁcation and graph matching, we further hire the READOUT function to generate the representation of the whole graph G :
[Equation]
Various GNNs, such as GIN (Xu et al., 2019) and GAT (Veliˇckovi´c et al., 2017), implement different
AGGREGATION , COMBINE and READOUT functions, so as to reﬁne the desired information from graph structures.
Graph Sparsiﬁcation or Sampling (Voudigari et al., 2016; Leskovec & Faloutsos, 2006) aims to ﬁnd small core-subgraphs from the original graph, which can remain effective for graph learning tasks.
Numerous strategies (Zeng et al., 2019; Franceschi et al., 2019;
Ying et al., 2019; Hamilton et al., 2017; Chen et al., 2018) are proposed to achieve efﬁcient graph learning.
For example, GraphSAGE (Hamilton et al., 2017) samples and aggregates feature from a node’s local neighborhood.
FastGCN (Chen et al., 2018) adopts the global importance sampling, which is more efﬁcient for training.
DropEdge (Rong et al., 2019) randomly drops edges from the input graph, which can be seen as a data augmenter.
Another research line selects the core-subgraph in an optimization way.
SGCN (Li et al., 2020b) adopts the ADMM optimization algorithm to sparsify the adjacency matrix.
UGS (Chen et al., 2021) utilizes a trainable mask for each edge to remove the potential task-irrelevant edges.
Distinct from them, our AutoMasker predicts the importance score of each edge from a global view, thus having better generalization ability in the inductive settings.
Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2018) states that a sparse subnetwork exists in a dense randomly-initialized network that can be trained to achieve comparable performance to the full models.
LTH is explored in many ﬁelds such as computer vision and natural language processing (Chen et al., 2020; Ma et al., 2021;
Yin et al., 2020; Liu et al., 2019; Wang et al., 2020; Savarese et al., 2020).
Recently, Chen et al.
(2021) extends the LTH to the GNNs and proposes the Graph Lottery
Ticket (GLT), which includes subgraph and subnetwork pairs that can be trained independently to reach comparable performance to the dense pairs.
However, due to the transductive nature of graph-speciﬁc masks, UGS (Chen et al., 2021) can not develop on inductive learning settings, such as graph classiﬁcation tasks.
To tackle the dilemma, we propose AutoMasker to globally learn the signiﬁcance of each edge from training graphs and predict importance scores for new coming graphs, which is graph-independent and inductive.
M ETHODOLOGY
Here we ﬁrst formulate the task of learning graph lottery tickets, and then present our inductive strategy of co-pruning the input graphs and model weights.
3. I NDUCTIVE G RAPH L OTTERY T ICKET
Without loss of generality, we consider the inductive task of graph classiﬁcation as an example.
Given a GNN classiﬁer f ( · , Θ g 0 ) , it starts from the randomly-initialized parameters Θ g 0 before training and arrives at the well-optimized parameters Θ g after training.
Once trained, it takes any graph G =
( A , X ) as the input and yields a probability distribution over C classes
ˆ y =
f
( G , Θ g ) .
Wherein, G is associated with the adjacency matrix A and the pre-existing node features X .
The goal of learning graph lottery tickets is to make the input graph G and the model weights Θ gsparse to reduce the computational costs, while preserving the classiﬁcation performance.
Formally, it aims to generate two masks m G and m Θ , which are applied on G and Θ g 0 correspondingly, so as to establish the sparser input graph G (cid:48) =
( m G (cid:12) A , X ) and initialized weights Θ (cid:48) g
= m Θ (cid:12) Θ g 0 .
Hereafter, through retraining the subnetwork f ( · , Θ (cid:48) g 0 ) on the sparse versions {G (cid:48) } of training graphs, we can get the new converged parameters Θ (cid:48) g .
If the well-optimized subnetwork can achieve comparable performance with full graphs and network, we term the pair of G (cid:48) and f ( · , Θ (cid:48) g 0 ) as graph lottery tickets (GLTs).
Although a very recent study, UGS (Chen et al., 2021), has proposed an approach to learn the GLTs, it focuses solely on the transductive setting but leaves the inductive setting untouched.
Speciﬁcally, it assigns a trainable mask to each edge of the input graph and trains such graph-speciﬁc masks individually and independently.
As a consequence, these edge-dependent masks are limited to the given graph, hardly generalizing to unseen edges or entirely new graphs.
Distinct from UGS, we aim to uncover GLTs in inductive learning settings.
3. A UTO M ASKER
Instead of assigning mask to single edge, our idea is extremely simple: we take a collection of graph instances and design a trainable model to learn to mask edges collectively.
The key ingredient towards this model is an additional GNN-based model, termed AutoMasker, whose parameters are shared across the population of observed graphs.
Here we represent AutoMasker as the combination of a graph encoder and a subsequent scoring function.
Formally, given a graph G =
( A , X ) ,
AutoMasker applies a GNN-based graph encoder g ( · ) to create representations of all nodes as:
[Equation]
where H ∈ R |V× d | stores d -dimension representations of all nodes, whose i -th row h i represents the representation of node v i ; g ( · ) is a GNN following the message-passing paradigm in Equation 1.
To assess the importance score of edge ( i, j ) between node v i and node v j , AutoMasker builds a multilayer perceptron (MLP) upon the concatenation of node representations h i and h j , which yields the score α ij .
In what follows, the sigmoid function σ ( · ) projects α ij into the range of (0 , 1) , which represents the probability of edge ( i, j ) being the winning ticket.
The scoring function is represented as follows:
[Equation]
By employing the scoring function over all possible edges, we are able to collect the matrix of edge masks s G , where s G
[ i, j ] =
s
ij if edge ( i, j ) holds, otherwise s G
[ i, j ] = 0 .
In a nutshell, we summarize the AutoMasker function as follows:
[Equation]
where Θ a is the trainable parameters of AutoMasker, covering the parameters of the GNN encoder and the MLP.
Although the key ingredient of AutoMasker is simple, it has several conceptual advantages over
UGS: (1) Global view: Although edge masks derived from UGS might preserve the ﬁdelity to local importance, they do not help to delineate the general picture of the whole graph population.
Distinct from UGS, our AutoMasker takes a global view of the graph population, which enables us to identify the edge coalitions.
Speciﬁcally, as edges usually collaborate with each other to make predictions, rather than working individually, they form a coalition like the functional groups of a molecule graph, the community of a social network.
Considering such coalition effects, AutoMasker is able to measure the importance of edges more accurately.
(2) Lightweight edge masks: When using UGS to prune graph data with millions of edges or nodes, the cost of assigning local edge masks oneby-one will be prohibitive with such a large-scale dataset in real-world scenarios.
Moreover, UGS introduces additional parameters, whose scale remains the same as the edge numbers (cid:80) G |E| and is much larger than the original parameters being pruned.
Hence, it somehow violates the purpose of pruning.
In our AutoMasker, the additional parameters are i.e., Θ a in Equation 5 only and remain invariant across the change of data scale.
(3) Generalization: AutoMasker can generalize the mechanism of mask generation to new graphs without retraining, making it more efﬁcient to prune unseen and large-scale graphs.
3. I NDUCTIVE C O -P RUNING S TRATEGY
Here we present the framework of Inductive Co-Pruning of GNNs (ICPG) to localize GLTs inductively.
Figure 1 demonstrates its overview, which consists of the following two steps:
• Step 1: Co-training AutoMasker and the GNN model of interest.
Given an input graph G =
( A , X ) , AutoMasker
ﬁrst generates the edge mask s G via Equation 5.
Then we apply s G to the adjacency matrix A to create the masked graph G s =
( s G (cid:12) A , X ) , which fully reﬂects
AutoMasker’s decision for the importance of each edge, such that less important edges are prone to have lower mask values.
Finally, we feed the masked graphs into the GNN model to co-train the
[Figure]
Co
[Figure]
GNN
[Figure]
Co-sparsifying the input
[Figure]
model
[Figure]
AutoMasker and the model.
The GNN model adopts the masked graph to learn the representation and make predictions, which can be viewed as the supervision signals to guide the AutoMasker to achieve a more accurate decision.
The detailed co-training process is shown in Algorithm 1 of
Appendix A1.1.
When the training is done, we conduct Step 2 to perform the pruning.
• Step 2: Co-sparsifying the input graphs and GNN model.
Having obtained the well-trained AutoMasker and GNN model
we can
apply
the
knowledge learned
from
numerous graphs
to co-
sparsify
the graphs and the model.
For graphs
we adopt AutoMasker to
predict
the importance of all the edges for each graph
Then the edges of a certain graph are sorted based on the mask values
and the edges with 5% the lowest values
are pruned
to
obtain
the
binary graph mask m G .
For GNN,
we
sort
the parameters based on the weight magnitude and prune
the lowest
- magnitude parameters to obtain the binary model mask m Θ .
Under the current sparsity level, we now successfully obtain the sparsiﬁed graph G (cid:48)
=
( m G (cid:12) A , X ) and the sparsiﬁed mask m Θ for the model.
Finally, we need to check whether the sparsity meets our condition.
If the sparsity is satisﬁed, the algorithm is completed; if not, we need to reuse the found GLT to update the original graphs and GNN model, and iteratively use Step 1 and Step 2 (dotted arrow in Figure 1) until the condition is met.
In Appendix A1.1, Algorithm 2 offers the detailed algorithm of ICPG.
E XPERIMENTS
In this section, we conduct extensive experiments on diverse benchmarks to validate the effectiveness of the ICPG.
We ﬁrst introduce the experimental settings in Section 4.1, and explore the existence of GLTs in graph classiﬁcation and node classiﬁcation in Sections 4.2 and 4.3, respectively.
We also validate the transferability of the AutoMasker in Section 4.4.
More ablation studies and visualizations are provided in Sections 4.5 and 4.6, respectively.
4. E XPERIMENTAL S ETTINGS
Datasets.
For graph classiﬁcation, we use two biological graphs (NCI1, MUTAG), four social graphs (COLLAB, RED-B, RED-M5K, RED-M12K) (Morris et al., 2020), two superpixel graphs (MNIST, CIFAR-10)
(Knyazev et al., 2019), and two large-scale Open Graph Benchmark (ogbgppa and ogbg-code2)
(Hu et al., 2020).
For node classiﬁcation, we choose the transductive learning dataset: Cora, and the inductive learning dataset: PPI.
More details are provided in Appendix A1.
Models and Training Details.
We adopt the same architecture for the GNN model and GNN encoder for AutoMasker.
For all graph classiﬁcation datasets and Cora dataset, we adopt the GCN (Kipf & Welling, 2016) model with different layers and hiddens.
For the PPI dataset, we choose the
GAT (Veliˇckovi´c et al., 2017) network to achieve a better baseline performance as work (Veliˇckovi´c et al., 2017).
More details about models and training are provided in Appendix A1.4 and A1.5.
[Table]
[Figure]
*UDSK6SDUVLW\     5('0.
53 ,&3*2XUV %DVHOLQH 53*/7
 ,&3**/7
[Figure]
   
01,6
[Figure]
    &,)$5 53 ,&3*2XUV %DVHOLQH 53*/7 ,&3**/7
4. G RAPH L OTTERY T ICKETS IN G RAPH C LASSIFICATION
We ﬁrst conduct experiments to ﬁnd the GLTs in graph classiﬁcation tasks.
The results are displayed in Figure 2 and Figure 3.
We also plot the random pruning (RP) for better comparison.
Stars denote the extreme sparsity, which is the maximal sparsity-level without performance degradation.
More results about GNNs sparsity are provided in Appendix A2.1.
We make the following Obs ervations:
Obs.1.
GLTs extensively exist in graph classiﬁcation tasks.
Utilizing the ICPG, we can successfully locate the GLTs with different sparsity-levels from different types of graphs.
For NCI1 and
MUTAG, we precisely identify GLTs with the extreme graph sparsity at 26.49% and 30.17%,
GNN sparsity at 73.79% and 79.03%, respectively.
On four social network datasets, we ﬁnd the GLTs with graph sparsity of 22.62% ∼ 51.23% and GNN sparsity-level in 67.23% ∼ 95.60%.
For MNIST and CIFAR-10, the GLTs are achieved with graphs sparsity of 43.13% and 14.26%, GNN sparsity of 91.41% and 48.80%.
These results show that ICPG can locate the high-quality GLTs in graph classiﬁcation tasks with different graph types and demonstrate the potential of efﬁcient training or inference with sparser graphs and lightweight GNNs without sacriﬁcing performance
Obs.2.
AutoMasker has good generalization ability.
The mainstream graph sparsiﬁcation techniques (Chen et al., 2021; Zheng et al., 2020; Li et al., 2020b) cannot inductively prune unseen graphs.
However, the AutoMasker can ﬂexibly overcome this challenge.
Compared with random pruning (RP), our proposed ICPG can ﬁnd more sparse subgraphs and subnetworks and keep a large gap with RP.
For instance, the RED-M5K and RED-M12K graphs pruned by ICPG can achieve 40.13% and 51.23% extreme graph sparsity, improving 25.87% and 41.48% compared with RP, which keeps an extremely large superiority.
The excellent results indicate that AutoMasker can precisely capture more signiﬁcant core-patterns from the training graphs and have a good generalization ability to predict the high-quality masks for unseen graphs.
[Table]
[Figure]
*UDSK6SDUVLW\      )   VF R U H     RJEJFRGH 5
,&3*2XUV %DVHOLQH 53*/7 ,&3**/7
Obs.3.
The extreme sparsity of GLTs depends on the property of graphs.
Although ICPG achieves higher sparsity than RP on most graphs, the improvements are not obvious on a small part of the graphs, such as biochemical molecule graphs: NCI1 or MUTAG.
We make the following conjectures: Firstly, most of the edges in these graphs are important, such as a certain edge may correspond to a crucial chemical bond, which may drastically affect the chemical properties of the molecule if pruned.
Secondly, the graph size is relatively small, which just includes a few dozen nodes and edges, so it is more sensitive to pruning.
On the contrary, the larger social network datasets, such as RED-M5K or RED-M12K, contain hundreds or thousands of edges for each graph, so there may be plenty of redundant edges that demonstrate the insensitivity to pruning.
Obs.4.
AutoMasker can well tackle larger-size and larger-quantity graphs.
Figure 3 demonstrates the results on the challenging OGB datasets, which are consist of larger-size graphs (2266.
edges and 243.4 nodes on average per graph for ogbg-ppa) and larger-quantity graphs (452, graphs for ogbg-code2).
We surprisingly ﬁnd the OGB datasets are so intractable that RP can only locate 5% graph sparsity of GLT on the ogbg-ppa, and it is even impossible to ﬁnd any sparser GLTs on the ogbg-code2.
Despite this, the proposed ICPG can locate the GLTs with 14.26% and 18.55% graph sparsity, 48.80% and 59.40% GNN sparsity on ogbg-ppa and ogbg-code2, respectively.
The superior performance further veriﬁes the strong scalability and generalization of the AutoMasker.
4. G RAPH L OTTERY T ICKETS IN N ODE C LASSIFICATION
Since ICPG can achieve excellent performance on diverse types and scales of graphs, we also want to explore that if it can also tackle node-level tasks.
To answer this question, we conduct experiments on Core and PPI datasets, which are commonly used in transductive and inductive node classiﬁcation tasks.
We also reproduce the recent work UGS (Chen et al., 2021) for Cora (cannot apply for inductive setting) for better comparison.
We can get the following Obs ervations:
[Equation]
[Figure]
Obs.5.
ICPG can achieve excellent performance on node classiﬁcation tasks.
Firstly, for Cora, both ICPG and
UGS can ﬁnd GLTs that are sparser than
RP, which demonstrates that both are applicable to transductive node classiﬁcation.
Secondly, ICPG can ﬁnd sparser
GLTs than UGS ( ↑ 7.94%), while the performance drop faster than UGS in the later stage, we give the following conjectures: (1) UGS just adopts simple trainable masks for edges without considering the global topological structure of the entire graph, while AutoMasker is constructed based on a GNN-encoder, which can provide a global understanding of the whole graph.
Hence, the AutoMasker can predict more high-quality masks than UGS.
(2) ICPG is worse than UGS in the later stage.
The reason is that AutoMasker has found the most signiﬁcant edges in advance, which consists of an extremely compact core-subgraph at a certain critical point, so further pruning over that point will seriously degrade the performance.
Thirdly, as for the PPI dataset, the performance of ICPG still keeps a large gap with RP and can achieve 22.62% graph sparsity and 67.23% GNN sparsity without sacriﬁcing performance, which further demonstrates the effectiveness of the ICPG on inductive learning.
4. T HE T RANSFERABILITY OF THE A UTO M ASKER
We consider two orthogonal perspectives to verify the transferability of the AutoMasker: GNN-level transferability and graph-level transferability.
From GNNs view, we transfer the sparse graphs found by AutoMasker to the other two popular GNN models: GIN (Xu et al., 2019) and GAT (Veliˇckovi´c et al., 2017).
From graphs view, we ﬁrst pre-train the AutoMasker on larger-scale RED-M12K graphs and then transfer the well-trained AutoMasker to other two smaller-scale graphs: RED-B and RED-M5K. Please notes that we keep the GNN models unpruned on transferred tasks.
The experimental results are provided in Figure 5 and Table 1.
We make the following Obs ervations:
Obs.6.
AutoMasker has both GNN-level and graph-level transferability.
For GNN-level, we can observe from Figure 5 that GIN and GAT can achieve ranging 9.75% ∼ 45.96% and 18.55% ∼ 22.62% extreme sparsity on NCI1 and RED-M12K without sacriﬁcing performance.
And AutoMasker also outperforms RP and keeps a large gap.
These results demonstrate that AutoMasker can effectively extract the GNN-independent subgraphs.
These sparse subgraphs represent signiﬁcant semantic information and can be universally transferred to any GNN architecture without performance degradation.
For graph-level, the performance of random pruning decreases as the graph sparsity increases.
For RED-B and RED-M5K, when the pruning sparsity increases from 0 to 55.99%, the performance
[Figure]
    $ FF X U D F\     1&,*,1 53 $XWR0DVNHU2XUV %DVHOLQH
[Figure]

[Figure]
    5('0.*$7
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
decreases by 7.39% and 2.97%, respectively.
While AutoMasker can achieve consistent improvement within all sparsity levels.
Furthermore, the GNN model trained with more sparse graphs even outperforms the GNN trained with the original dense graphs, such as RED-B at 9.75% and REDM5K at 9.75% ∼ 45.96%.
It demonstrates that the pre-trained AutoMasker can transfer the learned knowledge to small-scale downstream tasks, with lower computational cost and better performance.
In summary, AutoMasker can learn model-independent, general and signiﬁcant sparse subgraph structures from the graphs, so that it has outstanding GNN-level and graph-level transferability.
4. A BLATION S TUDY
[Figure]
In this section, we investigate the diverse encoder networks in AutoMasker and explore the two independent components in ICPG.
We can make the following ﬁndings:
Encoder networks.
AutoMasker is
designed
on a GNN-based encoder, which leads to a
global understanding
of each edge from the entire graphs
So we extensively investigate the impact of the diverse encoders, such as
GNN-based or MLP-based encoders.
We can observe the results from Figure 6 ( Left ) that, for all the GNN-based encoders, AutoMasker can achieve good performance: 45.96% extreme sparsity for GCN and 51.23% for GIN and GAT, while MLP-based encoder only achieves 33.66% extreme sparsity.
It indicates that the message passing scheme of the GNN encoder naturally considers the graph structure from a global view, while the MLP-based encoder does not.
Co-sparsiﬁcation.
To further study the effectiveness of each component in ICPG: mask-based pruning for graphs (PG) and magnitude-based pruning for model (PM), we separate them and explore their roles when applying on the graphs and the model independently.
We also plot the performance of random pruning only for graphs (RPG), only for models (RPM), both of all (RPGM), random pruning for graphs with magnitude-based pruning for model (RPG-PM) for comparison.
The results are summarized in Figure 6 ( Right ).
We can ﬁnd that: PG can also ﬁnd the matching subgraphs (9.75% sparsity), which indicates that apply AutoMasker separately can also extract the signiﬁcant edges from each graph.
PM can also locate the matching subnetworks at 14.26% sparsity, which is consistent with the LTH (Frankle & Carbin, 2018) in the computer vision ﬁeld.
ICPG signiﬁcantly outperforms RPGM and RPG-PM, and the gap gradually widens as the sparsity increases.
We also observe that ICPG is even better than PG ( ↑ 12.87%), we made the following conjectures: (1) As for
[Figure]
PG, with the sparsity gradually increasing, the graphs also become more simple.
If we still train the over-parameterized GNN model with simple graphs, it may cause over-ﬁtting.
(2) Slightly pruning the over-parameterized GNN through PM can be regarded as a kind of regularization, which will improve the performance, and it is consistent with LTH (Frankle & Carbin, 2018).
Further, the regularized GNN can additionally provide AutoMasker with more precise supervision signals from the gradient in backpropagation to make more wise decisions.
To summary, we should co-train the
AutoMasker and GNN and co-sparsify the input graphs and model to achieve better performance.
4. V ISUALIZATION AND A NALYSIS
To better illustrate the signiﬁcant subgraphs extracted by the AutoMasker, we visualize the matching subgraphs in GLTs found by the ICPG.
We select graphs with 64.15% sparsity level from the MNIST and CIFAR-10 superpixel datasets.
For better comparison, we also plot the original images, original graphs, random pruning graphs, which are depicted in Figure 7.
More wonderful visualizations are provided in Figure A11 and Figure A12.
We can make the following ﬁndings:
For MNIST and CIFAR-10, the edges between nodes that locate on the digitals and objects pixels (the dark blue nodes) should be denser, which are conducive to the graph classiﬁcation tasks.
RP evenly prunes the signiﬁcant edges or structures without considering any important reference, which makes the core-subgraphs
destroyed and seriously deteriorates the performance.
ICPG utilizes AutoMasker to learn the signiﬁcance of each edge from a global view and can precisely prune redundant edges.
As the ICPG graph in Figure 7 (a) shows, the pruned edges are mainly located on non-digital pixels, such as the upper-left, lower-right corners and the center part of the number 0; the lower-left corner of the number 8, while the remaining edges or nodes are mainly located on digital pixels, which demonstrate that AutoMasker can indeed extract signiﬁcant patterns.
C ONCLUSION
In this work, we endow the graph lottery tickets with inductive pruning capacity.
We propose a simple and effective pruning framework ICPG, to co-sparsify the input graphs and GNN model.
For graphs, we propose a generative probabilistic model to generate the importance score for each edge.
It provides a global understanding of edge importance from the entire graph topological structure to guarantee high-quality graph masks and has strong generalization ability and transferability in inductive learning settings.
For the model, we adopt the GNN weight’s magnitude to estimate their importance scores.
Then we co-sparsify the input graphs and GNN model based on their important scores to ﬁnd the graph lottery tickets.
Extensive experiments on diverse types (biochemical molecules, social networks, superpixel graphs, citation networks) and scales (small, medium, large) of graphs, diverse learning settings (inductive, transductive) and diverse tasks (graph classiﬁcation, node classiﬁcation) consistently validate the effectiveness of the ICPG.
E THICS S TATEMENT
In recent years, graph neural networks (GNNs) have been widely applied in learning and processing graph-structured data, such as recommendation systems and drug discovery.
With the rapidly growing size of graphs, deep graph neural networks will inevitably confront expensive computational costs and slow response times in the training and inference stage.
Hence, it is challenging to implement deep GNNs in real-world large-scale graphs.
However, the current GNNs pruning methods cannot solve the inductive learning settings, which are common in real-world scenes, such as dynamic social networks learning.
Fortunately, the proposed co-pruning framework provides an effective and practical solution to reduce computational costs, which can be universally applied to both transductive and inductive graph learning.
Furthermore, the proposed AutoMasker also gives a novel pre-training perspective to save the computational cost in downstream tasks.
R EPRODUCIBILITY S TATEMENT
We promise that all results in this paper are reproducible.
The models, datasets, and training and inference settings in this paper have been given in the main text and Appendix A1.
To help readers easily reproduce our results, we provide a more detailed explanation.
Codes available at https: //anonymous.4open.science/r/Inductive_Lottery_Ticket_Learning- (1) Models .
For simplicity, we keep the same GNN architecture for the proposed AutoMasker and the following GNN model.
Please note that AutoMasker usually requires fewer hidden units.
The
GNN encoder network of AutoMasker can also be changed as needed and we have already given an example of replacing GCN based encoder with GIN or GAT in the ablation study.
All the results in this paper are based on the GCN model, except for the PPI dataset.
According to the suggestion of (Veli ˇ ckovi ´ c et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2017) network can achieve a better baseline performance on PPI dataset.
More detailed model settings such as the number of layers or hidden units have been summarized in Appendix A1.4 and Table A3.
2) Datasets
We conduct all the experiments on common graph learning benchmarks (Dwivedi et al., 2020; Xu et al., 2019;
Veli ˇ ckovi ´ c et al., 2017; Kipf & Welling, 2016; You et al., 2020;
Hu et al., 2020).
We have summarized the detailed information of the datasets in Appendix A1.2, the statistics of the datasets in Table A2, the method of the datasets splitting in Appendix A1.3.
(3) Baselines .
We adopt the following two baselines for comparison: random pruning (RP) and UGS (Chen et al., 2021).
For random pruning, we simply adopt a completely random selection strategy to prune the edges of the graph or the parameters of the GNN model.
For UGS, we reproduce the results of the original paper (Chen et al., 2021) based on the released code.
(4) Training and inference settings .
We extensively refer to the training settings for numerous literature in the ﬁeld of graph learning (Chen et al., 2019; Xu et al., 2019; Dwivedi et al., 2020;
Chen et al., 2021;
Veli ˇ ckovi ´ c et al., 2017), and conﬁgure the training hyperparameters according to their suggestions.
The detailed training settings have been given in the Appendix A1.5.
As for inference settings, we adopt the method consistent with (Chen et al., 2021; Xu et al., 2019;
Hu et al., 2020).
For the TUDataset, we adopt the 10-fold cross validation and report the mean and standard deviation.
For the OGB datasets, we follow the ofﬁcial method (Hu et al., 2020) to report the results.
For other datasets, we report the test accuracy at the epoch with the best validation accuracy.
A M ORE I MPLEMENTATION D ETAILS A1. A LGORITHMS
We summarize the detailed implementation of the proposed ICPG in Algorithm 2.
The Algorithmrepresents the co-training and co-sparsifying for a single iteration in ICPG.
Algorithm
Input: D , f ( · , Θ g 0 ) , AutoMasker ( · , Θ a 0 ) ,
M , m Θ , Epoch T .
Output: Sparsiﬁed masks { m (cid:48)G i } Ni =1 , m (cid:48) Θ .
1 for t = 0 to T − 1 do for G i ∈ D and m G i ∈ M do
G i ← ( m G i (cid:12)
A i , X i ) s G i ← AutoMasker
( G i , Θ a t )
G i ← ( s G i (cid:12)
A i , X i )
Forward f
( G i , m Θ (cid:12) Θ g t )
Backward to update Θ a t +1 , Θ g t +  end 9 end 10 for G
i ∈ D do s G i ← AutoMasker ( G i , Θ a T )
Set 5% of the lowest mask values in s G i to 0 and others to 1 , creating m (cid:48)G i .
13 end 14 Prune 20% of the lowest magnitude parameters in Θ g T , creating m (cid:48) Θ .
Input: Graphs D =
{G i =
( A i , X i ) } Ni =1 , f ( · , Θ g 0 ) , AutoMasker ( · , Θ a 0 ) , sparsity levels s d , s θ .
Output: GLT {G (cid:48)
i =
( m G i (cid:12) A i , X i ) } Ni =1 , f ( · ; m Θ (cid:12) Θ g 0 ) .
1 Initialize masks set M ← { m G i ←
A i } Ni = 2 Initialize GNN mask m Θ ← 1 ∈ R (cid:107) Θ g 0 (cid:107)3 while the sparsity of M < s d , m Θ < s θ do
Sparsify the GNN f ( · ; Θ g 0 ) with m Θ and dataset D
= {G i =
( A i , X i ) } Ni = with the mask set M and get the new masks as presented in Algorithm 1.
Update M ← { m G i ← m (cid:48)G i } Ni =
Update m Θ ← m (cid:48)
Θ
Rewind AutoMasker’s weight to Θ a 0 .
Rewind GNN’s weight to Θ g 0 .
9 end
A1. D ATASETS D ETAILS
As for graph classiﬁcation, we conduct the experiments with three scale levels: small-scale, mediumscale and large-scale.
We provide the following details:
• Small-scale : We adpot the TUDataset (Morris et al., 2020), which is a collection of benchmark datasets for graph classiﬁcation and regression.
We choose two biological graphs: NCI1, MUTAG and four social graphs: COLLAB, Reddit-Binary (RED-B),
Reddit-Multi-5K (RED-M5K), Reddit-Multi-12K (RED-M12K) as numerous works (Xu et al., 2019;
You et al., 2020; Dwivedi et al., 2020) does.
• Medium-scale : We use MNIST and CIFAR-10 superpixel graphs.
The original MNIST and CIFAR-10 images are converted to graphs using superpixels, which represent small regions of homogeneous intensity in images and can be extracted with the SLIC (Achanta et al., 2012) technique.
These datasets are commonly used in numerous graph representation learning researches (Dwivedi et al., 2020; You et al., 2020; Knyazev et al., 2019).
• Large-scale , we adopt the Open Graph Benchmark (OGB) (Hu et al., 2020), which is a collection of realistic, large
-scale, and diverse benchmark datasets for machine learning on graphs.
We choose two commonly used large-scale graph property prediction dataset: ogbg-ppa and ogbgcode2.
The ogbg-ppa dataset is a set of undirected protein association neighborhoods extracted from the protein-protein association networks of 1,581 different species that cover 37 broad taxonomic groups and span the tree of life.
The ogbg-code2 dataset is a collection of Abstract Syntax
Trees (ASTs) obtained from approximately 450 thousands Python method deﬁnitions.
As for node classiﬁcation, we use the commonly used citation network Cora dataset for semisupervised learning and the popular inductive learning dataset PPI.
The task for PPT is classifying protein functions across various biological protein-protein interaction (PPI) graphs.
These datasets are commonly used in transductive learning (Kipf & Welling, 2016; Veliˇckovi´c et al., 2017) and inductive learning (Hamilton et al., 2017; Zheng et al., 2020) researches.
All the detail statistics about theaforementioned datasets are summerized in Table A2.
[Table]
A1. D ATASETS S PLITTING
TUDataset : We perform the commonly used 10-fold cross validation.
Consistent with the work (Chen et al., 2019; Xu et al., 2019), we select the epoch with the best cross-validation accuracy averaged over the 10 folds and report the average and standard deviation of test accuracies at the selected epoch over 10 folds.
Superpixel dataset : Consistent with (Dwivedi et al., 2020), we split them to 55000 train/5000 validation/10000 test for MNIST, and 45000 train/5000 validation/10000 test for CIFAR10, respectively.
We report the test accuracy at the epoch with the best validation accuracy.
OGB dataset : We adopt the ofﬁcial dataset splitting method for ogbg-ppa and ogbg-code2 in the following links: https://ogb.stanford.edu/docs/graphprop/#ogbg-ppa and https://ogb.stanford.edu/docs/graphprop/#ogbg-code2 .
Cora : Following the work (Chen et al., 2021), we use 140 labeled data for training, 500 nodes for validation and 1000 nodes for testing and report the test accuracy at the epoch with the best validation accuracy.
PPI :
We adopt the same splitting as works (Hamilton et al., 2017; Veli ˇ ckovi ´ c et al., 2017).
The dataset contains 20 graphs for training, 2 for validation and 2 for testing.
Critically, testing graphs remain completely unobserved during training.
A1. M ODEL C ONFIGURATIONS
TUDataset: we adopt the ResGCN (Chen et al., 2019) with 3 layers and 128 hidden dimensions.
Superpixel dataset: we use GCN with 4 layers and 146 hidden dimensions as work (Dwivedi et al., 2020).
OGB dataset: we use ﬁve-layer GCN model with 300 hidden dimensions for all experiments.
Cora dataset: we adopt two-layer GCN network with 512 hidden dimensions as work (Chen et al., 2021).
PPI dataset: we adopt two-layer GAT network with 256 hidden and 4 attention heads as work (Veliˇckovi´c et al., 2017).
For GCN encoder in AutoMasker, we use the same structure with different hidden dimensions as the GNN model.
More details are summarized in Table A3.
A1. T RAINING D ETAILS
All training hyper-parameters such as epochs, learning rate for GNN model and AutoMasker, optimizer, batch size, weight decay are summarized in Table A3.
A
[Table]
A M ORE E XPERIMENTAL R ESULTS A2. G RAPH L OTTERY T ICKETS IN G RAPH C LASSIFICATION
The graph classiﬁcation performance with different GNN sparsity levels is shown in Figure A8.
The performance on large-scale datasets is provided in Figure A
[Table]
                           
    *116SDUVLW\  
 
$ FF X U D F\     1&, 53 ,&3*2XUV %DVHOLQH 53*/7 ,&3**/7
                            
   
[Table]
 


[Table]
%DVHOLQH
[Figure]
[Figure]
*116SDUVLW\    $ FF X U D F\     RJEJSSD 5
[Figure]
53*/7
[Figure]
RJEJFRGH 5
[Figure]
%DVHOLQH 53*/7
[Figure]
A2. G RAPH L OTTERY T ICKETS IN N ODE C LASSIFICATION
More results about the transductive and inductive node classiﬁcation on achieved GNN sparsity are shown in Figure A10.
We can observe that ICPG can achieve 73.79% GNN sparsity for transductive node classiﬁcation and 67.23% GNN sparsity for inductive node classiﬁcation.
[Figure]
33,
[Figure]
A M ORE V ISUALIZATION R ESULTS
More visualization results about the MNIST and CIFAR-10 superpixel graphs are shown in Figure
A11 and Figure A12. For MNIST, the important subgraphs are mainly located on the digital pixel area. We can observe that RP evenly prunes the connections between any two nodes without considering any important information.
While the proposed AutoMasker mainly prunes the insigniﬁcant edges that locate on non-digital pixel background. For CIFAR-10, objects are critical information for classiﬁcation, so the edges between the nodes that locate on the objects (the dark blue nodes) should be denser.
We can observe that RP prunes on both the objects and the background, which also makes all the vital edges become sparse.
As a comparison, the subgraphs extracted by AutoMasker remain most of the edges located on objects while the sparsify edges are located on the backgrounds. In summary, the AutoMasker effectively extracts the signiﬁcant core patterns while ﬁlter out the redundant edges from dense graphs, which necessarily leads to less degradation on performance even the graphs are heavily pruned.
[Figure]
A
[Figure]
A