G RAPH C ONDENSATION FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Given the prevalence of large-scale graphs in real-world applications, the storage and time for training neural models have raised increasing concerns.
To alleviate the concerns, we propose and study the problem of graph condensation for graph neural networks (GNNs).
Speciﬁcally, we aim to condense the large, original graph into a small, synthetic and highly-informative graph, such that GNNs trained on the small graph and large graph have comparable performance.
We approach the condensation problem by imitating the GNN training trajectory on the original graph through the optimization of a gradient matching loss and design a strategy to condense node futures and structural information simultaneously.
Extensive experiments have demonstrated the effectiveness of the proposed framework in condensing different graph datasets into informative smaller graphs.
In particular, we are able to approximate the original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer, while reducing their graph size by more than 99.9%, and the condensed graphs can be used to train various GNN architectures.
I NTRODUCTION
Many real-world data can be naturally represented as graphs such as social networks, chemical molecules, transportation networks, and recommender systems (Battaglia et al., 2018;
Wu et al., 2019b;
Zhou et al., 2018).
As a generalization of deep neural networks for graph-structured data, graph neural networks (GNNs) have achieved great success in capturing the abundant information residing in graphs and tackle various graph-related applications (Wu et al., 2019b; Zhou et al., 2018).
However, the prevalence of large-scale graphs in real-world scenarios, often on the scale of millions of nodes and edges, poses signiﬁcant computational challenges for training GNNs.
More dramatically, the computational cost continues to increase when we need to retrain the models multiple times, e.g., under incremental learning settings, hyperparameter and neural architecture search.
To address this challenge, a natural idea is to properly simplify, or reduce the graph so that we can not only speed up graph algorithms (including GNNs) but also facilitate storage, visualization and retrieval for associated graph data analysis tasks.
There are two main strategies to simplify graphs: graph sparsiﬁcation (Peleg & Sch¨affer, 1989;
Spielman & Teng, 2011) and graph coarsening (Loukas & Vandergheynst, 2018; Loukas, 2019) .
Graph sparsiﬁcation approximates a graph with a sparse graph by reducing the number of edges, while graph coarsening directly reduces the number of nodes by replacing the original node set with its subset.
However, these methods have some shortcomings: (1) sparsiﬁcation becomes much less promising in simplifying graphs when nodes are also associated with attributes as sparsiﬁcation does not reduce the node attributes; (2) the goal of sparsiﬁcation and coarsening is to preserve some graph properties such as principle eigenvalues (Loukas & Vandergheynst, 2018) that could be not optimal for the downstream performance of GNNs.
In this work, we ask if it is possible to signiﬁcantly reduce the graph size while providing sufﬁcient information to well train GNN models.
Motivated by dataset distillation (Wang et al., 2018) and dataset condensation (Zhao et al., 2021) which generate a small set of images to train deep neural networks on the downstream task, we aim to condense a given graph through learning a synthetic graph structure and node attributes.
[Figure]
Correspondingly, we propose the task of graph condensation 1 .
It aims to minimize the performance gap between GNN models trained on a synthetic, simpliﬁed graph and the original training graph.
In this work, we focus on attributed graphs and the node classiﬁcation task.
We show that we are able to reduce the number of graph nodes to as low as 0.1% while training various GNN architectures to reach surprisingly good performance on the synthetic graph.
For example, in Figure 1, we condense the graph of the Reddit dataset with 153,932 training nodes into only 154 synthetic nodes together with their connections.
In essence, we face two challenges for graph condensation: (1) how to formulate the objective for graph condensation tractable for learning; and (2) how to parameterize the to-be-learned node features and graph structure.
To address the above challenges, we adapt the gradient matching scheme in (Zhao et al., 2021) and match the gradients of GNN parameters w.r.t.
the condensed graph and original graph.
In this way, the GNN trained on condensed graph can mimic the training trajectory of that on real data.
Further, we carefully design the strategy for parametrizations for the condensed graph.
In particular, we introduce the strategy of parameterizing the condensed features as free parameters and model the synthetic graph structure as a function of features, which takes advantage of the implicit relationship between structure and node features, consumes less number of parameters and offers better performance.
Our contributions can be summarized as follows:
We make the ﬁrst attempt to condense a large-real graph into a small-synthetic graph, such that the GNN models trained on the large graph and small graph have comparable performance.
We introduce a proposed framework for g raph condensation (GC OND ) which parameterizes the condensed graph structure as a function of condensed node features, and leverages a gradient matching loss as the condensation objective.
Through extensive experimentation, we show that GC OND is able to condense different graph datasets and achieve comparable performance to their larger counterparts.
For instance, GC OND approximates the original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer, while reducing their graph size by more than 99.9%.
Our approach consistently outperforms coarsening, coreset and dataset condensation baselines.
We make several important ﬁndings towards the condensation problem: (a) condensed graphs can generalize well to different GNN test models; (b) condensing node features and structural information simultaneously can beneﬁt the performance; and (c) the condensed graphs are able to extract meaningful patterns hidden in the original data.
R ELATED W ORK
Dataset Distillation & Condensation.
Dataset distillation (DD) (Wang et al., 2018; Bohdal et al., 2020; Nguyen et al., 2021) aims to distill knowledge of a large training dataset into a small synthetic dataset, such that a model trained on the synthetic set is able to obtain the comparable performance to that of a model trained on the original dataset.
To improve the efﬁciency of DD, dataset condensation (DC) (Zhao et al., 2021; Zhao & Bilen, 2021) is proposed to learn the small synthetic dataset by matching the gradients of the network parameters w and small training data.
However, these methods are designed exclusively for image data and are not applicable to non-Euclidean graph-structured data where samples (nodes) are interdependent.
In this work, we generalize the problem of dataset condensation to graph domain and we seek to jointly learn the synthetic node features as well as graph structure.
Additionally, our work relates to coreset methods (Welling, 2009; Sener & Savarese, 2017; Rebufﬁ et al., 2017), which seek to ﬁnd informative samples from the original datasets.
However, they rely on the presence of representative samples, and tend to give suboptimal performance.
Graph Sparsiﬁcation & Coarsening.
Graph sparsiﬁcation and coarsening are two means of reducing the size of a graph.
Sparsiﬁcation reduces the number of edges while approximating pairwise distances (Peleg & Sch¨affer, 1989), cuts (Karger, 1999) or eigenvalues (Spielman & Teng, 2011) while coarsening reduces the number of nodes with similar constraints (Loukas & Vandergheynst, 2018; Loukas, 2019;
Deng et al., 2020), typically by grouping original nodes into super-nodes, and deﬁning their connections.
Cai et al.
(2021) proposes a GNN-based framework to learn these connections to improve coarsening quality.
Huang et al.
(2021) adopts coarsening as a preprocessing method to help scale up GNNs.
Graph condensation also aims to reduce the number of nodes, but aims to learn synthetic nodes and connections in a supervised way, rather than unsupervised grouping as in these prior works.
Graph pooling is also related to our work, but it targets at improving graph-level representation learning (see Appendix D).
Graph Neural Networks.
Graph neural networks (GNNs) are a modern way to capture the intuition that inferences for individual samples (nodes) can be enhanced by utilizing graph-based information from neighboring nodes (Kipf & Welling, 2016; Hamilton et al., 2017; Klicpera et al., 2018;
Veliˇckovi´c et al., 2018; Wu et al., 2019b;a).
Due to their prevalence, various real-world applications have been tremendously facilitated including recommender systems (Ying et al., 2018a;
Fan et al., 2019), computer vision (Li et al., 2019) and drug discovery (Duvenaud et al., 2015).
Graph Structure Learning.
Our work is also related to graph structure learning, which explores methods to learn graphs from data.
One line of work (Dong et al., 2016; Egilmez et al., 2017) learns graphs under certain structural constraints (e.g. sparsity) based on graph signal processing.
Recent efforts aim to learn graphs by leveraging GNNs (Franceschi et al., 2019; Jin et al., 2020; Chen et al., 2020).
However, these methods are incapable of learning graphs with smaller size, and are thus not applicable for graph condensation.
M ETHODOLOGY
In this section, we present our proposed graph condensation framework, GC OND .
Consider that we have a graph dataset T
= { A , X , Y } , where A ∈ R N × N is the adjacency matrix, N is the number of nodes, X ∈ R N ×
d is the d -dimensional node feature matrix and Y ∈ { 0 , . . .
, C − 1 } N denotes the node labels over C classes.
Graph condensation aims to learn a small, synthetic graph dataset
S =
{ A (cid:48) , X (cid:48) , Y (cid:48) } with A (cid:
) ∈ R N (cid:48)
× N (cid:48) , X (cid:48) ∈ R N (cid:48)
× D , Y (cid:48) ∈ { 0 , . .
, C − 1 } N (cid:48) and N (cid:48) (cid:28) N , such that a GNN trained on S can achieve comparable performance to one trained on the much larger T .
Thus, the objective can be formulated as the following bi-level problem,
[Equation]
where GNN θ denotes the GNN model parameterized with θ , θ S denotes the parameters of the model trained on S , and L denotes the loss function used to measure the difference between model predictions and ground truth, i.e. cross-entropy loss.
However, optimizing the above objective can lead to overﬁtting on a speciﬁc model initialization.
To generate condensed data that generalizes to a distribution of random initializations P θ 0 , we rewrite the objective as follows:
[Equation]
where θ ( θ 0 ) indicates that θ is a function acting on θ 0 .
Note that the setting discussed above is for inductive learning where all the nodes are labeled and test nodes are unseen during training.
We can easily generalize graph condensation to transductive setting by assuming Y is partially labeled.
3. G RAPH C ONDENSATION VIA G RADIENT M ATCHING
To tackle the optimization problem in Eq.
(2), one strategy is to compute the gradient of L w.r.t S and optimize S via gradient descent, as in dataset distillation (Wang et al., 2018).
However, this requires solving a nested loop optimization and unrolling the whole training trajectory of the inner problem, which can be prohibitively expensive.
To bypass the bi-level optimization, we follow the gradient matching method proposed in (Zhao et al., 2021) which aims to match the network parameters w.r.t.
large-real and small-synthetic training data by matching their gradients at each training step.
In this way, the training trajectory on small-synthetic data S can mimic that on the large-real data T , i.e., the models trained on these two datasets converge to similar solutions (parameters).
Concretely, the parameter matching process for GNNs can be modeled as follows:
[Equation]
with
[Equation]
and
[Equation]
where D ( · , · ) is a distance function, T is the number of steps of the whole training trajectory, opt θ is the update rule for model parameters, and θ S t , θ T t denote the model parameters trained on S and T at time step t , respectively.
Since our goal is to match the parameters step by step, we then consider one-step gradient descent for the update rule opt θ :
[Equation]
and
[Equation]
where η is the learning rate for the gradient descent.
Based on the observation made in Zhao et al.
(2021) that the distance between θ S t and θ T t is typically small, we can simplify the objective as a gradient matching process as follows,
[Equation]
where θ S t and θ T t are replaced by θ t , i.e., training the same GNN on both synthetic graph and original graph.
The distance D is further deﬁned as the sum of the distance dis at each layer.
Given two gradients G S ∈ R d 1 × d 2 and G T ∈ R d 1 × d 2 at a speciﬁc layer, the distance dis ( · , · ) used for condensation is deﬁned as follows,
[Equation]
where G S i , G T i are the i -th column vectors of the gradient matrices.
With the above formulations, we are able to achieve parameter matching through an efﬁcient strategy of gradient matching.
We note that jointly learning the three variables A (cid:48) , X (cid:48) and Y (cid:48) is highly challenging, as they are interdependent.
Hence, to simplify the problem, we ﬁx the node labels Y (cid:48) while keeping the class distribution the same as the original labels Y .
Graph Sampling.
GNNs are often trained in a full-batch manner (Kipf & Welling, 2016; Wu et al., 2019b).
However, as suggested by previous works that reconstruct data from gradients (Zhu et al., 2019), large batch size tends to make reconstruction more difﬁcult because more variables are involved during optimization.
To make things worse, the computation cost of GNNs gets expensive on large graphs as the forward pass of GNNs involves the aggregation of enormous neighboring nodes.
To address the above issues, we sample a ﬁxed-size set of neighbors on the original graph in each aggregation layer of GNNs and adopt a mini-batch training strategy.
To further reduce memory usage and ease optimization, we calculate the gradient matching loss for nodes from different classes separately, as matching the gradients w.r.t.
the data from a single class is easier than that from all classes.
Speciﬁcally, for a given class c , we sample a batch of nodes of class c together with a portion of their neighbors from large-real data T .
We denote the process as ( A c , X c , Y c ) ∼ T .
For the condensed graph A (cid:48) , we sample a batch of synthetic nodes of class c but do not sample their neighbors.
In other words, we use all of their neighbors, i.e., all other nodes, during the aggregation process, since we need to learn the connections with other nodes.
We denote the process as ( A (cid:48) c , X (cid:48) c , Y (cid:48) c ) ∼ S .
3. M ODELING C ONDENSED G RAPH D ATA
One essential challenge in the graph condensation problem is how to model the condensed graph data and resolve dependency among nodes.
The most straightforward way is to treat both A (cid:48) and X (cid:48) as free parameters.
However, the number of parameters in A (cid:48) grows quadratically as N (cid:48) increases.
The increased model complexity can pose challenges in optimizing the framework and increase the risk of overﬁtting.
Therefore, it is desired to parametrize the condensed adjacency matrix in a way where the number of parameters does not grow too fast.
On the other hand, treating A (cid:48) and X (cid:48) as independent parameters overlooks the implicit correlations between graph structure and features, which have been widely acknowledged in the literature (Pfeiffer III et al., 2014; Shalizi &
Thomas, 2011); e.g., in social networks, users interact with others based on their interests, while in e-commerce, users purchase products due to certain product attributes.
Hence, we propose to model the condensed graph structure as a function of the condensed node features:
[Equation]
where MLP Φ is a multi-layer neural network parameterized with Φ and [ · ; · ] denotes concatenation.
In Eq.
(7), we intentionally control A (cid:48) ij =
A (cid:48) ji to make the condensed graph structure symmetric since we are mostly dealing with symmetric graphs.
It can also adjust to asymmetric graphs by setting A
[Equation]
ij =
Sigmoid ( MLP Φ ([ x i ; x (cid:48) j ]) .
Then we rewrite our objective as
[Equation]
Note that there are two clear beneﬁts of the above formulation over the na ¨ ıve one (free parameters).
Firstly, the number of parameters for modeling graph structure no longer depends on the number of nodes, hence avoiding jointly learning O ( N (cid:48) 2 ) parameters; as a result, when N (cid:48) gets larger,
GC OND suffers less risk of overﬁtting.
Secondly, if we want to grow the synthetic graph by adding more synthetic nodes condensed from real graph, the trained MLP Φ can be employed to infer the connections of new synthetic nodes, and hence we only need to learn their features.
Alternating Optimization Schema.
Jointly optimizing X (cid:48) and Φ is often challenging as they are directly affecting each other.
Instead, we propose to alternatively optimize X (cid:48) and Φ : we update Φ for the ﬁrst τ 1 epochs and then update X (cid:48) for τ 2 epochs; the process is repeated until the stopping condition is met – we ﬁnd empirically that this does better as shown in Appendix B.
Sparsiﬁcation.
In the learned condensed adjacency matrix A (cid:48) , there can exist some small values which have little effect on the aggregation process in GNNs but still take up a certain amount of storage (e.g. 4 bytes per ﬂoat).
Thus, we remove the entries whose values are smaller than a given threshold δ to promote sparsity of the learned A (cid:48) .
We further justify that suitable choices of δ for sparsiﬁcation do not degrade performance a lot in Appendix B.
The detailed algorithm can be found in Algorithm 1 in Appendix C.
In detail, we ﬁrst set the condensed label set Y (cid:48) to ﬁxed values and initialize X (cid:48)
as node features randomly selected from each class.
In each outer loop, we sample a GNN model initialization θ from a distribution P θ .
Then, for each class we sample the corresponding node batches from T and S , and calculate the gradient matching loss within each class.
The sum of losses from different classes are used to update X (cid:48) or Φ .
After that we update the GNN parameters for τ θ epochs.
When ﬁnishing the updating of condensed graph parameters, we use A (cid:48)
=
ReLU ( g Φ ( X (cid:48) ) − δ ) to obtain the ﬁnal sparsiﬁed graph structure.
A “Graphless” Model Variant.
We now explore another parameterization for the condensed graph data.
We provide a model variant named GC OND - X that only learns the condensed node features
X (cid:48) without learning the condensed structure A (cid:48) .
In other words, we use a ﬁxed identity matrix I as the condensed graph structure.
Speciﬁcally, this model variant aims to match the gradients of GNN parameters on the large-real data ( A , X ) and small-synthetic data ( I , X (cid:48) ) .
Although GC OND - X is unable to learn the condensed graph structure which can be highly useful for downstream data analysis, it still shows competitive performance in Table 2 in the experiments because the features are learned to incorporate relevant information from the graph via the matching loss.
E XPERIMENTS
In this section, we design experiments to validate the effectiveness of the proposed framework
GC OND .
We ﬁrst introduce experimental settings, then compare GC OND against representative baselines with discussions and ﬁnally show some advantages of GC OND .
4. E XPERIMENTAL SETUP
Datasets.
We evaluate the condensation performance of the proposed framework on three transductive datasets, i.e., Cora , Citeseer (Kipf & Welling, 2016) and Ogbn-arxiv (Hu et al., 2020), and two inductive datasets, i.e., Flickr (Zeng et al., 2020) and Reddit (Hamilton et al., 2017).
We use the public splits for all the datasets.
For the inductive setting, we follow the setup in (Hamilton et al., 2017) where the test graph is not available during training.
Dataset statistics are shown in Appendix A.
Baselines.
We compare our proposed methods to ﬁve baselines: (i) one graph coarsening method (Loukas, 2019; Huang et al., 2021), (ii-iv) three coreset methods ( Random , Herding (Welling, 2009) and K-Center (Farahani & Hekmatfar, 2009; Sener & Savarese, 2017)), and (v) dataset condensation (DC).
For the graph coarsening method, we adopt the variation neighborhoods method implemented by Huang et al.
For coreset methods, we ﬁrst use them to select nodes from the original dataset and induce a subgraph from the selected nodes to serve as the reduced graph.
In Random, the nodes are randomly selected.
The Herding method, which is often used in continual learning (Rebufﬁ et al., 2017; Castro et al., 2018), picks samples that are closest to the cluster center.
K-Center selects the center samples to minimize the largest distance between a sample and its nearest center.
We use the implementations provided by Zhao et al.
(2021) for
Herding, K-Center and DC.
As vanilla DC cannot leverage any structure information, we develop a variant named DC-Graph, which additionally leverages graph structure during test stage, to replace
DC for the following experiments.
A comparison between DC, DC-Graph, GC OND and GC OND - X is shown in Table 1 and their training details can be found in Appendix A.3.
Evaluation.
We ﬁrst use the aforementioned baselines to obtain condensed graphs and then evaluate them on GNNs for both transductive and inductive node classiﬁcation tasks.
For transductive datasets, we condense the full graph with N nodes into a synthetic graph with rN ( 0 < r < 1 ) nodes, where r is the ratio of synthetic nodes to original nodes.
For inductive datasets, we only condense the training graph since the rest of the full graph is not available during training.
The choices of r 2 are listed in Table 2.
For each r , we generate 5 condensed graphs with different seeds.
To evaluate the effectiveness of condensed graphs, we have two stages: (1) a training stage, where we train a GNN model on the condensed graph, and (2) a test stage, where the trained GNN uses the test graph (or full graph in transductive setting) to infer the labels for test nodes.
The resulting test performance is compared with that obtained when training on original datasets.
All experiments are repeated 10 times, and we report average performance and variance.
Hyperparameter settings.
As our goal is to generate highly informative synthetic graphs which can beneﬁt GNNs, we choose one representative model, GCN (Kipf & Welling, 2016), for performance evaluation.
For the GNN used in condensation, i.e., the GNN θ ( · ) in Eq.
(8), we adopt
SGC (Wu et al., 2019a) which removes the ReLU activation from GCN.
In our gradient matching problem, SGC enjoys the advantage of being twice-differentiable while sharing similar graph ﬁltering behavior as GCN.
Unless otherwise stated, we use 2-layer models with 256 hidden units.
The weight decay and dropout for the models are set to 0 in condensation process.
More details for hyper-parameter tuning can be found in Appendix A.
4. C OMPARISON WITH B ASELINES
In this subsection, we test the performance of a 2-layer GCN on the condensed graphs, and compare the proposed GC OND and GC OND - X with baselines.
Notably, all methods produce both structure and node features, i.e. A (cid:48) and X (cid:48) , except DC-Graph and GC OND - X .
Since DC-Graph and GC ONDX do not produce any structure, we simply use an identity matrix as the adjacency matrix when training GNNs solely on condensed features.
However, during inference, we use the full graph (transductive setting) or test graph (inductive setting) to propagate information based on the trained
GNNs.
This training paradigm is similar to the C&S model (Huang et al., 2020) which trains an
MLP without the graph information and performs label propagation based on MLP predictions.
Table 2 reports node classiﬁcation performance; we make the following observations:
Obs 1.
Condensation methods achieve promising performance even with extremely large reduction rates.
Condensation methods, i.e., GC OND , GC OND - X and DC-Graph, outperform coreset methods and graph coarsening signiﬁcantly at the lowest ratio r for each dataset.
This shows the
[Table] [Table]
importance of learning synthetic data using the guidance from downstream tasks.
Notably, GC OND achieves 79.8%, 80.1% and 79.3% at 1.3%, 2.6% and 5.2% condensation ratios at Cora , while the whole dataset performance is 81.2%.
The GC OND variants also show promising performance on
Cora , Flickr and Reddit at all coarsening ratios.
Although the gap between whole-dataset Ogbn-arxiv and our methods is larger, they still outperform baselines by a large margin.
Obs 2.
Learning X (cid:48) instead of ( A (cid:48) , X (cid:48) )
as the condensed graph can also lead to good results.
GC OND - X achieves close performance to GC OND on 11 of 15 cases.
Since our objective in graph condensation is to achieve parameter matching through gradient matching, training a GNN on the learned features X (cid:48) with identity adjacency matrix is also able to mimic the training trajectory of
GNN parameters.
One reason could be that X (cid:48) has already encoded node features and structural information of the original graph during the condensation process.
However, there are many scenarios where the graph structure is essential such as the generalization to other GNN architectures (e.g., GAT) and visualizing the patterns in the data.
We will provide more details in the following subsections.
Obs 3.
Condensing node features and structural information simultaneously can lead to better performance.
In most cases, GC OND and GC OND - X obtain much better performance than DCGraph.
One key reason is that GC OND and GC OND - X can take advantage of both node features and structural information in the condensation process.
We notice that DC-Graph achieves a highly comparable result (90.5%) on Reddit at 0.2% condensation ratio to the whole dataset performance (93.9%).
This may indicate that the original training graph structure might not be useful.
To verify this assumption, we train a GCN on the original Reddit dataset without using graph structure (i.e., setting A train = I ), but allow using the test graph structure for inference using the trained model.
The obtained performance is 92.5%, which is very close to the original performance 93.9%, indicating that training without graph structure can still achieve comparable performance.
We also note that learning X (cid:48) , A (cid:48) simultaneously creates opportunities to absorb information from graph structure
[Table]
directly into learned features, lessening reliance on distilling graph properties reliably while still achieving good generalization performance from features.
Obs 4.
Larger condensed graph size does not strictly indicate better performance.
Although larger condensed graph sizes allow for more parameters which can potentially encapsulate more information from original graph, it simultaneously becomes harder to optimize due to the increased model complexity.
We observe that once the condensation ratio reaches a certain threshold, the performance becomes stable.
However, the performance of coreset methods and graph coarsening is much more sensitive to the reduction ratio.
Coreset methods only select existing samples while graph coarsening groups existing nodes into super nodes.
When the reduction ratio is too low, it becomes extremely difﬁcult to select informative nodes or form representative super nodes by grouping.
4. G ENERALIZABILITY OF C ONDENSED G RAPHS
Next, we illustrate the generalizability of condensed graphs from the following two perspectives: (1) the graphs that are condensed based on one speciﬁc GNN can be used to train other GNNs; and (2) graphs can be condensed using different GNNs and they all show reasonable transfer performance.
Different Architectures.
Next, we show the generalizability of the graph condensation procedure.
Speciﬁcally, we show test performance when using a graph condensed by one GNN model to train different GNN architectures.
Speciﬁcally, we choose APPNP (Klicpera et al., 2018), GCN,
SGC (Wu et al., 2019a), GraphSAGE (Hamilton et al., 2017), Cheby (Defferrard et al., 2016) and
GAT (Veli ˇ ckovi ´ c et al., 2018).
We also include MLP and report the results in Table 3.
From the table, we ﬁnd that the condensed graphs generated by GC OND show good generalization on different architectures.
We may attribute such transferability across different architectures to similar ﬁltering behaviors of those GNN models, which have been studied in Ma et al.
Zhu et al.
Versatility of GC OND .
The proposed GC OND is highly composable in that we can adopt various
GNNs inside the condensation network.
We investigate the performances of various GNNs when using different GNN models in the condensation process, i.e., GNN θ ( · ) in Eq.
We choose APPNP,
Cheby, GCN, GraphSAGE and SGC to serve as the models used in condensation and evaluation.
Note that we omit GAT due to its deterioration under large neighborhood sizes (Ma et al., 2021).
We choose Cora and Ogbn-arxiv to report the performance in Table 4 where C and T denote condensation and test models, respectively.
The graphs condensed by different GNNs all show strong transfer performance on other architectures.
4. A NALYSIS ON C ONDENSED D ATA
Statistics of Condensed Graphs.
In Table 5, we compare several properties between condensed graphs and original graphs.
Note that a widely used homophily measure is deﬁned in (Zhu et al., 2020) but it does not apply to weighted graphs.
Hence, when computing homophily, we binarize the
[Table]
[Table]
The condensed graphs have fewer nodes and are more dense.
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
graphs by removing edges whose weights are smaller than 0.5.
We make the following observations.
First, while achieving similar performance for downstream tasks, the condensed graphs contain fewer nodes and take much less storage.
Second, the condensed graphs are less sparse than their larger counterparts.
Since the condensed graph is on extremely small scale, there would be almost no connections between nodes if the condensed graph maintains the original sparsity.
Third, for
Citeseer , Cora and Flickr , the homophily information are well preserved in the condensed graphs.
Visualization.
We present the visualization results for all datasets in Figure 4, where nodes with the same color are from the same class.
Notably, as the learned condensed graphs are weighted graphs, we use black lines to denote the edges with weights larger than 0.5 and gray lines to denote the edges with weights smaller than 0.5.
From Figure 4, we can observe some patterns in the condensed graphs, e.g., the homophily patterns on Cora and Citeseer are well preserved.
Interestingly, the learned graph for Reddit is very close to a star graph where almost all the nodes only have connections with very few center nodes.
Such a structure can be meaningless for GNNs because almost all the nodes receive the information from their neighbors.
In this case, the learned features X (cid:48) play a major role in training GNN parameters, indicating that the original training graph of Reddit is not very informative, aligning with our observations in Section 4.2.
C ONCLUSION
The prevalence of large-scale graphs poses great challenges in training graph neural networks.
Thus, we study a novel problem of graph condensation which targets at condensing a large-real graph into a small-synthetic one while maintaining the performances of GNNs.
Through our proposed framework, we are able to signiﬁcantly reduce the graph size while approximating the original performance.
The condensed graphs take much less space of storage and can be used to efﬁciently train various GNN architectures.
In the future, we plan to investigate interpretable graph condensation and graph condensation for graph-level applications.
A D ATASETS AND H YPER -P ARAMETERS A. D ATASETS
We evaluate the proposed framework on three transductive datasets, i
[Table]
e., Cora , Citeseer (Kipf &
Welling, 2016) and Ogbn-arxiv (Hu et al., 2020), and two inductive datasets,
i
[Table]
e., Flickr (Zeng et al., 2020) and Reddit (Hamilton et al., 2017).
Since all the datasets have public splits, we download them from PyTorch Geometric (Fey & Lenssen, 2019) and use those splits throughout the experiments.
Dataset statistics are shown in Table 6.
[Table]
A. H YPER -P ARAMETER S ETTING
Condensation Process.
For DC, we tune the number of hidden layers in a range of { 1 , 2 , 3 } and ﬁx the number of hidden units to 256 .
We further tune the number of epochs for training DC in a range of { 100 , 200 , 400 } .
For GC OND , without speciﬁc mention, we adopt a 2-layer SGC (Wu et al., 2019a) with 256 hidden units as the GNN used for gradient matching.
The function g Φ that models the relationship between A (cid:48) and X (cid:48) is implemented as a multi-layer perceptron (MLP).
Speciﬁcally, we adopt a 3-layer MLP with 128 hidden units for small graphs ( Cora and Citeseer ) and 256 hidden units for large graphs ( Flickr , Reddit and Ogbn-arxiv ).
We tune the training epoch for
GC OND in a range of { 400 , 600 , 1000 } .
For GC OND - X , we tune the number of hidden layers in a range of { 1 , 2 , 3 } and ﬁx the number of hidden units to 256 .
We further tune the number of epochs for training GC OND - X in a range of { 100 , 200 , 400 } .
We tune the learning rate for all the methods in a range of { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } .
Furthermore, we set δ to be 0 .
01 for
Citeseer , Cora , Ogbn-arxiv , Flickr and Reddit , respectively.
For the choices of condensation ratio r , we divide the discussion into two parts.
The ﬁrst part is about transductive datasets.
For Cora and Citeseer , since their labeling rates are very small (5.2% and 3.6%, respectively), we choose r to be { 25% , 50% , 100% } of the labeling rate.
Thus, we ﬁnally choose { 1 . 3% , 2 .
6% , 5 . 2% } for Cora and { 0 .
6% } for Citeseer .
For Ogbn-arxiv , we choose r to be { 0 . 1% , 0 .
5% , 1% } of its labeling rate ( 53% ), thus being { 0 .
The second part is about inductive datasets.
As the nodes in the training graphs are all labeled in inductive datasets, we simply choose { 0 .
1% } for Flickr and 0 .
2% for Reddit .
Evaluation Process.
During the evaluation process, we set dropout rate to be 0 and weight decay to be 0 .
0005 when training various GNNs.
The number of epochs is set to 3000 for GAT while it is set to 600 for other models.
The initial learning rate is set to 0.01.
Settings for Table 3 and Table 4.
In both condensation stage and evaluation stage, we set the depth of GNNs to 2.
During condensation stage, we set weight decay to 0, dropout to 0 and training epochs to 1000.
During evaluation stage, we set weight decay to 0.0005, dropout to 0 and training epochs to 600.
A. T RAINING D ETAILS OF DC-G RAPH , GC OND - X AND GC OND
DC-Graph : During the condensation stage, DC-Graph only leverages the node features to produce condensed node features X (cid:48) .
During the training stage of evaluation, DC-Graph takes the condensed features X (cid:48) together with an identity matrix as the graph structure to train a GNN.
In the later test stage of evaluation, the GNN takes both test node features and test graph structure as input to make predictions for test nodes.
[Table]
[Table]
GC OND - X : During the condensation stage, GC OND - X leverages both the graph structure and node features to produce condensed node features X (cid:48) .
During the training stage of evaluation, GC OND - X takes the condensed features X (cid:48) together with an identity matrix as the graph structure to train a
GNN.
In the later test stage of evaluation, the GNN takes both test node features and test graph structure as input to make predictions for test nodes.
GC OND : During the condensation stage, GC OND leverages both the graph structure and node features to produce condensed graph data ( A (cid:48) , X (cid:48) ) .
During the training stage of evaluation, GC OND takes the condensed data ( A (cid:48) , X (cid:48) ) to train a GNN.
In the later test stage of evaluation, the GNN takes both test node features and test graph structure as input to make predictions for test nodes.
B M ORE E XPERIMENTS B. A BLATION S TUDY
Different Parameterization.
We study the effect of different parameterizations for modeling A (cid:48) and compare GC OND with modeling A (cid:48) as free parameters in Table 7.
From the table, we observe a signiﬁcant improvement by taking into account the relationship between A (cid:48) and X (cid:48) .
This suggests that directly modeling the structure as a function of features can ease the optimization and lead to better condensed graph data.
Joint optimization versus alternate optimization.
We perform the ablation study on joint optimization and alternate optimization when updating Φ and X (cid:48) .
The results are shown in Table 8.
From the table, we can observe that joint optimization always gives worse performance and the standard deviation is much higher than alternate optimization.
B. N EURAL A RCHITECTURE S EARCH
We focus on APPNP instead of GCN since the architecture of APPNP involves more hyperparameters regarding its architecture setup.
The detailed search space is shown as follows:
(a) Number of propagation K : we search the number of propagation K in the range of { 2 , 4 , 6 , 8 , 10 } .
(b) Residual coefﬁcient α : for the residual coefﬁcient in APPNP, we search it in the range of { 0 . 1 , 0 . 2 } .
(c) Hidden dimension: We collect the set of dimensions that are widely adopted by existing work as the candidates, i.e., { 16,32,64,128,256,512 } .
(d) Activation function: The set of available activation functions is listed as follows: { Sigmoid, Tanh, ReLU, Linear, Softplus, LeakyReLU, ReLU6, ELU }
In total, for each dataset we search 480 architectures of APPNP and we perform the search process on Cora , Citeseer and Ogbn-arxiv .
Speciﬁcally, we train each architecture on the reduced graph for epochs on as the model converges faster on the smaller graph.
We use the best validation accuracy to choose the ﬁnal architecture.
We report (1) the Pearson correlation between validation accuracies obtained by architectures trained on condensed graphs and those trained on original graphs, and (2) the average test accuracy of the searched architecture over 20 runs.
[Table]
T IME C OMPLEXITY AND R UNNING T IME
Time Complexity.
We start from analyzing the time complexity of calculating gradient matching loss, i.e., line 8 to line 11 in Algorithm 1.
Let the number of MLP layers in g Φ be L and r be the number of sampled neighbors per node.
For simplicity, we assume all the hidden units are d for all layers
and we use L -layer GCN for the analysis.
The forward process of g Φ has a complexity of
O ( N (cid:48) 2 d 2 ) .
The forward process of GCN on the original graph has a complexity of O ( r L Nd 2 ) and that on condensed graph has a complexity of O ( LN (cid:48) 2 d + LN (cid:48) d ) .
The complexity of calculating the second-order derivatives in backward propagation has an additional factor of O ( | θ t | 2 ) , which can be reduced to O ( | θ t | ) with approximated Hessian-vector products.
Although there are C iterations in line 7-11, we note that the process is easily parallelizable.
Furthermore, the process of updating θ t in line 16 has a complexity of τ θ ( LN (cid:48) 2 d + LN (cid:48) d ) .
Considering there are T iterations and K different initializations, we multiply the aforementioned complexity by KT .
To sum up, we can see that the time complexity linearly increases with number of nodes in the original graph.
Running Time.
We now report the running time of the proposed GC OND for different condensation rates.
Speciﬁcally, we vary the condensation rates in the range of { 0.1%, 0.5%, 1% } on Ogbn-arxiv and { 1%, 5%, 10% } on Cora .
The running time of 50 epochs on one single A100-SXM4 GPU is reported in Table 10. [Table] B.
[Table]
S PARSIFICATION
In this subsection
[Table]
we investigate the effect of threshold δ on the test accuracy and sparsity.
In detail [Table] we vary the values of the threshold δ used for truncating adjacency matrix in a range of { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 }
[Table]
and report the corresponding test accuracy and sparsity in Figure 3.
From the ﬁgure, we can see that increasing δ can effectively increase the sparsity of the obtained adjacency matrix without affecting the performance too much.
[Table]
[Figure]
C \ T      80.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
B. D IFFERENT D EPTH AND H IDDEN U NITS .
Depth Versus Hidden Units.
We vary the number of model layers (GCN) in a range of { 1 , 2 , 3 , 4 } and the number of hidden units in a range of { 16 , 32 , 64 , 128 , 256 } , and test them on the condensed graphs of Cora and Citeseer .
The results are reported in Table 11.
From the table, we can observe that changing the number of layers impacts the model performance a lot while changing the number of units does not.
Propagation Versus Transformation.
We further study the effect of propagation and transformation on the condensed graph.
We use Cora as an example and use SGC as the test model due to its decoupled architecture.
Speciﬁcally, we vary both the propagation layers and transformation layers of SGC in the range of { 1 , 2 , 3 , 4 , 5 } , and report the performance in Table 13.
As can be seen, the condensed graph still achieves good performance with 3 and 4 layers of propagation.
Although the condensed graph is generated under 2-layer SGC, it is able to generalize to 3-layer and 4-layer SGC.
When increasing the propagation to 5, the performance degrades a lot which could be the cause of the oversmoothing issue.
On the other hand, stacking more transformation layers can ﬁrst help boost the performance but then hurt.
Given the small scale of the graph, SGC suffers the overﬁtting issue in this case.
Cross-depth Performance.
We show the cross-depth performance in Table 12.
Speciﬁcally, we use
SGC of different depth in the condensation to generate condensed graphs and then use them to test on SGC of different depth.
Note that in this table, we set weight decay to 0 and dropout to 0.5.
We can observe that usgin a deeper GNN is not always helpful.
Stacking more layers do not necessarily mean we can learn better condensed graphs since more nodes are involved during the optimization, and this makes optimization more difﬁcult.
[Table]
[Table]
P ERFORMANCES ON O RIGINAL G RAPHS
We show the performances of various GNNs on original graphs in Table 14 to serve as references.
[Table]
N P UBMED .
We also show the experiments for Pubmed with condensation ratio of 0.3% in Table 15.
From the table, we can observe that GC OND approximates the original performance very well (77.92 vs.
79.32 on GCN).
It also generalizes well to different architectures and outperforms GC OND - X and
DC-Graph, indicating that it is important to leverage the graph structure information and learn a condensed structure.
[Table]
VISUALIZATION OF NODE FEATURES .
In addition, we provide the t-SNE (Van der Maaten & Hinton, 2008) plots of condensed node features to further understand the condensed graphs.
In Cora and Citeseer , the condensed node features are well clustered.
For Ogbn-arxiv and Reddit , we also observe some clustered pattern for the nodes within the same class.
In contrast, the condensed features are less discriminative in Flickr , which indicates that the condensed structure information can be essential in training GNN.
C A LGORITHM
We show the detailed algorithm of GC OND in Algorithm 1.
In detail, we ﬁrst set the condensed label set Y (cid:48) to ﬁxed values and initialize X (cid:48)
as node features randomly selected from each class.
In each outer loop, we sample a GNN model initialization θ from a distribution P θ .
Then, for each class we sample the corresponding node batches from T and S , and calculate the gradient matching
[Figure]
loss within each class.
The sum of losses from different classes are used to update X (cid:48) or Φ .
After that we update the GNN parameters for τ θ epochs.
When ﬁnishing the updating of condensed graph parameters, we use A (cid:48)
=
ReLU ( g Φ ( X (cid:48) ) − δ ) to obtain the ﬁnal sparsiﬁed graph structure.
do
Sample
(cid:66) detailed
Section 3.
Compute L T
L θ t c c ) and L S =
L ( GNN θ t
( A c , X c ) , Y c )
D
D θ t
T
∇ θ t
S )
if
D M ORE R ELATED W ORK
Graph pooling.
Graph pooling (Zhang et al., 2018;
Ying et al., 2018b; Gao & Ji, 2019) also generates a coarsened graph with smaller size.
Zhang et al.
(2018) is one the ﬁrst to propose an end-to-end architecture for graph classiﬁcation by incorporating graph pooling.
Later, DiffPool (Ying et al., 2018b) proposes to use GNNs to parameterize the process of node grouping.
However, those methods are majorly tailored for the graph classiﬁcation task and the coarsened graphs are a byproduct graph during the representation learning process.