D ECOUPLE G RAPH N EURAL N ETWORKS : T RAIN M ULTIPLE S IMPLE GNN S S IMULTANEOUSLY I NSTEAD OF O NE
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks (GNN) suffer from severe inefficiency due to the exponential growth of node dependency with the increase of layers.
It extremely limits the application of stochastic optimization algorithms so that the training of GNN is usually time-consuming.
To address this problem, we propose to decouple a multi-layer GNN as multiple simple modules for more efficient training, which is comprised of classical forward training ( FT ) and designed backward training ( BT ).
Under the proposed framework, each module can be trained efficiently in
FT by stochastic algorithms without distortion of graph information owing to its simplicity.
To avoid the only unidirectional information delivery of FT and sufficiently train shallow modules with the deeper ones, we develop a backward training mechanism that makes the former modules perceive the latter modules, inspired by the classical backward propagation algorithm.
The backward training introduces the reversed information delivery into the decoupled modules as well as the forward information delivery.
To investigate how the decoupling and greedy training affect the representational capacity, we theoretically prove that the error produced by linear modules will not accumulate on unsupervised tasks in most cases.
The theoretical and experimental results show that the proposed framework is highly efficient with reasonable performance, which may deserve more investigation.
I NTRODUCTION
In recent years, neural networks (Simonyan & Zisserman, 2015; He et al., 2016), due to the impressive performance, have been extended to graph data, known as graph neural networks (GNN) (Scarselli et al., 2008).
As GNNs significantly improve the results of graph tasks, it has been extensively investigated from different aspects, such as graph convolution network (GCN) (Kipf & Welling, 2017;
Niepert et al., 2016), graph attention networks (GAT) (Velickovic et al., 2018; Kreuzer et al., 2021), spatial-temporal GNN (STGNN) (Wang et al., 2020), graph auto-encoder (Kipf & Welling, 2016;
Park et al., 2019), graph contrastive learning (Hassani & Khasahmadi, 2020), etc .
Except for the variants that originate from different perspectives, an important topic is motivated by the well-known inefficiency of GNN.
In classical neural networks (He et al., 2016), the optimization is usually based on stochastic algorithms with limited batch (Duchi et al., 2011; Kingma & Ba, 2015) since samples are independent of each other.
However, the aggregation-like operations defined in (Hamilton et al., 2017) result in the dependency of each node on its neighbors and the amount of dependent nodes for one node increases exponentially with the growth of layers, which results in the unexpected increases of batch size.
Some works are proposed based on neighbor sampling (Hamilton et al., 2017;
Chen et al., 2018b;a; Zeng et al., 2020) and graph approximation (Chiang et al., 2019) to limit the batch size, while some methods (Wu et al., 2019; Zhu & Koniusz, 2021) attempt to directly apply high-order graph operation and sacrifice the most non-linearity.
The training stability is a problem for neighbor sampling methods (Hamilton et al., 2017; Chen et al., 2018b; Zeng et al., 2020) though VRGCN (Chen et al., 2018a) has attempted to control the variance via improving sampling.
Note that the required nodes may still grow (slowly) with the increase of depth.
ClusterGCN (Chiang et al., 2019) finds an approximate graph with plenty of connected components so that the batch size is strictly upper-bounded.
The major challenge of these methods is the information missing during sampling.
The simplified methods (Wu et al., 2019; Zhu & Koniusz, 2021) are efficient but the limited
To apply stochastic optimization while retaining the exact graph structure, we propose a framework, namely stacked graph neural network (SGNN), which decouples a multi-layer GNN as multiple simple
GNN modules and then trains them simultaneously rather than connecting them with the increase of the depth.
Inspired by the backward propagation algorithm, we find that the main difference between stacked networks (Vincent et al., 2010) and classical networks is no training information propagated from the latter modules to the former ones .
The lack of backward information delivery may be the main reason of the performance limitation of stacked models.
The contributions are concluded as: (1) We accordingly propose a backward training strategy to let the former modules receive the information from the final loss and latter modules, which leads to a cycled training framework to control bias and train shallow modules correctly.
(2) Under this framework, a multi-layer GNN can be decoupled into multiple simple GNNs, named as separable GNNs in this paper, so that every training step could use the stochastic optimization without any samplings or changes on graph.
Therefore,
SGNN could take both non-linearity and high efficiency into account.
(3) We investigate how the decoupling and greedy training affect the representational capacity of the linear SGNN.
It is proved that the error would not accumulate in most cases when the final objective is graph reconstruction.
B ACKGROUND
Graph Neural Networks Aiming to extend convolution operation into graph, graph convolution became a hot topic (Niepert et al., 2016; Bruna et al., 2014; Defferrard et al., 2016) and graph convolution network (GCN)
(Kipf & Welling, 2017) has become an important baseline.
By introducing self-attention techniques (Vaswani et al., 2017), graph attention networks (GAT) (Velickovic et al., 2018; Kreuzer et al., 2021) are proposed and applied to other applications (Kim et al., 2021; Guo et al., 2021).
As Li et al.
(2018) claimed that GNNs suffer from the over-smoothing problem, GALA (Park et al., 2019) develops the graph sharpening and ResGCN (Li et al., 2019) attempts to designs a deeper architecture.
The theoretical works (Li et al., 2018; Oono & Suzuki, 2020; Cong et al., 2021) have different views towards the depth of GNNs.
Some works (Li et al., 2018; Oono & Suzuki, 2020) claimed that the expressive power of GNN decreases with the increase of layers, while the others argue that the assumptions in (Oono & Suzuki, 2020) may not hold and deeper GNNs have stronger power (Cong et al., 2021).
Moreover, some works (Morris et al., 2019; Xu et al., 2019) investigate the expressive capability by showing the connection between Weisfeiler-Lehman test (Weisfeiler &
Leman, 1968) and GNNs.
Nevertheless, most of them neglect the inefficiency problem of GNNs .
Efficient Graph Neural Networks To accelerate the optimization through batch gradient descent to GNN without too much deviation, several models (Hamilton et al., 2017; Chen et al., 2018b;a;
Zeng et al., 2020) propose to sample data points according to graph topology.
These models propose different sampling strategies to obtain stable results.
GraphSAGE (Hamilton et al., 2017) produces a subgraph with limited neighbors for each node while FastGCN (Chen et al., 2018b) samples fixed nodes for each layer with the importance sampling.
The variance of sampling is further controlled in (Chen et al., 2018a).
ClusterGCN (Chiang et al., 2019) aims to generate an approximate graph with plenty of connected components so that each component can be used as a batch per step.
SGC (Wu et al., 2019) simplifies GCN by setting all activations of middle layers as linear functions and SSGC (Zhu & Koniusz, 2021) further improves it.
In summary, SGNN proposed in this paper retains the non-linearity and requires no node sampling or sub-graph sampling.
Connections to Existing Models Stacked Auto-Encoder ( SAE ) (Vincent et al., 2010) is a model applied to the pre-training of neural networks.
It trains the current two-layer auto-encoder (Hinton & Salakhutdinov, 2006) and then feeds the latent features output by the middle layer to the next auto-encoder.
The model is often used as a pre-training model instead of a formal model.
MGAE (Wang et al., 2017) is an extension of SAE and its fundamental module is graph auto-encoder (Kipf & Welling, 2016).
The main difference compared with the proposed model is whether each module could be perceived by modules from both forward and backward directions.
The stack paradigm is similar to the classical boosting models (Freund & Schapire, 1997; Friedman, 2001; Chen &
Guestrin) while some works (Schwenk & Bengio, 2000; Zhou et al., 2002) also investigated the boosting algorithm of neural networks.
In recent years, some boosting GNN models (Ivanov &
Prokhorenkova, 2021; Sun et al., 2021) are also developed.
The most boosting algorithms ( e.g. , (Schwenk & Bengio, 2000; Ivanov & Prokhorenkova, 2021)) aim to learn a prediction function gradually while the proposed SGNN aims to learn ideal embeddings gradually.
Note that AdaGCN (Sun et al., 2021) is also trained gradually and the features are combined using AdaBoost (Freund &
Schapire, 1997).
More importantly, all these boosting methods for GNNs are only trained forward and the backward training is missing .
Deep neural interface (Jaderberg et al., 2017) proposes to decouple neural networks to asynchronously accelerate the computation of gradients.
The decoupling is an acceleration trick to compute the gradients of L -layer networks , while SGNN proposed in this paper explicitly separates an L -layer GNN into L simple modules.
In other words, the ultimate goal of SGNN is not to optimize an L -layer GNN .
P ROPOSED M ETHOD
Motivated by SAE (Vincent et al., 2010) and the fact that the simplified models (Wu et al., 2019;
Zhu & Koniusz, 2021) are highly efficient for GNN, we therefore rethink the substantial difference between the stacked networks and multi-layer GNNs.
To sum up, we attempt to answer the following two questions in this paper:
Q1:
How to decouple a complex GNN into multiple simple GNNs and train them simultaneously?
Q2:
How does the decoupling affect the representational capacity and final performance?
We will discuss the first question in this section and then elaborate on another one in Section 4.
3. P RELIMINARY
Each decoupled GNN model of the proposed model is named as a module and the t -th module is denoted by M t for simplicity.
The vector and matrix are denoted by lower-case and upper-case letters in bold, respectively.
∥ · ∥ represents the Frobenius norm.
Given a graph, let A ∈ R n × n be adjacency matrix and X ∈ R n × d be node features.
A typical GNN layer can be usually defined as
[Equation]
where W is projection coefficient and P =
ϕ ( A ) is a function of A .
When we discuss each individual module, we assume that W ∈ R d × k for simplicity.
For example, GCN (Kipf & Welling, 2017) defines P as P GCN =
D − 12 ( A + I ) D − 12 and D is the degree matrix of A
+ I .
When multiple layers are integrated, the learned representation given by multiple GNN layers can be written as
[Equation]
where L is the amount of layers.
Assume that the average number of neighbors is c .
To compute H , each sample will need O ( c L ) extra samples.
If the depth is large and the graph is connected, then all nodes have to be engaged to compute for one node.
The uncontrolled batch size results in the timeconsuming training.
In vanilla GNNs, the computational complexity is O ( KL ∥
A ∥ 0 (cid:80) m − 1 i =
d i d i +1 ) on sparse graphs where K is the number of iterations and d i is the dimension of W i .
For large-scale datasets, both time and space complexity are too expensive.
3. S TACKED G RAPH N EURAL N ETWORKS
Although the stacked networks usually have more parameters than multi-layer networks, which frequently indicates that the stacked networks may be more powerful, they only serve as a technique for pre-training.
Specifically speaking, they simply transfer the representations learned by the current network to the next one but no feedback is passed back .
It causes the invisibility of the succeeding modules and the final objective.
As a result of the unreliability of the former modules, the stacked model is conventionally used as an unsupervised pre-training model.
Rethinking the learning process of a network, multiple layers are optimized simultaneously by gradient-based methods where the gradient is calculated by the well-known backward propagation algorithm
(Rumelhart et al., 1986).
The algorithm consists of forward propagation (FP) and backward propagation (BP).
FP computes the required values for BP, which can be viewed as an information delivery process.
Note that FP is similar to the training of the stacked networks.
Specifically, transferring the output of the current module to the next one in the stacked network is like the computation of neurons layer by layer during FP.
Inspired by this, we aim to design a BP-like training strategy, namely backward training (BT) , so that the former modules could be tuned according to the feedback.
The core idea of our stacked graph neural network (SGNN) is shown in Figure 1.
Before introducing SGNN in detail, we formally introduce the key concept and core motivation of how to accelerate GNN via SGNN.
Definition 3.1.
If a GNN model can be formulated as f ( A , X , W ) = f 1 ( f 0 ( A , X ) , W ) , then it is a separable GNN.
If it can be further formulated as f ( A , X , W ) = f 1 ( f 0 ( A , X ) , W )
=
g
( A , g 0 ( X , W )) , then it is a fully-separable GNN.
To keep simplicity, define the set of separable GNNs as F k = { f : R n × n × R n × d × R d × k (cid:55)→ R n × k | f
( A , X , W ) =
f 1 ( f
( A , X ) , W ) } and the set of fully-separable GNNs as F ∗ k = { f :
R n × n × R n × d × R d × k
(cid:55)→ R | f ( A , X , W ) =
f 1 ( f
( A , X ) , W ) =
g
( A , g 0 ( X , W )) } .
Note that most single-layer GNN models are fully-separable.
For instance, SGC (Wu et al., 2019) is fully-separable where f 0 ( A , X )
=
P m X and f 1 ( f
( A , X ) , W ) =
φ ( f 0 ( A , X ) · W ) , while the single-layer GIN (Scarselli et al., 2008) is separable but not fully-separable since P · MLP( X ) ̸ =
MLP( P X ) usually holds.
However, a single-layer GAT (Velickovic et al., 2018) is not separable since the graph operation is relevant to W .
The separable property actually factorizes a GNN model to 2 parts, graph operation f 0 and neural operation f 1 .
Since all dependencies among nodes in GNNs are caused by the graph operation, one can compute X ′
=
f
( A , X ) once (like preprocessing ) in separable GNNs and then the GNN is converted into a typical network.
After computing X ′ , the information contained in graph has been passed into X ′ and the succeeding sampling would not affect the topology of graph.
Therefore, we can obtain a highly efficient GNN model that can be optimized by SGD, provided that each module is separable.
On the other hand, the fully-separable condition is essential for the backward training to pass back the information over multiple modules .
Since most single-layer GNNs are separable but not fully-separable, we show how to revise separable GNNs to introduce the fully-separability .
Then, we introduce the core idea of SGNN by aiming to handle the two main problems.
Forward Training (FT)
The first challenge is how to set the training objective for each module
M t .
It is crucial to apply SGNN to both supervised and unsupervised scenes.
Suppose that we have a separable GNN module M and let H =
f ( A , X , W ) be the features learned by the separable GNN module.
For the unsupervised cases, if M t is a GAE, then the loss of FT is formulated as
[Equation]
where d ( · , · ) represents the metric function and κ : R n × k
(cid:55)→ R n × n is a mapping function.
For instance, a simple loss introduced by (Kipf & Welling, 2016) is d
( A , κ ( H ))
=
KL ( A ∥ σ ( HH T )) where σ ( · ) is the sigmoid function, and KL ( ·∥· ) is the Kullback-Leibler divergence.
The other options include but not limited to symmetric content reconstruction (Park et al., 2019) and graph contrastive learning (Hassani & Khasahmadi, 2020).
For modules with supervision information, a projection matrix R ∈ R k × c is introduced to map the k -dimension embedding vector into soft labels with c classes.
For the node classification, the loss can be simply set as
[Equation]
where Y ∈ R n × c is the supervision information for supervised tasks.
Note that the above loss is equivalent to the classical softmax regression if H is constant.
The loss could also be link prediction, graph classification, etc .
Although base modules can utilize diverse losses, we only discuss the situation that all modules use the same kind of loss in this paper for simplicity.
Algorithm 1: Procedure of Stacked Graph Neural Networks Composed of L Modules
Input : Adjacency matrix A , feature matrix X , balance coefficient η , the number of epochs K ,
L separable GNN modules {M t } Lt =1 , H 0 ← X .
Output : Features output by M L .
for i = 1 , . . .
, K do # Forward Train Stacked Graph Neural Networks for t = 1 , 2 , . . .
, L − 1 do
Feed the current features to M t and reset U t : X t ← H t − 1 , U t ← I .
# Train with only L FT at the first forward training
X t ′ =
f ( t )
( A , X t )
# Preprocessing for mini-batch algorithms
Compute loss L ( t ) ← L ( t ) FT
if i ==
1 else L ( t ) FT + η L ( t ) BT
Train M t by optimizing min W t L ( t ) based on mini-batch algorithms.
Obtain the features: H t ← f ( t ) 1 ( X t ′ , W t ) .
X L ← H t , U L ← I , X ′ L = f ( L )
( A , X L ) .
Train M L by optimizing min W L , U L L ( L ) FT based on mini-batch algorithms.
# Backward Train Stacked Graph Neural Networks for t =
L − 1 , L − 2 , . .
. , 1 do
Compute the expected output feature of M t : Z t +1 ← ( g ∗ 0 )
t +1 ( X t +1 , U t +1 ) .
Train M t by optimizing min W t , U t L ( t ) FT + η L ( t ) BT based on mini-batch algorithms.
Backward Training (BT)
The second challenge is how to train multiple separable GNNs simultaneously in order to ensure performance.
Roughly speaking, the gradients of all layers in neural networks are computed exactly due to the repeated delivery of information by FP and BP.
BP lets the shallow layers perceive the deep ones through the feedback, while the tail modules are invisible to the head ones in FT.
We accordingly design the backward training ( BT ) for SGNN.
To achieve the reverse information delivery, a separable GNN layer is modified by introducing the fully-separability ,
H =
[Equation]
A
[Equation]
F
[Equation]
F
[Equation]
(5) where U ∈ R d × d and W ∈ R d × k .
Note that f ∗ ( A , X , U ) =
g ∗ 1 ( A , g ∗ 0 ( X , U )) .
Clearly, if U is fixed as a constant, then the modified layer is equivalent to the original separable GNN layer ˆ f .
Denote Z = g ∗
( X , U ) and Z is the expected features .
Specifically, Z t is the learned feature during the backward training of M t , and it is also the expected input of M t , i.e. , from M t − 1 .
In the forward training, the delivery of information is based on the learned features H t , and Z plays the similar role in the backward training.
The loss of backward training attempts to shrink the difference between the output feature H t of M t and expected input Z t +1 of M t +1 ,
[Equation]
Note that L BT is only activated after the first forward training leading to the final loss of M t as
[Equation]
and it is updated during each backward training.
The introduction of Z will not limit the application of stochastic optimization since the expected features can also be sampled at each iteration without restrictions.
The procedure is summarized as Algorithm 1.
[Figure]
Remark that U t remains as the identity matrix during FT.
This setting leads to each forward computation across L base modules being equivalent to a forward propagation L -layer GNN.
In other words, an SGNN with L modules can be regarded as a decomposition of an L -layer GNN.
One may concern that why not to learn U t and W t together in FT.
In this case, we prefer to use U t only for learning the expected features of M t +1 and the capability improvement from the co-learning U t in FT could also be implemented by W , which is equivalent to use GIN (Xu et al., 2019) as base modules.
The complexity can be found in Appendix D due to the limitation of space.
T HEORETICAL A NALYSIS
To answer the second question raised in the beginning of Section 3, we discuss the impact of the decoupling in this section.
Intuitively speaking, if an L -layer GNN achieves satisfactory results, then there exists { W t } Lt =1 such that an SGNN with L modules could achieve the same results.
However, each M t is trained greedily according to the forward training loss L FT , while middle layers of a multi-layer GNN are trained according to the same objective.
The major concern is whether the embedding learned by a greedy strategy leads to an irreversible deviation in the forward training.
In this section, we investigate the possible side effects on a specific SGNN comprised of unsupervised modules defined in Eq.
(3) with linear activations.
The conclusion is not apparent since simply setting
W as an identity matrix does not prove it for GNN due to the existence of P .
Remark that a basic premise is that the previous module has achieved a reasonable result.
Given a linear separable-GNN module M which is defined as H
= P XW ∈ R n × k , suppose that the forward training uses the reconstruction error
∥
P − HH T ∥ as L FT .
We first introduce the matrix angle to better understand whether the preconditions of Theorem 4.1 are practicable.
Definition 4.1.
Given two matrices B 1 , B 2 ∈ R n × n , we define the matrix angle as θ ( B 1 , B 2 ) =
arccos ⟨ B 1 , B 2 ⟩ / ( ∥ B 1 ∥ · ∥ B 2 ∥ ) .
Before elaborating on theorems, we introduce the following assumption, which separates the discussions into two cases.
Assumption 4.1.
XX T does not share the same eigenspace with P − XX T .
Note that the above assumption is weak and frequently holds in most cases.
For simplicity, U r ∈ R n × r ( r ∈ N + ) is the eigenvectors associated with r leading eigenvalues.
Under this assumption, we find that the error of M is upper-bounded by ε .
Theorem 4.1.
Let δ = 1 − cos( θ ∗ / 2) and θ ∗ =
θ (( P − XX T ) Q , Q ( P − XX T )) where Q =
U o U
To − I / 2 and o = min(rank( X ) , k ) .
Under Assumption 4.1, if ∥ P − HH T ∥ = ε ≤ O ( δ ) and σ ∗ ≤ O ( √ δ ) where σ ∗ is the ( o + 1) -th largest singular value of X , then there exists W ∈ R d × k so that ∥ P − HH T ∥ ≤ ε .
In other words, if ε is small enough, then HH T could be a better approximation than XX T .
Specially, if rank( X ) ≤ k or k = d , σ ∗ = 0 so that σ ∗ ≤ O ( √ δ ) holds.
From the above theorem, we claim that the error through M will not accumulate (i.e., bound by ε ) provided that the input X , the output of the previous modules, is well-trained.
We further provide an upper-bound of error if
Assumption 4.1 does not hold.
The following theorem shows the increasing speed of error is at most linear with the tail singular-values.
Theorem
[Table]
Corollary 4.1.
Given an SGNN with L linear modules {M t } Lt =1 with L ( t ) FT =
∥
P − H t H Tt ∥ , if
P − H 1 H T 1 and H 1 H T 1 share the same eigenspace, then L ( L ) FT ≤ L (1) FT
+ (cid:80) L − 1 t =1 O ( σ 2 ∗ ( H t )) .
Based on Theorem 4.2, we conclude that the residual would not accumulate rapidly when Assumption 4.1 does not hold.
All proofs and more discussions are put in Appendix A-C.
E XPERIMENTS
In this section, we conduct experiments to investigate whether the performance of SGNN could approach the performance of the original L -layer GNN in a highly-efficient way and what the impact of the non-linearity and flexibility brought by the decoupling is .
To sufficiently answer the aboveproblems, both node clustering and semi-supervised node classification are used.
Due to the limitation of space, only 4 common datasets, including Cora, Citeseer, Pubmed (Sen et al., 2008), and Reddit (Hamilton et al., 2017).
More experiments ( e.g. , on OGB datasets) can be found in Appendix E.
5. N ODE C LUSTERING
Experimental Settings We first testify the effectiveness of SGNN on the node clustering.
We compare our method against 10 methods, including a baseline clustering model K-means, three GCN models without considering training efficiency (GAE (Kipf & Welling, 2016), ARGA (Pan et al., 2018), MGAE (Wang et al., 2017)), and six fast GCN models with GAE-loss (GraphSAGE (Hamilton et al., 2017), FastGAE (Chen et al., 2018b), ClusterGAE (Chiang et al., 2019), AGC (Zhang et al., 2019) (an unsupervised extension of SGC (Wu et al., 2019)), S 2 GC (Zhu & Koniusz, 2021), and
GAE-S 2 GC).
The used codes are based on the publicly released implementation.
To ensure fairness, all multi-layer
GNN models consist of two layers and SGNN is comprised of two modules.
For models that could be trained by stochastic algorithms, the size of mini-batch is set as 128.
The learning rate is set as 0.001 and the number of epochs is set as 100.
We set the size first layer as 128 and the second layer size as 64.
In particular, when training every module of SGNN, 20% entries of the adjacency matrix are ignored.
U is initialized as an identity matrix, and η is set as 10 3 by default.
The number of backward training is set as 5 or 10.
To investigate the effectiveness of backward training, we report the experimental results with sufficient training for each module, which is denoted by SGNN-FT, while
SGNN with the proposed backward training is represented by SGNN-BT.
To study the performance of SGNN with different base models, we choose a fully-separable GNN, S 2 GC, as the base model and this method is marked as SGNN-S 2 GC.
As S 2 GC does not use the GAE framework, we also added a competitor, namely GAE-S 2 GC, which uses S 2 GC as the encoder, to ensure fairness.
All methods are run five times and the average scores are recorded.
The result are summarized in Table 1.
[Table] [Table]
Performance From Table
[Table]
we find that SGNN outperforms in most datasets.
If the released codes could not run on Reddit due to out-of-memory (OOM), we put the notation “N/A” instead of results.
In particular, SGNN-BT obtains good improvements on Reddit with high efficiency. Specifically speaking, it is about 8% higher than the well-known GraphSAGE.
SGNN-FT performs above the average on some datasets.
It usually outperforms GraphSAGE but fails to exceed SGC.
Due to the deeper structure caused by multiple modules, the performance of SGNN excels the simple GAE.
It also outperforms SGC due to more non-linearity brought by multiple modules.
Note that S 2 GC and
SGC are strong competitors, while SGNN can easily employ them as a base module since they are separable.
From the ablation experiments, SGNN-BT works better than SGNN-FT, which indicates the necessity of the backward training.
We also investigate how the number of modules L affects the node clustering accuracy and the results averaged over 5 runs are reported in Appendix E.
[Figure]
Efficiency Figure 3 shows the consuming time of several GNNs with higher efficiency on Pubmed and Reddit.
Instead of neglecting the preprocessing operation, we measure the efficiency through a more rational way.
We record the totally consuming time after loading data into RAM and then divide the total number of updating parameters of GNNs.
The measurement could reflect the real difference of diverse training techniques aiming to apply batch-based algorithms to GNN.
It should be emphasized the reason why
SGC is worse than SGNN regarding the consuming time.
The key point is the different costs of their preprocessing operation.
For an L -order SGC, the computation cost of P L X is at least O ( ∥ A ∥
Ld ) while SGNN with L first-order modules totally requires
O ( ∥ A ∥ 0 (cid:80) Li =
1 d i ) for the same preprocessing operation.
The metric also provides a fair comparison between SGC and other models since the stopping criteria are always different.
5. N ODE C LASSIFICATION
Experimental Setting We also conduct experiments of semi-supervised classification on four datasets.
The split of datasets following (Wu et al., 2019) could be found in Appendix E. We compare
SGNN against GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), DGI (Velickovic et al., 2019), FastGCN (Chen et al., 2018b), GraphSAGE (Hamilton et al., 2017), Cluster-GCN (Chiang et al., 2019), SGC (Wu et al., 2019), GCNII (Chen et al., 2020), and S 2 GC (Zhu & Koniusz, 2021).
Similarly, we testify SGNN with two different base models, namely SGNN-BT and SGNN-S 2 GC.
For GraphSAGE, we use the mean operator by default and some notations are added if the extra operators are used.
On citation networks, the learning rate is set as 0 .
01 , while it is 10 − 4 on Reddit.
Since the nodes for training are less than 200 on citation networks, we use all training points in each iteration for all methods while we sample 256 points as a mini-batch for approaching expected features during backward training of SGNN.
On Reddit, the batch size of all batch-based models is set as 512.
We do not apply the early stopping criterion used in (Kipf & Welling, 2017) and the max iteration follows the setting of SGC.
The embedding dimensions of each module are the same as the setting in node clustering.
For the sake of fairness, we report the results obtained by SGNN with two modules using first-order operation.
The forward training loss is defined in Eq.
Moreover, all compared models share an identical implementation of their mini-batch iterators, loss function and neighborhood sampler (when applicable).
The balance coefficient of L FT and L BT is set as 1 by default.
We report the results averaged over 10 runs on citation datasets and 5 runs on Reddit in Table 2 and Table 3.
The hyper-parameters are shared for different datasets which are optimized on Cora.
Performance
The results of compared methods in Table 2 are taken from the corresponding papers.
When the experimental results are missed, we run the publicly released codes and the corresponding records are superscripted by † .
From Tables 2 and 3, we conclude that SGNN outperforms the models with neighbor sampling such as GraphSAGE, FastGCN, and ClusterGCN on citation networks and the performance of SGNN exceeds most models on Reddit.
On simple citation networks, SGNN loses the least accuracy compared with other batch-based models, which is close to GCN.
Owing to the separability of each module, the batch sampling requires no neighbor sampling and causes no loss of graph information.
Note that we simply employ the single-layer GCN as the base modules in our experiments, while some high-order methods that obtain competitive results are also available for SGNN.
Although some methods achieve preferable results, they either fail to run or obtain unsatisfactory results on large-scale datasets.
[Figure]
The order of SGC is set as L .
The backward training significantly decreases the final loss and the non-linearity also plays an important role.
Visualization to Show Impact of the Decoupling
In Figure 2, we visualize the output of a 3-module
SGNN and a 3-layer GCN to directly show that the decoupling would not cause the trivial features , which corresponds to the theoretical conclusion in Section 4.
To show the benefit of the non-linearity brought by SGNN and the backward training, the convergence curves of SGC, SGNNFT, and SGNN-BT are shown in Figure 4.
Note that the figure shows the variation of the final loss .
In SGNN, the final loss is the loss of M L , while it is the unique training loss in SGC.
SGC with
L -order graph operation is used.
From this figure, we can conclude that: (1) The non-linearity does lead to a better loss value; (2) The backward training significantly decreases the loss.
More visualization and discussions could be found in Appendix
E.
In summary, the decoupling empirically does not cause the negative impact.
C ONCLUSION AND F UTURE W ORKS
In this paper, we propose the Stacked Graph Neural Networks (SGNN).
We first decouple a multilayer GNN into multiple simple GNNs, which is formally defined as separable GNNs in our paper to ensure the availability of batch-based optimization without loss of graph information.
The bottleneck of the existing stacked models is that the information delivery is only unidirectional, and therefore a backward training mechanism is developed to make the former modules perceive the latter ones.
We also theoretically prove that the residual of linear SGNN would not accumulate in most cases for unsupervised graph tasks.
The theoretical and experimental results show that the proposed framework is more than an efficient method and it may deserve further investigation in the future.
The theoretical analysis focuses on linear SGNN and the generalization bound is also not investigated in this paper.
Therefore, they will be the core of our future work.
Moreover, as L FT could be any losses, how to choose the most appropriate loss for each module will be also a crucial topic in our future works.
A L EMMA FOR P ROOFS
Lemma A.1.
For two given symmetric matrices A and B , A and B share the same eigenspace if and only if A and B commute.
Proof.
First, if A and B share the same eigenspace, then there exists P such that
[Equation]
Accordingly, we have AB = P Λ A Λ B P T = BA .
Then, we turn to prove the converse.
If A and B commute, suppose that Au = λ u and then
[Equation]
Apply eigenvalue decomposition, we have A
=
U Λ U T .
Note that if Au 1 = λ 1 u 1 , Au 2 = λ 2 u 2 , and λ 1 ̸
= λ 2 , then u T 2 Bu 1 = 0 since Bu 1 is also an eigenvector associated with λ 1 .
Therefore,
U T BU is a block-diagonal matrix, i.e. ,
[Equation]
Apply eigendecomposition to C i ,
C i =
U i Λ i
U Ti .
(A.4)
Denote
T = 

U
U. . .
U k ,   (A.5) and V = UT , which leads to
T T U T BUT = Λ B and T T U T AUT
= Λ A (A.6) where V T V
= I .
Hence, the lemma is proved.
B P ROOF OF T HEOREM 4.
Theorem.
Let δ = 1 − cos( θ ∗ / 2) and θ ∗ =
θ (( P − XX T ) Q , Q ( P − XX T )) where Q =
U o U T o − I / 2 and o = min(rank( X ) , k ) .
Under Assumption 4.1, if ∥ P − HH T ∥ = ε ≤ O ( δ ) and σ ∗ ≤ O ( √ δ ) where σ ∗ is the ( o + 1) -th largest singular value of X , then there exists W ∈ R d × k so that ∥ P − HH T ∥ ≤ ε In other words, if ε is small enough, then HH T could be a better approximation than XX T .
Proof.
Use the notation ℓ ( · , · , · ) as the reconstruction loss
[Equation]
According to the conditions, we define
[Equation]
Apply SVD, we can factorize X as
[Equation]
where e = d − o .
Clearly, we have V To V e = 0 and thus
[Equation]
Therefore, HH T can be written as
[Equation]
Let W 0 be a valid solution as
[Equation]
By the above definition
[Equation]
V Te W =
[Equation]
Therefore
[Equation]
with rank( X ) > k , ∥
[Equation]
T
[Equation]
P ∥
[Equation]
Similarly, if rank( X ) = r ≤ k ,
[Equation]
Now we focus on the general case, rank( X ) >
k and the conclusion can be easily extended into the low-rank case.
Note that
[Equation]
and the first term can be written as
[Equation]
where
[Equation]
Due to that
[Equation]
we have
[Equation]
Let Q =
( U k U
Tk − I / 2) and s can be reformulated as
[Equation]
and we have the following definition
[Equation]
According to Lemma A.1, Assumption 4.1 indicates that
[Equation]
And therefore, (cid:112) (1 + s ) / 2 ∈ [0 , 1) .
Let δ = 1 − (cid:112) (1 + s ) / 2 = 1 − cos( θ ∗ / 2) >
and the above equation can be reformulated as
[Equation]
The second term can be formulated as
[Equation]
while the third term is
[Equation]
To sum up, the error of M is bounded as
[Equation] [Equation]
If
[Equation]
and
[Equation]
23) then ∥ HH T − P ∥ ≤ ε .
In other words, when ε ≤ O ( δ ) and σ ∗ ≤ O ( √ δ ) , the error will be bounded by ε .
For the case that rank( X ) =
r < k , it is not hard to verify that
[Equation]
As
[Equation]
and σ ∗ = 0 ≤ O ( √ δ ) , we get ∥ HH T − P ∥ ≤ ε .
Hence, we have
[Equation]
and the theorem is proved.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
C P ROOF OF T HEOREM 4.
Theorem.
If Assumption 4.1 does not hold, then there exists W ∈ R d × k so that ∥ P − HH ∥ ≤ ε + O ( σ 2 ∗ ) .
Proof.
According to Ineq.
(B.7),
[Equation]
(C.1)
Suppose that E =
U Λ U T so that P =
U ( S + Λ ) U T where S = ΣΣ T = diag( Σ 2 k ; 0 ) .
P
[Equation] [Equation]
r
[Equation]
U Te
[Equation]
=
[Equation]
( Σ
[Equation]
Then let V k W
=
( Σ 3 k + Λ k Σ k ) † ( Σ 2 k + Λ k ) 1 /∥
[Equation]
where ˆ I =
( Σ 3 k + Λ k Σ k )( Σ 3 k + Λ k Σ k ) † = Σ k
( Σ 2 k + Λ k )( Σ 2 k + Λ k ) † Σ − 1 k .
Clearly, ˆ I = 1 { Σ 2 r + Λ r ̸ = 0 } .
Therefore, Hence, the theorem is proved.
Corollary C.1.
If Assumption 4.1 does not hold and rank( X ) ≤ k , then there exists W ∈ R d × k so that ∥ P − HH ∥ ≤ ε .
[Figure] [Figure]
Proof.
Suppose that E =
U Λ U T so that P =
U ( S + Λ ) U T where S = ΣΣ T = diag( Σ 2 r ; 0 ) .
P =
U ( S + Λ ) U T
=
[ U r , U e ]
[Equation]
Let W =
[ W r ; 0 ] subjected to V Tr W r = ( Σ 3 r + Λ r Σ r ) † ( Σ 2 + Λ r ) 1 / 2 Then
[Equation]
where ˆ I =
( Σ 3 r + Λ r Σ r )( Σ 3 r + Λ r Σ r ) † = Σ
r
( Σ 2 r + Λ r )
( Σ 2 r + Λ r ) † Σ − 1 r .
Clearly, ˆ I = 1 { Σ 2 r + Λ r ̸ = 0 } .
Therefore,
[Equation]
Hence, the corollary is proved.
D C OMPLEXITY
As each base module is assumed as a separable GNN, both FT and BT of M t can be divided into two steps, the preprocessing step for graph operation and the training step for parameters learning.
Denote the output dimension of M t as d t and the dimension of original content feature as d 0 = d .
The preprocessing to compute X ′ t = f
( A , X t ) requires O ( ∥ A ∥ 0 d t ) cost.
Suppose that each module is trained E iterations and the batch size is set as m .
Then the computation cost of the training step is O
( Emd t − 1 d t ) .
Note that only the GNN mapping is considered and the computation of the loss is ignored.
Overall, the computational complexity of an SGNN with L modules is approximately
[Figure]
N M I
ACCNMI
[Figure]
O ( K ∥
A ∥ 0 (cid:80) L − 1 t =
d t + Em (cid:80)
Lt =1 d t − 1 d t ) .
Remark that the graph is only used once during every epoch and no sampling is processed on the graph such that the graph structure is completely retained which is unavailable in the existing fast GNNs.
The coefficient of ∥ A ∥ 0 is only K (cid:80)
L − 1 t =1 d t .
The space complexity is only O ( ∥ A ∥
+ md t − 1 d t ) .
Therefore, the growth of graph scale will not affect the efficiency of SGNN.
Due to the efficiency, all experiments can be conducted on a PC with an
NVIDIA 1660 (6GB) and 16GB RAM.
E M ORE E XPERIMENTS E. O MITTED I NFORMATION IN S ECTION
The details of four common datasets are shown in Table 4.
We also provide more visualizations which are not included in the main paper in Figures 5 and 6 due to the limitation of space.
We run SGNN with 3 GNN modules and visualize the input and output of M 1 , M 2 , and M 3 through t -SNE on
Cora and Citeseer, for node clustering and node classification.
The purpose of these two figures is to empirically investigate whether the decoupling would cause the accumulation of residuals and errors .
The experimental results support the theoretical results that are provided in Section 4.
One may concern the impact of η (trade-off coefficient between L FT and L B T ) on the performance.
We testify
SGNN with different η from { 10 − 5 , 10 − 3 , 10 − 1 , 10 1 , 10 3 , 10 5 } and find that η = 10 3 usually leads to good results .
Accordingly, we only report results SGNN with η = 10 3 in this paper.
Moreover, we show the impact of η to node clustering on Cora and Citeseer in Figure 8.
Moreover, we show the output of M 1 of different periods in Figure 7, in order to show the impact of the backward training.
From the figure, we find that BT indeed affects the latent features, which is particularly apparent in Figure 7d.
For SGNN-S 2 GC and GAE-S 2 GC, all extra hyper-parameters are simply set according to the original paper of S 2 GC for both node clustering and node classification.
We do not tune the hyper-parameters of S 2 GC manually.
E. E XPERIMENTS ON M ORE D ATASETS
We further show some experiments of node classification on two OGB datasets, OGB-Products and
OGB-Arxiv, which are downloaded from https://ogb.stanford.edu/docs/nodeprop/ .
The OGB-Products contains more than 2 million nodes and OGB-Arxiv contains more thanthousand nodes.
It should be emphasized that we only use the simple single-layer GCN as the base module of SGNN.
The performance can be further improved by incorporating different models such as GCNII, GIN, etc .
In particular, we only tune hyper-parameters on Arxiv, and we simply report results of SGNN with settings from Reddit.
[Table] [Table]
E. I NVESTIGATION OF D EPTH
We also conduct sufficient experiments on the depth of SGNN.
All results are reported in Table 6.
To ensure fairness, we also show the performance of GAE with the same depth though deeper GCN and GAE usually return unsatisfied results.
Note that the dimensions are set as [256 , 128 , 64 , 32 , 16 , 16 , 16] .
[Equation]
OURCE C ODE
All codes will be publicly available on our GitHub homepage after the formal publication.
The source codes are also uploaded in supplementary materials.