Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again
Abstract
Despite the enormous success of Graph Convolutional Networks (GCNs) in modeling graph-structured data, most of the current GCNs are shallow due to the notoriously challenging problems of over-smoothening and information squashing along with conventional difficulty caused by vanishing gradients and over-fitting.
Previous works have been primarily focused on the study of over-smoothening and over-squashing phenomena in training deep GCNs.
Surprisingly, in comparison with CNNs/RNNs, very limited attention has been given to understanding how healthy gradient flow can benefit the trainability of deep GCNs.
In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow , we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA) level performance from vanilla-GCNs [1].
Next, we argue that blindly adopting the Glorot initialization for GCNs is not optimal, and derive a topologyaware isometric initialization scheme for vanilla-GCNs based on the principles of isometry.
Additionally, contrary to ad-hoc addition of skip-connections, we propose to use gradient-guided dynamic rewiring of vanilla-GCNs with skip connections.
Our dynamic rewiring method uses the gradient flow within each layer during training to introduce on-demand skip-connections adaptively.
We provide extensive empirical evidence across multiple datasets that our methods improve gradient flow in deep vanilla-GCNs and significantly boost their performance to comfortably compete and outperform many fancy state-of-the-art methods.
Codes are available at: https://github.com/VITA-Group/GradientGCN .
Introduction
Graphs are omnipresent and Graphs convolutional networks (GCNs)[1] and their variants[2, 3, 4, 5, 6, 7, 8, 9, 10] are a powerful family of neural networks which can learn from graph-structured data.
GCNs have been enormously successful in numerous real-world applications such as recommendation systems [11, 12], social and academic networks[13, 11, 5, 4, 14], modeling proteins for drug discovery[15, 16, 12], computer vision[17, 18], etc.
Despite their popularity, training deep GCNs are notoriously hard and many classical GCNs[1, 3, 19] achieve their best performance with shallow depth and completely bilk trainability with increasing stacks of layers and non-linearity[20, 21].
Previous works have focused on studying the roadblocks of training deep GCNs from the perspective of over-smoothening [20, 21] and information bottleneck [22] and many approaches broadly categorized as architectural [23, 24, 20, 25], and regularization & normalization [26, 27, 28, 29] tweaks has been proposed for mitigation.
In recent work, [30] theoretically validated that the deeper GCN model is at least as expressive as the shallow GCN model, as long as deeper GCNs are trained properly.
Furthermore, [31] studied the training difficulty of GCNs from the perspective of graph signal energy loss, and proposed modifying the GCN operator, from the energy perspective
but it failed to recover the full potential of vanilla-GCNs.
Surprisingly, unlike traditional neural networks (CNNs and RNNs), limited effort has been given to understand hurdles in the trainability of deep GCNs from the signal propagation perspective, i.e. gradient flow .
In this paper, we comprehensively show that deep GCNs suffer from poor gradient flow (error signals) under back-propagation across multiple datasets and architectures, which significantly hurt their trainability and lead to substandard performance.
Further, we are motivated to explore an orthogonal direction: Can we make vanilla-GCNs [1] go deeper and comparable/better than SOTA, by improvising healthy gradient flow during training?
In this work, we firstly look into the effect of initialization (surprisingly overlooked) in GCNs, and find that blindly adopting the Glorot initialization [32] from CNNs has critical impacts on the gradient flow of GCN training, especially for deep GCNs.
When the initial weights are not chosen appropriately, the propagation of the input signal into the layers of random weights can result in poor error signals under back-propagation [33].
Inspired from the signal propagation perspective, we leverage the principle of isometry [34, 35] to derive a theoretical initialization scheme for vanilla-GCNs, by incorporating the graph topological information - which yields a remarkably easy-to-compute form.
Skip-connections have been very resourceful to train deep convolutional networks [36, 37] by improving the gradient flow, and recently some works [23, 24, 38, 20] have identified their benefits for deep GCNs.
Although their benefits are known, most of the existing works attempt to add skipconnections in an ad-hoc and non-optimal fashion in deep GCNs which creates additional overhead of storing multiple activations during training along with substandard performance.
Secondly , we study layerwise gradient flow during the training of deep vanilla-GCNs and identify that there exist some layers which receive almost zero error signal (i.e. gradients) during backpropagation and lose trainability and expressiveness.
We use computationally efficient Gradient Flow ( p -norm of gradients within a layer) as a metric to dynamically identify such layers during training and propose a principled way of injecting skip-connections on-demand to facilitate healthy gradient flow in deep GCNs.
Our main contributions can be summarized as:
• Deep GCNs suffer from poor gradient flow during training.
We rigorously investigate the gradient flow during the training of GCNs and observe that GCNs suffer from poor gradient flow which worsens with the increase in depth.
We hypothesize that by improving the gradient flow in deep vanilla-GCNs, along with effectively improving their trainability, we can achieve SOTA-level performance from vanilla-GCNs [1].
• Topology-Aware Isometric Initialization of GCNs.
Unlike blind adoption of Glorot initialization [32] for GCNs, we derive a theoretical initialization scheme for vanilla-GCNs [1] based on the principle of isometry, which relates benign initial parameters with graph topology.
We experimentally validate that our new initialization strategy significantly improves the gradient flow in deep GCNs and provide huge performance benefits.
• Gradient-Guided Dynamic Rewiring of GCNs.
Contrary to ad-hoc addition of skipconnections to improve GCNs performance, in this paper, we leverage Gradient Flow to introduce dynamic rewiring strategy of vanilla-GCNs with skip-connections.
Our new rewiring improves gradient flow, reduces the overhead of storing multiple intermediate activations (dense, initial, residual, jumping skips), as well as outperforms many ad-hoc skip connection mechanisms.
Methodology
In this section, we aim to discuss the trainability of deep GCNs from the signal propagation perspective, i.e. gradient flow.
We will provide a detailed introduction and theoretical derivation of our topology-aware isometric initialization.
Furthermore, we will present a novel way to incorporate skip-connections using Gradient Flow to improve the trainability of deep GCNs.
2.1 Preliminaries
We begin by formulating Graph Convolutional Networks (GCN) along the way introducing our notation.
Let G =
( V , E ) be a simple and undirected graph with vertex set V and edge set E .
Let
[Figure] A ∈ R N × N be the associated adjacency matrix, such that A ij = 1 if ( i, j ) ∈ E , and A ij =otherwise.
N = |V| is the number of nodes.
We also define d i = (cid:80)
j A ij as the node degree of the i -th vertex.
Let X ∈ R C ×
N be the node feature matrix, whose i -th column represents a d -dimension feature vector for the i -th node.
GCNs aim to learn an embedding for each node via learnable graph convolutional filters [2, 1].
A graph convolutional layer can be formulated as 2 :
[Equation]
where adding I to adjacency matrix is known as the self-loop trick [1], W ∈ R C
×
C is the weight matrix of this layer, and C ′ is the dimension of output channel.
An L -layer GCN cascades L layer of graph convolutional layers followed by non-linear activations:
[Equation]
X ) , (2) where Y ∈ R C ′ is the final output of the GCN, and σ ( · ) denotes a non-linear activation, which is typically chosen as ReLU.
Although there exists lots of message-passing based Graph Neural
Networks (GNNs), which are incorporated with highly sophisticated mechanisms, such as attention [3] and gating [39, 40], the focus of our work only lies on the most classical GCN model (Eq. 2 or [1]) that exactly follows the rigorous definition of graph convolutions [41].
As we will show in our experiments, our proposal will bring this theoretically clean GCN back to the SOTA performance.
2.2 Understanding Gradient Flow in Deep GCNs
[Figure]
Gradient flow and its associated problems such as exploding and vanishing gradients have been extensively explored in the dynamics of learning of neural networks (CNNs/RNNs)
GCNs [1] are a special category of neural networks, which uses “graph convolution” operation (linear transformation of all neighbors of a node followed by non-linear activation) to learn the graph representation.
Despite being powerful in learning high-quality node representations, GCNs have limited ability to extract information from high-order neighbors and significantly lose their trainability and performance.
Recently, many works [20, 21, 22, 24] have been proposed to address this issue from the perspective of over-smoothing and information bottleneck.
However, the signal propagation perspective to understand the trainability and performance of GCNs has been highly overlooked compared to the attention it has received for CNNs/RNNs.
Gradient flow is used to study the optimization dynamics of neural networks and it has been widely known that healthy gradient flow facilitates better optimization [32, 33, 35, 42].
Consider an L -layer
GCN (see Eq. 2), we denote the weight parameter for the l -th layer f ( l ) as W l .
Suppose we have cost function L , we calculate the gradient across each layer and effective gradient flow (GF) as:
[Equation]
where for every layer l , ∂ L ∂ W l represents gradients of the learnable parameters W l and we have used p = 2 in our experiments.
Figure 2 represents the gradient flow during training of vanilla-GCNs with layer 4, 6, and 10 on the Cora dataset.
Figure 3 illustrates the comparison of validation loss and gradient flow in vanilla-GCNs with 2 and 10 layers on Cora, Citeseer
[Figure]
and Pubmed.
We consistently observed across each dataset that with increasing depth, the gradient flow across layers decreases significantly along with the performance.
In this paper, we hypothesize that by improving the gradient flow in deep vanilla-GCNs, along with effectively improving their trainability, we can achieve SOTAlevel performance from vanilla-GCNs.
We propose a new topology-aware initialization strategy based on the principles of isometry and dynamic architecture rewiring for vanilla-GCNs to improvise the gradient flow and subsequently performance (Figure 1(a),(b))
[Figure]
gradient flow in vanilla-GCNs with 2 and 10 layers.
2.3 Topology-Aware Isometric Initialization
Recent work of [43] has revealed that initializing network parameters nearly isometric can train very deep CNNs without any normalization.
We are inspired to properly initialize each GCN layer to make it an isometric mapping.
To begin with, we first give the formal definition of isometry below:
Definition 1.
[43] A map f : X → Y between two inner-product spaces is called an isometry if
[Equation]
for all x , x ′ ∈ X .
Existing isometric initialization techniques are not free lunch for GCNs.
Qi et al.
[43] showed that a convolutional operator is isometric if and only if their cross-channel weights are orthogonal to each other.
The straightforward implication is that delta kernel can be served as the isometric initialization.
However, computing delta kernel in graph signal processing needs to involve complex numbers and eigen-decomposition of the adjacency matrix [44], which is infeasible for a common deep learning framework.
To this end, we re-establish theoretical results for GCNs from Definition 1.
Consider one GCN layer (Eq. 1), our goal is to seek a weight matrix such that pair-wise angles between node features are invariant after transforming by the layer.
First of all, we denote the output feature of the i -th node as y i =
W Xa i where a i is the i -th column of ( A + I ) .
The inner product of two node features i, j ∈ [ N ] can be computed as:
[Equation]
where vec( · ) denotes vectorization of a matrix, Eq. 6 follows from the equality vec( ABC )
=
( C ⊤ ⊗ A ) vec B [45].
On the other hand, x i = I
C
Xe i where e i is the i -th canonical basis, we can obtain a similar form of ⟨ x
i , x j ⟩ :
[Equation]
To achieve isometry
[Equation]
we hope to solve a W such that Eq.
8 equals to Eq.
However, a simple closed-form solution does not exist in general.
Instead, we seek a W to minimize the difference between Eq. 8 and Eq.
[Equation]
However, minimizing objective Eq.
10 will involve data matrix into the solution, which limits computational efficiency and generalization.
Hence, we optimize a looser upper bound of Eq. 10 by minimizing the difference between ( a i a ⊤ j ) ⊗ ( W ⊤ W and ( e i e ⊤ j ) ⊗ I C .
The intuition is that once this difference reaches zero, Eq.
10 will be minimized to zero as well.
[Equation]
At the first glance, Eq. 19 is non-convex and Kronecker product will produce a computational prohibitive matrix, which makes this problem intractable.
However, it has not escaped our notice that the structure of a i a j is highly sparse.
Hence, the entire norm minimization can be decomposed into block-wise minimization problems.
We defer the detailed derivation into Appendix A. As the consequence, the optimal solution to Eq.
19 should satisfy:
[Equation]
where w k denotes the k -th column of optimal weight matrix W ∗ .
Eq.
13 suggests that isometry requires weights for different channels to be orthogonal, which conforms with the general results of [43].
But different from [43], our results indicate that the magnitude of channel-wise weights should be constrained by a degree-related constant (Eq. 12), which binds the initial weights with topological information.
To randomly initialize W in practice, one can draw each column of W from independent distributions (e.g., white Gaussian) with variance Σ 2 =
N 2 / (cid:104) C ′ (cid:80) i,j ( d i + 1)( d j + 1) (cid:105) and to initialize weights with bounded values, we suggest employ the following uniform distribution:
[Equation]
2.4 Gradient-Guided Dynamic Rewiring
Graph Convolutions can be considered as a type of Laplacian smoothing, and repeatedly applying Laplacian smoothing many times in the case of multi-layer GCNs, leads the representations of the nodes in GCN to converge to a certain value and thus become indistinguishable, i.e. lose expressiveness and trainability [20, 46].
Despite numerous efforts from architectural, regularization, and normalization perspectives [23, 26, 27, 24, 20, 25, 28, 29], there still exists a wide gap in fully understanding the trainability issue of deep GCNs which can aid in developing techniques which can prevent performance deterioration with increasing depth.
In this work, we look for the effect of increasing depth on the error signal propagation during training and found that deep GCNs suffer from poor gradient flow and it worsens dramatically with increasing depth, thereby completely blocking the gradient propagation and potentially leading to a catastrophic failure to update during training.
More specifically, we performed gradient flow analysis during training of deep vanilla-GCNs (Figure 4 (a)) and found that with the progress in training, many deep
GCN layers receive almost zero error signal during backpropagation (i.e. gradients) and unable to train.
Motivated by this finding, we use computationally efficient Gradient Flow as a metric to identify layers that block healthy error signal back-propagation during training and propose to dynamically rewire the vanilla-GCNs using skip-connections (widely known to improve gradient flow), which we call on-demand dynamic rewiring .
Our dynamic rewiring technique improves gradient flow, mitigate sudden feature collapse and significantly helps in training deep vanilla-GCNs with high performance (Figure 1(b) and 4(c)).
Mathematically, our dynamic rewiring can be written as:
[Equation]
where, 1 [ · ] is an indicator function, the subscript t denotes the training epoch, the superscript l denotes the layer index, GF denotes gradient flow, α denotes skip information ratio, and p denotes gradient flow drop threshold.
Precisely, we observe the Gradient Flow of each vanilla-GCN layer during training, and if the flow drop by p % (hyperparameter) of its initial value at the start of training, we assist the layer with a skip-connection.
We use a modified version of initial residual connection [47, 24], which we define as the output feature matrix of first layer of vanilla-GCN (gives better performance than [47, 24]), to supplement the layers that quickly loses energy.
Once the layers which are more prone to lose expressiveness are identified and rewired using the skip-connections, the vanilla-GCN training can proceed as normal.
In comparison with previous work [23, 24, 38, 20] which blindly inherits skip-connection techniques from CNNs/RNNs, our work provides a principled approach to perform architectural rewiring of vanilla-GCNs.
Static skips introduced in [23, 24, 38, 20], by default incorporate skips for training shallow GCNs with 2-3 layers (which do not require it), and tend to hurt the final performance.
For example, in Table 3, it can be observed that training a 2-layered GCN with initial, jumping, or residual skips suffers -2.1%, -0.12%, -6.37% performance drop compared to a plain vanilla-GCN (81.10%) on Cora.
Our method only introduces skip-connections when it is required by tracking the gradient flow of the layer, and thereby overcomes the issue of adding skips in shallow GCNs, and for layers that receive healthy gradient flow.
Our extensive experiments on multiple datasets reveal that our approach of guided rewiring not only brings benefits on the efficiency front (memory overhead to store intermediate activations) but also comfortably outperforms all SOTA ad-hoc skip-connection.
Experiment
In this section, we first provide experimental evidence to augment our signal propagation hypothesis and show that our newly proposed methods facilitate healthy gradient flow during the training of deep-vanilla GCNs.
Next, we extensively evaluate our methods against state-of-the-art graph neural network models and techniques to improve vanilla-GCNs on on a wide variety of open graph datasets.
[Figure]
[Table]
3.1 Dataset and Experimental Setup
We use three standard citation network datasets Cora, Citeseer, and Pubmed [48] in GNN domain for evaluating our proposed methods against state-of-the-art GNN models and techniques.
We have used the basic vanilla-GCN implementation in PyTorch provided by the authors of [1] to incorporate our proposed techniques and show their effectiveness in making traditional GCN comparable/better with
SOTA.
For our evaluation on Cora, Citeseer, Pubmed, and OGBN-ArXiv, we have closely followed the data split settings and metrics reported by the recent benchmark [49].
See details in Appendix C.
For comparison with SOTA models, we have used JKNet [50], InceptionGCN [51], SGC [52], GAT [3], GCNII [24], and DAGNN [53].
We use Adam optimizer for our experiments and performed a grid search to tune hyperparameters for our proposed methods and reported our settings in Table 1.
For all our experiments, we have trained our modified GCNs for 1500 epochs and 100 independent repetitions following [49] and reported average performances with the standard deviations of the node classification accuracies.
All experiments on large graph datasets, e.g., OGBN-ArXiv, are conducted on single 48G Quadro RTX 8000 GPU, while small graph experiments are completed using a single 16G RTX 5000 GPU.
3.2 Gradient Flow and our proposed methods
Despite significant efforts to improve deep neural networks (CNNs/RNNs) training from the signal propagation perspective, in-depth analysis of deep GCNs training with a focus on the gradient flow during back-propagation is highly overlooked.
We use Equation 4 to study gradient flow during the training of GCNs and Figure 2 indicates that with an increase in depth, gradient flow in the network drop significantly hurting the trainability of GCNs.
Figure 3 indicates the relation between gradient flow and drop in validation loss (performance) for 2-layer and 10-layer GCNs for Cora, Citeseer, and
Pubmed.
It can be observed that across all datasets, 10-layer GCN has poor gradient flow compared to 2-layer GCN and we observed a negligible drop in validation loss during training.
[Figure]
Vanilla-GCN 0.
Epoch
[Figure]
Layer
[Figure]
Layer
[Figure]
Layer Layer
Vanilla-GCN + Our Init 0.
Epoch
Layer 1 Layer 3 Layer 5 Layer
Layer
Vanilla-GCN + Gradient Rewiring 0.
To better understand the behavior of each hidden layer in deep GCN training, we estimated the gradient flow per layer independently of a 10-layer GCN.
To our surprise, we found that many layers receive zero error signals (gradients) under backpropagation, potentially leading to a catastrophic failure to update during training.
Figure 4(a)
illustrate layer-wise analysis of gradient flow in 10-layer vanilla-GCN on Cora.
We observed that within a few epochs of training, the gradient flow saturates, and then onwards, many layers of GCN receive zero error signal during the backpropagation and they fail to update leading to no drop in validation loss.
Dataset
[Table]
CiteSeer
[Table]
Initialization of neural networks is tightly coupled with the propagation of error signals [33] and effect their trainability.
Our newly proposed topology-aware isometric initialization and gradient-guided rewiring significantly improve gradient flow (Figure 1(a) and (b)) as well as mitigate the issue of feature collapse of layers in deep vanilla-GCNs.
In Figure 4(b) and (c), the detailed layer-wise analysis of vanilla-GCN with our newly proposed methods clearly illustrate how our methods are able to uniformly restore healthy gradients across all layers leading to improved trainability and significant performance gains
[Table]
75.36 Dense 77.
Normalization
BatchNorm 69.
55.98 PairNorm 74.
61.54 NodeNorm 79.
40.93 CombNorm 80.
Random Dropping
DropNode 77.
40.38 DropEdge 79.
40.50 LADIES 77.
Identity Mapping82.
Gradient
[Table]
78.51 ± 0.
± 0.
± 0. ± 0.
± 0. ± 0.
± 0. ± 0.
± 0.
3.3 Vanilla-GCNs and our proposed methods
In this section, we conduct a systematic study to understand the performance gain by improving gradient flow using our proposed methods: topology-aware isometric initialization and Gradient
Guided Dynamic Rewiring, by incorporating them into the training process of deep vanilla-GCNs.
Table 2 demonstrate the performance comparison of vanilla-GCNs [1] with and without our new initialization method with an increase in depth.
Our experiments on Cora, Citeseer, and PubMed using vanilla-GCNs with layers l ∈ { 2 , 3 , ..., 12 } illustrate how our new initialization has been successful in retaining the trainability of vanilla-GCNs with increasing depth along with improving performance at shallow depths.
The highlighted (red) box represents the depths at which vanilla-GCNs lose trainability, i.e, validation loss shows no improvement during training.
Trainability issue of deep GCNs has been extensively studied from the perspective of oversmoothening [20, 21] and information bottleneck [22] and many approaches broadly categorized as architectural tweaks [23, 24, 20, 25], and regularization & normalization [26, 27, 28, 29] has been proposed for mitigation.
We defer their detailed description to Appendix B. Table 3 demonstrates the performance comparison of our gradient-guided dynamic rewiring using skip-connections with respect to various state-of-the-art techniques to improve deep GCNs training.
It can be clearly observed that our on-demand dynamic rewiring method of introducing new skip-connections significantly outperforms all fancy normalization and regularization techniques as well as comfortably beats ad-hoc skip-connections techniques.
[Table]
To further validate the effectiveness improving gradient flow in vanilla-GCNs, we perform comparisons with previous state-of-the-art frameworks, including SGC[52], GAT[3], JKNet[50], APPNP[47],
InceptionGCN[51], GPRGNN[54], and GCNII[24].
We first observe that Gradient guided rewiring and Topology-aware isometric initialization brings orthogonal benefits and combining them helps vanilla-GCNs to achieve better performance.
Table 4 reports the mean classification accuracy and the standard deviation on the test set of Cora and Pubmed of our methods using vanilla-GCNs with layer l ∈ { 2 , 4 , 8 , 16 , 32 , 64 } .
One key benefit to note is that with the help of our methods, we can train vanilla-GCNs as deep as 64 layers.
It can be observed that vanilla-GCNs with our methods outperform all SOTA methods on Cora (layers 2 and 4) and Pubmed (layers 2, 4, 8, and 64), while holding the second position for all other layers.
To evaluate on a large graph, we have chosen
OGBN-AxRiv dataset, and Table 5 represents the performance comparison of vanilla-GCNs with our method against SOTA baselines.
It can be observed that with the help of our methods, vanilla-GCNs can be efficiently trained deeper without any significant loss in performance.
Very similar to Cora and
Pubmed, our methods beats all SOTA methods while training 2-layer vanilla-GCNs and holds second position for all other layers.
It is interesting to note that for OGBN-AxRiv dataset, vanilla-GCNs with our proposed method consistently have performance improvement with an increase in depth possibly because of being able to capture long-term information effectively, which is in contrast to most of the SOTA methods whose performance suffers with increased in depth.
Note that we are
NOT trying to beat SOTA, but instead, our objective is to reveal the correct initialization and training scheme for vanilla-GCNs, and scale deep efficiently.
The last sanity check is whether our proposed methods can make deep vanilla-GCNs effective across multiple different graph datasets.
Specially, we evaluate it on seven other open-source graph datasets: (i) one Co-author datasets [55] (CS), (ii) two Amazon datasets [55] (Computers and Photo), (iii) three [Table]
OGBN
WebKB datasets [56] (Texas, Wisconsin, Cornell), and (iv) the Actor dataset [56].
Table 6 reports the performance comparison of deep vanilla-GCN with 32-layers with state-of-the-art methods having the same depth.
Our methods universally encourage significant trainability benefits to deep vanilla-GCNs across all datasets achieving state-of-the-art performance on Computers dataset while performing in top-2 for all remaining datasets.
[Table]
3.5 Dirichlet Energy-based analysis of our proposed methods
In this section
[Table]
we use Dirichlet Energy to illustrate how our proposed techniques help in mitigating the issue of losing expressiveness, enable deep GNNs to leverage the high-order neighbors.
Node pair distance has been widely adopted to quantify the embedding similarities, and Dirichlet energy is simple and expressive metric to estimate the expressiveness of node embeddings learned by GCNs, in a topology-aware fashion [46, 57].
Following [46], we define Dirichlet Energy as:
Definition 2.
[46] Dirichlet energy E ( x ) of a scalar function x ∈ R N on the graph G is defined as: [Equation]
[Equation]
For the vector field X = [ x· · · x N ] ∈ R C × N , where x i ∈ R C , Dirichlet energy is defined as:
[Equation]
[Equation]
[Figure]
We track the Dirichlet energy of feature matrix w.r.t.
the (augmented) normalized Laplacian at different layers of the vanilla-GCN during training and observed that during training of a deep vanilla-GCN, the Dirichlet energy of some layers decreases exponentially and becomes close to zero within a few epochs of training, i.e lose expressiveness (Figure 5 row 1).
Our newly proposed methods: topology-aware isometric initialization and gradient-guided rewiring significantly improve gradient flow, as well as mitigate the issue of exponentially dropping Dirichlet energy across all the layers of deep GCNs, thereby restoring the expressiveness of node embeddings (Figure 5 row 2).
Other Related Works
GNNs have established state-of-the-art performance in numerous real-world applications [11, 12, 13, 11, 5, 4, 14, 15, 16, 12, 17, 18].
While deep architectures improve the representational power of neural networks [58, 59], not every useful GNN has to be deep.
Many real-world graph are “small-world”[60], where a node can reach any other node in a few hops, hence a few layers would suffice to provide global coverage.
However, when the graph data has no small-world property, or the related task requires long-range information, then deeper GNNs become very necessary.
Many works [20, 21] revealed that when we start stacking spatial aggregations recursively in GNNs, the node representations will collapse to indistinguishable vectors and it will hamper the training of deep GNNs.
More generally, this phenomenon happens for any message-passing mechanism via stochastic matrices including attention [61].
Recently, there has been a series of techniques developed to handle the over-smoothing issue, which can be broadly categorized under skip connection, graph normalization, random dropping.
Skip-connections [23, 24, 38, 20] have been applied to GNNs to exploit node embeddings from the preceding layers, to relieve the over-smoothing issue.
Graph normalization techniques [28, 62, 63, 64, 65] re-scale node embeddings over an input graph to constrain pairwise node distance and thus alleviate over-smoothing.
Dropout methods [66, 26, 27, 67] can be regarded as data augmentations, which help relieve both the over-fitting and over-smoothing issues in training very deep GNNs.
Conclusion
This paper makes an important step towards understanding the substandard performance of deep vanilla-GCNs from the signal propagation perspective and hypothesizes that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA) level performance or close using merely vanilla-GCNs.
This paper derives a topology-aware isometric initialization scheme for vanilla-GCNs based on the principles of isometry.
Additionally, this paper proposes to use Gradient Flow for the dynamic rewiring of vanilla-GCNs with skip-connections.
Extensive experiments across multiple datasets illustrate that these methods improvise gradient flow in deep vanilla-GCNs and significantly boost their performance.
An interesting direction for future work includes building theoretical relations between our proposed methods and gradient flow
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Derivation of Eq. 12 and
Recall that, in Section 2.3 our goal is to make initialized GCN isometric.
Instead of exactly compute such initialization by solving an optimization, we propose the following objective, which has a closed-form solution:
[Equation]
In this section, we give the detailed derivation to reach the closed-form solution.
Notice that both a i a ⊤ j and e i e ⊤ j are sparsely structured.
Let
A ( i,j ) = a i a ⊤ j and E ( i,j ) =
e i e ⊤ j , then we have:
A ( i,j ) m,n = (cid:26) 1 ( i, m ) ∈ E and ( j, n ) ∈
E
Otherwise ,
E ( i,j ) m,n = (cid:26) 1 m = i and n = j 0 Otherwise , (20) where A ( i,j ) m,n , E ( i,j ) m,n denotes the m -th row and the n -th column entry of A ( i,j ) , E ( i,j ) , respectively.
Hence, for m, n ∈ [ N ] such that ( i, m ) ∈ E and ( j, n ) ∈ E but m
̸
= i or n ̸
= j , the block-wise difference at location ( m, n ) remains W ⊤ W .
While for m =
i and n = j , the block difference is W ⊤ W − I C
as A contains self-loop (i.e., ( i, i )
∈ E and ( j, j ) ∈ E ).
Then we can simply our objective Eq.
19 as below:
[Equation]
To solve Eq.
22, we decompose the diagonal and off-diagonal components from W ⊤ W .
For off-diagonal terms, we can minimize them to zeros, thus we have w ⊤ k w l = 0 , where we note that w k denotes the k -th column of W .
For on-diagonal terms, it is equivalent to minimizing (cid:80) i,j ∈ [ N ]
[Equation]
d i + d j + d i d j ) ∥ w k ∥
[Equation]
N 2 ( ∥ w k ∥ 22 − 1) 2 for every k ∈ [ C ] .
Then combining both arguments above, the optimal solution should satisfy:
[Equation]
which reaches our final conclusion in Eq. 14.
To compute this magnitude given a graph, one needs to first compute the degree of each node, and plug them into our Eq.
The complexity to compute this value is as small as O ( N 2 ) , which is significantly lower than directly optimizing Eq.
B Description of Methods in Table
Table 3 presents a comparison of our Gradient-Guided Rewiring dynamic rewiring with respect to various fancy methods to improve the trainability of deep vanilla-GCNs [1] such as skip-connections, regularization.
Our formal description of these methods are:
B.1 Skip Connections
Skip-connections [23, 24, 38, 20] helps in alleviating the problem of over-smoothing and significantly improve the accuracy and training stability of deep GCNs.
For a L layer GCN, we can apply various type of skip-connections after certain graph convolutional layers with the current and preceding embeddings X l , 0 ≤ l ≤ L .
We have compared our method with the following four representative types of skip connections:
Residual Connection:
X l = (1 − α ) · X l + α · X l −
Initial Connection:
X l = (1 − α ) · X l + α · X3.
Dense Connection: X l = COM ( { X k , 0 ≤ k ≤ l } ) 4.
Jumping Connection:
X L = COM ( { X k , 0 ≤ k ≤ L } )
where α is residual and initial connections is a hyperparameter to weight the contribution of a node features from the current layer l and previous layers.
Jumping connection is a simplified case of dense connection and it is only applied at the end of the whole forward propagation process to combine the node features from all previous layers.
B.2 Graph Normalization
Graph normalization [28, 62, 63, 64, 65] re-scale node embeddings over an input graph to constrain pairwise node distance and thus alleviate over-smoothing.
Our investigated normalization mechanisms are formally depicted as follows:
BatchNorm: x : ,j = γ · x : ,j − E ( x : ,j ) std( x : ,j )
[Equation]
PairNorm: (cid:101) x
i = x
i − 1 n (cid:80) n i =1 x i , PairNorm ( x i ; s )
= s · (cid:101) x i ( 1 n (cid:80) ni =1 ∥ (cid:101) x i ∥ 2 2 ) 1 /3.
NodeNorm: NodeNorm ( x i ; p ) =
x i std( x i ) 1 /p
where x : ,j ∈ R N denotes the j -th row of X (the j -th channel), x i ∈ R C denotes the i -th column of X (the i -th node features), s in PairNorm is a hperparameter controlling the average pair-wise variance and p in NodeNorm denotes the normalization order.
C Dataset Details
Table 7 provided provides the detailed properties and download links for all adopted datasets.
We adopt the following benchmark datasets since i) they are widely applied to develop and evaluate GNN models, especially for deep GNNs studied in this paper; ii) they contain diverse graphs from smallscale to large-scale or from homogeneous to heterogeneous; iii) they are collected from different applications including citation network, social network, etc. Dataset Nodes Edges
[Table]
Classes
[Table]
Cora 2,
[Table] [Table]
[Table] [Table]
Citeseer
[Table] [Table] [Table] [Table]
[Table]
PubMed 19, 44, [Table] [Table] [Table]
[Table] [Table] [Table] stanford
[Table] CoauthorCS 18, 81,   https:// [Table] /shchur/gnn
[Table]
benchmark/raw/master/
[Table]
[Table] Computers 13,381 245,   https:// [Table] /shchur/gnn-benchmark/
[Table]
[Table]
[Table] [Table] Photo 7, 119,   https:// [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] Texas  1,
[Table]
[Table] [Table]
master
Wisconsin
[Table] [Table]
[Table]
master
Cornell
[Table]
[Table] [Table]
master
Actor 7, 33,
[Table] [Table]
[Figure]