Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite the enormous success of Graph Convolutional Networks (GCNs) in modelling graph-structured data, most of the current GCNs are shallow due to thenotoriously challenging problems of over-smoothening and information squashingalong with conventional difficulty caused by vanishing gradients and over-fitting.
Previous works have been primarily focused on the study of over-smoothening andover-squashing phenomenon in training deep GCNs.
Surprisingly, in comparisonwith CNNs/RNNs, very limited attention has been given towards understandinghow healthy gradient flow can benefit the trainability of deep GCNs.
In this paper,firstly, we provide a new perspective of gradient flow to understand the substandardperformance of deep GCNs and hypothesize that by facilitating healthy gradientflow , we can significantly improve their trainability, as well as achieve state-of-theart (SOTA) level performance from vanilla-GCNs [1].
Next, we argue that blindlyadopting the Glorot initialization for GCNs is not optimal, and derive a topologyaware isometric initialization scheme for vanilla-GCNs based on the principlesof isometry.
Additionally, contrary to ad-hoc addition of skip-connections, wepropose to use Dirichlet Energy for dynamic rewiring of vanilla-GCNs withskip-connections.
Our dynamic rewiring method uses the expressiveness of featureembedding learned by each layer during training to introduce skip-connectionson-demand basis.
We provide extensive empirical evidence across multiple datasetsthat our methods improves gradient flow in deep vanilla-GCNs and significantlyboost their performance to comfortably compete and outperform many fancy stateof-the-art methods.
Codes are available in the supplement.
Introduction
Graphs are omnipresent and Graphs convolutional networks (GCNs)[1] and their variants[2, 3, 4, 5, 6,7, 8] are a powerful family of neural networks which can learn from graph-structured data.
GCNs havebeen enormously successful in numerous real-world applications such as recommendation systems[9, 10], social and academic networks[11, 9, 5, 4, 12], modeling proteins for drug discovery[13, 14,10], computer vision[15, 16], etc.
Despite their popularity, training deep GCNs are notoriously hardand many classical GCNs[1, 3] achieve their best performance with shallow depth and completelybilk trainability with increasing stacks of layers and non-linearity[17, 18].
Previous works have focused on studying the roadblocks of training deep GCNs from the perspectiveof over-smoothening [17, 18] and information bolltleneck [19] and many approaches broadly categorized as architectural [20, 21, 17, 22], and regularization & normalization [23, 24, 25, 26] tweaks hasbeen proposed for mitigation.
In a recent work, [27] theoretically validated that deeper GCN modelis at least as expressive as the shallow GCN model, as long as deeper GCNs are trained properly.
Furthermore, [28] studied the training difficulty of GCNs from the perspective of graph signal energy loss, and proposed modifying the GCN operator, from the energy perspective
but it failed to recoverthe full potential of vanilla-GCNs.
Surprisingly, unlike traditional neural networks (CNNs and RNNs),limited effort has been given to understand hurdles in trainability of deep GCNs from the signalpropagation perspective, i.e. gradient flow .
In this paper, we comprehensively show that deep GCNssuffer from poor gradient flow (error signals) under back-propagation across multiple datasets andarchitectures, which significantly hurt their trainability and lead to substandard performance.
Further,we are motivated to explore an orthogonal direction: Can we make vanilla-GCNs [1] go deeper andcomparable/better than SOTA, by improvising healthy gradient flow during training?
In this work, we firstly look into the effect of initialization (surprisingly overlooked) in GCNs, and findthat blindly adopting the Glorot initialization [29] from CNNs has critical impact on the gradient flowduring GCN training, specially for deep GCNs.
When the initial weights are not chosen appropriately,the propagation of the input signal into the layers of random weights can result in poor error signalsunder back-propagation [30].
Inspired from the signal propagation perspective, we leverage theprinciple of isometry [31, 32] to derive a theoretical initialization scheme for vanilla-GCNs, byincorporating the graph topological information - which yields a remarkably easy-to-compute form.
Skip-connections have been very resourceful to train deep convolutional networks [33, 34] byimproving the gradient flow, and recently some works [20, 21, 35, 17] have identified their benefitsfor deep GCNs.
Although their benefits are known, most of the existing works attempt to add skipconnections in an ad-hoc and non-optimal fashion in deep GCNs which creates additional overheadof storing multiple activations during training along with substandard performance.
Secondly , westudy layerwise gradient flow during the training of deep vanilla-GCNs and identify that thereexist some layers which receive almost zero error signal (i.e. gradients) during backpropagationand lose trainability and expressiveness.
We use computationally efficient Dirichlet Energy [36]to dynamically identify such layers during training and propose a principled way of injectingskip-connections on-demand to facilitate healthy gradient flow in deep GCNs.
Our main contributionscan be summarized as:
• Deep GCNs suffer from poor gradient flow during training.
We rigorously investigate
the gradient flow during the training of GCNs and observe that GCNs suffer from poorgradient flow which worsen with increase in depth.
We hypothesize that by improving thegradient flow in deep vanilla-GCNs, along with effectively improving their trainability,
we
can achieve SOTA-level performance from vanilla-GCNs [1].
• Topology-Aware Isometric Initialization of GCNs.
Unlike blind adoption of Glorot
initialization [29] for GCNs, we derive a theoretical initialization scheme for vanilla-GCNs
[1] based on the principle of isometry, which relates benign initial parameters with graph
topology.
We experimentally validate that our new initialization strategy significantly
improves the gradient flow in deep GCNs and provide huge performance benefits.
• Dirichlet Energy Guided Dynamic Rewiring of GCNs.
Contrary to ad-hoc addition ofskip-connections to improve GCNs performance, in this paper, we leverage Dirichlet Energy
[36] (which measures the expressiveness of feature embedding) to introduce dynamic
rewiring strategy of vanilla-GCNs with skip-connections.
Our new strategy improves
gradient flow, reduces the overhead of storing multiple intermediate activations (dense,residual, jumping skips), as well as outperforms many ad-hoc skip connection mechanisms.
Methodology
In this section, we aim to discuss the trainability of deep GCNs from the signal propagation perspective, i.e. gradient flow.
We will provide a detailed introduction and theoretical derivation of ourtopology-aware isometric initialization.
Furthermore, we will present a novel way to incorporateskip-connections using Dirichlet Energy to improve the gradient flow and trainability of deep GCNs.
2.1 Preliminaries
We begin by formulating Graph Convolutional Networks (GCN) along the way introducing ournotation.
Let G =
( V , E ) be a simple and undirected graph with vertex set V and edge set E .
Let
A ∈ R N × N be the associated adjacency matrix, such that A ij = 1 if ( i, j ) ∈ E , and A ij = 0otherwise.
N = |V| is the number of nodes.
We also define d i = (cid:80)
j A ij as the node degree of the
[Figure]
i -th vertex.
Let X ∈ R C ×
N be the node feature matrix, whose i -th column represents a d -dimensionfeature vector for the i -th node.
GCNs aim to learn an embedding for each node via learnable graphconvolutional filters [2, 1].
A graph convolutional layer can be formulated as 1 :
[Equation]
where adding I to adjacency matrix is known as the self-loop trick [1], W ∈ R C
×
C is the weightmatrix of this layer, and C ′ is the dimension of output channel.
An L -layer GCN cascades L layer ofgraph convolutional layers followed by non-linear activations:
[Equation]
where Y ∈ R C ′ is the final output of the GCN, and σ ( · ) denotes a non-linear activation, whichis typically chosen as ReLU.
Althrough there exists lots of message-passing based Graph Neural
Networks (GNNs), which incorporate with more sophisticate mechanisms, such as attention [3]and gating [37, 38], the focus of our work only lies on the most classical GCN model (Eq. 2 or[1]) that exactly follows the rigorous definition of graph convolutions [39].
As we will show in ourexperiments
[Figure]
our proposal will bring this theoretical clean GCN back to the SOTA performance
[Figure]
2.2 Understanding Gradient Flow in Deep GCNs
[Figure]
Gradient flow and its associated problems such as ex-
[Figure]
ploding and vanishing gradients have been extensivelyexplored in the dynamics of learning of neural networks(CNNs/RNNs)
GCNs [1] are a specialcategory of neural networks, which uses “graph convolution” operation (linear transformation of all neighborsof a node followed by non-linear activation) to learnthe graph representation.
Despite being powerful inlearning high-quality node representations, GCNs havelimited ability to extract information from high-orderneighbors and significantly lose their trainability andperformance.
Recently, many works [17, 18, 19, 21]have been proposed to address this issue from the perspective of over-smoothing and information bottleneck.
However, the signal propagation perspective to understand the trainability and performanceof GCNs has been highly overlooked compared to the attention it has received for CNNs/RNNs.
Gradient flow is used to study the optimization dynamics of neural networks and it has been widelyknown that healthy gradient flow facilitates better optimization [29, 30, 32, 40].
Consider an L -layer
GCN (see Eq. 2), we denote the weight parameter for the l -th layer f ( l ) as W l .
Suppose we havecost function L , we calculate the gradient across each layer and effective gradient flow (GF) as:
[Equation]
where for every layer l , ∂ L ∂ W l represents gradients of the learnable parameters W l and we have usedp = 2 in our experiments.
Figure 3 represents the gradient flow during training of of vanilla-GCNswith layer 4, 6, and 10 on Cora dataset.
Figure 2 illustrate the comparison of validation loss andgradient flow in vanilla-GCNs with 2 and 10 layers on Cora, Citeseer, and Pubmed.
We consistentlyobserved across each dataset that with increasing depth, the gradient flow across layers decreasessignificantly along with the performance.
In this paper, we hypothesize that by improving the gradientflow in deep vanilla-GCNs, along with effectively improving the their trainability, we can achieve
SOTA-level performance from vanilla-GCNs.
We propose a new topology-aware initializationstrategy based on the principles of isometry and dynamic architecture rewiring for vanilla-GCNsusing Dirichlet energy to improvise the gradient flow and subsequently performance (Figure 1(a),(b)).
2.3 Topology-Aware Isometric Initialization
Recent work of [41] has revealed that initializing network paramters near isometric can train verydeep CNNs without any normalization.
We are inspired to properly initialize each GCN layer tomake it an isometric mapping.
To begin with, we first give the formal definiton of isometry as below:
Definition 1.
[41] A map f : X → Y between two inner-product spaces is called an isometry if
[Equation]
for all x , x ′ ∈ X .
Existing isometric initialization techniques are not free lunch for GCNs.
Qi et al.
[41] showed that aconvolutional operator is isometric if and only if their cross-channel weights are orthogonal to eachother.
The straightforward implication is that delta kernel can be served as the isometric initialization.
However, computing delta kernel in graph signal processing needs to involve complex numbers andeigen-decomposition of the adjacency matrix [42], which is infeasible for a common deep learningframework.
To this end, we re-establish theoretical results for GCNs from Definition 1.
Consider one GCN layer (Eq. 1), our goal is to seek a weight matrix such that pair-wise anglesbetween node features are invariant after transforming by the layer.
First of all, we denote the outputfeature of the i -th node as y i =
W Xa i where a i is the i -th column of ( A + I ) .
The inner productof two node features i, j ∈ [ N ] can be computed as:
[Equation]
where vec( · ) denotes vectorization of a matrix, Eq. 6 follows from the equality vec( ABC ) =(
C ⊤ ⊗ A ) vec B [43].
On the other hand, x i = I
C
Xe i where e i is the i -th canonical basis, we canobtain a similar form of ⟨ x i ,
[Equation]
[Equation]
To achieve isometry
[Equation]
we hope to solve a W such that Eq.
8 equals to Eq.
However, a simpleclosed-form solution does not exist in general.
Instead
[Equation]
we seek a W to minimize the differencebetween Eq. 8 and Eq. 7.
[Equation]
However, minimizing objective Eq.
10 will involve data matrix into the solution, which limitscomputational efficiency and generalization.
Hence, we optimize a looser upper bound of Eq.
10 byminimizing the difference between ( a i a ⊤ j ) ⊗ ( W ⊤ W and ( e i e ⊤ j ) ⊗ I C .
The intuition is that oncethis difference reaches zero, Eq.
10 will be minimized to zero as well.
[Equation]
At the first glance, Eq. 17 is non-convex and Kronecker product will produce a computationalprohibitive matrix, which makes this problem intractable.
However, it has not escaped our notice thatthe structure of a i a j is highly sparse.
Hence, the entire norm minimization can be decomposed intoblock-wise minimization problems.
We defer the detailed derivation into the Appendix 1.
As theconsequence, the optimal solution to Eq.
17 should satisfy:
[Equation]
where w k denotes the k -th column of optimal weight matrix W ∗ .
Eq.
13 suggests that isometryrequires weights for different channels to be orthogonal, which conforms with the general results of[41].
But different from [41], our results indicate that the magnitude of channel-wise weights shouldbe constrained by a degree-related constant (Eq. 12), which binds the initial weights with topologicalinformation.
To randomly initialize W in practice, one can draw each column of W from independentdistributions (e.g., white Gaussian) with variance Σ 2 =
N 2 / (cid:104) C ′ (cid:80) i,j ( d i + 1)( d j + 1) (cid:105) , where C ′is the number of output channels (a.k.a.
fan out).
To initialize weights with bounded values, wesuggest employ the following uniform distribution:
[Equation]
2.4 Dirichlet Energy Guided Rewiring
Graph Convolutions can be considered as a type of Laplacian smoothing, and repeatedly applying
Laplacian smoothing many times in the case of multi-layer GCNs, leads features of the nodes inthe graph to converge to similar values, i.e. lose expressiveness and trainability [17].
Recently, [36]argued to use Dirichlet energy to measure the expressiveness of node embeddings learned by GCNs.
Dirichlet Energy is computationally efficient since it is calculated across the connected pair of nodesand can be an indicator of expressiveness of learned node embeddings during the training.
We track the Dirichlet energy of feature matrix w.r.t.
the (augmented) normalized Laplacian atdifferent layers of the vanilla-GCN during training.
Surprisingly, we observed that during trainingof a deep vanilla-GCN, the Dirichlet energy of some layers decreases exponentially and becomesclose to zero within a few epochs of training, i.e lose expressiveness.
Detailed gradient flow analysisof these layers (Figure 4 and 5 (a)) further indicate that they receive almost zero error signalduring backpropagation (i.e. gradients) and unable to train.
Motivated by this finding, we usecomputationally efficient Dirichlet energy to identify layers that block healthy gradient flow duringtraining and propose to dynamically rewire the vanilla-GCNs using skip-connections (widely knownto improve gradient flow), which we call on-demand dynamic rewiring .
Our dynamic rewiringtechnique improves gradient flow, retains Dirichlet Energy, and significantly helps in training deepvanilla-GCNs with high performance (Figure 1(b) and 4).
In our work, Dirichlet energy is defined as:
Definition 2.
[36] Dirichlet energy E ( x ) of a scalar function x ∈ R N on the graph G is defined as:
[Equation]
[Figure]
For the vector field X = [ x· · · x N ] ∈ R C × N , where x i ∈ R C , Dirichlet energy is defined as:
[Equation]
More specifically, we observe the Dirichlet energy of each vanilla-GCN layer during training, and ifthe energy drop by p % (hyperparameter) of its initial value at the start of training, we assist the layerwith a skip-connection.
We use a modified version of initial residual connection [44, 21], whichwe define as the output feature matrix of first layer of vanilla-GCN (gives better performance than[44, 21]), to supplement the layers that quickly loses energy.
Mathematically, our dynamic rewiringcan be written as X l = ⊮
[ E ( X ( l t ) ) <
pE ( X ( l 0 ) )]
αX (1) + ˜ X l where ⊮
[ · ] is an indicator function, tdenotes the training epoch, and
˜ X
l = W l − 1 X ( l − 1)
( A + I ) .
Once the layers which are more prone tolose expressiveness are identified and rewired using the skip-connections, the vanilla-GCN trainingcan proceed as normal.
In comparison with previous work [20, 21, 35, 17] which blindly inheritsskip-connection techniques from CNNs/RNNs, our work provides a principled approach to performarchitectural rewiring of vanilla-GCNs.
Our extensive experiments on multiple datasets reveal thatour approach of guided rewiring not only brings benefits on the efficiency front (memory overhead tostore intermediate activations) but also comfortably outperforms all SOTA adhoc skip-connection.
Experiment
In this section, we first provide experimental evidence to augment our signal propagation hypothesisand show that our newly proposed methods facilitate healthy gradient flow during the training ofdeep-vanilla GCNs.
Next, we extensively evaluate our methods against state-of-the-art graph neuralnetwork models and techniques to improve vanilla-GCNs on on a wide variety of open graph datasets. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
3.1 Dataset and Experimental Setup
We use three standard citation network datasets Cora, Citeseer, and Pubmed [45] in GNN domain forevaluating our proposed methods against state-of-the-art GNN models and techniques.
We have usedthe basic vanilla-GCN implementation in PyTorch provided by the authors of [1] to incorporate ourproposed techniques and show their effectiveness in making traditional GCN comparable/better with
SOTA.
For our evaluation on Cora, Citeseer, Pubmed, and OGBN-ArXiv, we have closely followedthe data split settings and metrics reported by the recent benchmark [46].
For comparison with
SOTA models, we have used JKNet [47], InceptionGCN [48], SGC [49], GAT [3], GCNII [21], and
DAGNN [50].
We use Adam optimizer for our experiments and performed a grid search to tune hyperparameters for our proposed methods and reported our settings in Table 1.
For all our experiments,we have trained our modified GCNs for 1500 epochs and 100 independent repetitions following [46]and reported average performances with the standard deviations of the node classification accuracies.
All experiments on large graph datasets, e.g., OGBN-ArXiv, are conducted on single 48G Quadro
RTX 8000 GPU, while small graph experiments are completed using a single 16G RTX 5000 GPU.
3.2 Gradient Flow and our proposed methods
Despite significant efforts to improve deep neural networks (CNNs/RNNs) training from the signalpropagation perspective, in-depth analysis of deep GCNs training with a focus on the gradient flowduring back-propagation is highly overlooked.
We use Equation 4 to study gradient flow during thetraining of GCNs and Figure 3 indicates that with increase in depth, gradient flow in the networkdrop significantly hurting the trainability of GCNs.
Figure 2 indicates the relation between gradientflow and drop in validation loss (performance) for 2-layer and 10-layer GCNs for Cora, Citeseer, and
Pubmed.
It can be observed that across all datasets, 10-layer GCN has poor gradient flow comparedto 2-layer GCN and we observed a negligible drop in validation loss during training.
[Figure]
To better understand the behavior of each hidden layer in deep GCN training, we estimated thegradient flow per layer independently of a 10-layer GCN.
To our surprise, we found that many layersreceive zero error signals (gradients) under backpropagation, potentially leading to a catastrophicfailure to update during training.
Figure 5(a) illustrate layer-wise analysis of gradient flow in 10-layervanilla-GCN on Cora.
We observed that within a few epochs of training, the gradient flow saturates,and then onwards, many layers of GCN receive zero error signal during the backpropagation and theyfail to update leading to no drop in validation loss.
This conforms very well with our observation in
Figure 4 that within a few epochs of training, Dirichlet Energy of many layers drop to zero and theylose expressiveness.
[Table]
Initialization of neural networks is tightly coupled with the propagation of error signals [30] andeffect their trainability.
Our newly proposed topology-aware isometric initialization and Dirichlet
Energy guided rewiring significantly improve gradient flow (Figure 1(a) and (b)) as well as mitigatethe issue of exponentially dropping expressiveness of layers in deep GCNs (Figure 4).
In Figure 5(b)and (c), the detailed layer-wise analysis of vanilla-GCN with our newly proposed methods clearlyillustrate how our methods are able to uniformly restore healthy gradients across all layers leading toimproved trainability and significant performance gains.
3.3 Vanilla-GCNs and our proposed methods
In this section, we conduct a systematic study to understand the performance gain by improvinggradient flow using our proposed methods: topology-aware isometric initialization and Dirichlet
Energy Guided Dynamic Rewiring, by incorporating them into the training process of deep vanillaGCNs.
Table 2 demonstrate the performance comparison of vanilla-GCNs [1] with and withoutour new initialization method with increase in depth.
Our experiments on Cora, Citeseer, and
PubMed using vanilla-GCNs with layers l ∈ { 2 , 3 , ..., 12 } illustrate how our new initializationhas been successful in retaining the trainability of vanilla-GCNs with increasing depth along withimproving performance at shallow depths.
The highlighted (red) box represents the depths at whichvanilla-GCNs lose trainability, i.e, validation loss shows no improvement during training.
[Table]
Trainability issue of deep GCNs has been extensively studied from the perspective of oversmoothening [17, 18] and information bottleneck [19] and many approaches broadly categorized asarchitectural tweaks [20, 21, 17, 22], and regularization & normalization [23, 24, 25, 26] has beenproposed for mitigation.
Table 3 demonstrate the performance comparison of our Dirichlet Energyguided dynamic rewiring using skip-connections with respect to various state-of-the-art techniquesto improve deep GCNs training.
It can be clearly observed that our on-demand dynamic rewiringmethod of introducing new skip-connections significantly outperforms all fancy normalization andregularization techniques as well as comfortably beats ad-hoc skip-connections techniques.
[Table]
3.4 Comparison with state-of-the-art methods
To further validate the effectiveness improving gradient flow in vanilla-GCNs, we perform comparisons with previous state-of-the-art frameworks, including SGC[49], GAT[3], JKNet[47], APPNP[44],
InceptionGCN[48], GPRGNN[51], and GCNII[21].
We first observe that Dirichlet Energy guidedrewiring and Topology-aware isometric initialization brings orthogonal benefits and combining themhelps vanilla-GCNs to achieve better performance.
Table 4 reports the mean classification accuracyand the standard deviation on the test set of Cora and Pubmed of our methods using vanilla-GCNswith layer l ∈ { 2 , 4 , 8 , 16 , 32 , 64 } .
One key benefit to note is that with the help of our methods,we can train vanilla-GCNs as deep as 64 layers.
It can be observed that vanilla-GCNs with ourmethods outperform all SOTA methods on Cora (layers 2 and 4) and Pubmed (layers 2, 4, 8, and 64),while holding the second position for all other layers.
To evaluate on a large graph, we have chosen
OGBN-AxRiv dataset, and Table 5 represents the performance comparison of vanilla-GCNs with ourmethod against SOTA baselines.
It can be observed that with the help of our methods, vanilla-GCNscan be efficiently trained deeper without any significant loss in performance.
Very similar to Cora and
Pubmed, our methods beats all SOTA methods while training 2-layer vanilla-GCNs and holds secondposition for all other layers.
Note that we are NOT trying to beat SOTA, but instead our objective isto reveal the correct initialization and training scheme for vanilla-GCNs, and scale deep efficiently.
[Table]
The last sanity check is whether our proposed methods can make deep vanilla-GCNs effective acrossmultiple different graph datasets.
Specially, we evaluate it on seven other open-source graph datasets:(i) one Co-author datasets [52] (CS), (ii) two Amazon datasets [52] (Computers and Photo), (iii) three
WebKB datasets [53] (Texas, Wisconsin, Cornell), and (iv) the Actor dataset [53].
Table 6 reports theperformance comparison of deep vanilla-GCN with 32-layers with state-of-the-art methods having thesame depth.
Our methods universally encourage significant trainability benefits to deep vanilla-GCNsacross all datasets achieving state-of-the-art performance on Computers dataset while performing intop-2 for all remaining datasets.
[Table]
the-art frameworks.
Other Related Works
Deep Graph Neural Networks.
GNNs have established state-of-the-art performance in numerousreal-world applications [9, 10, 11, 9, 5, 4, 12, 13, 14, 10, 15, 16].
While deep architectures imporvesthe representational power of neural networks, not every useful GNN has to be deep.
Many real-worldgraph are "small-world"[54], where a node can reach any other node in a few hops, hence a few layerswould suffice to provide the global coverage.
However, when the graph data doesn’t have small-worldproperties, or the related task requires long-range information, then deeper GNNs become verynecessary.
Many works [17, 18] revealed that when we start stacking spatial aggregations recursivelyin GNNs, the node representations will collapse to indistinguishable vectors and it will hamper thetraining of deep GNNs.
Recently, there has been a series of techniques developed to handle theover-smoothing issue, which can be broadly categorized under skip connection, graph normalization,random dropping.
Skip-connections [20, 21, 35, 17] have been applied to GNNs to exploit nodeembeddings from the preceding layers, to relieve the over-smoothing issue.
Graph normalizationtechniques [25, 55, 56, 57, 58] re-scale node embeddings over an input graph to constrain pairwisenode distance and thus alleviate over-smoothing.
Dropout methods [59, 23, 24, 60] can be regardedas data augmentation methods, which help relieve both the over-fitting and over-smoothing issues intraining very deep GNNs.
Conclusion
This paper makes an important step towards understanding the substandard performance of deepvanilla-GCNs from the signal propagation perspective and hypothesizes that by facilitating healthygradient flow, we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA)level performance or close using merely vanilla-GCNs.
This paper derives a topology-aware isometricinitialization scheme for vanilla-GCNs based on the principles of isometry.
Additionally, this paperproposes to use Dirichlet Energy for the dynamic rewiring of vanilla-GCNs with skip-connections.
Extensive experiments across multiple datasets illustrate that these methods improvise gradient flowin deep vanilla-GCNs and significantly boost their performance.
An interesting direction for futurework includes building theoretical relation between our proposed methods and gradient flow.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Derivation of Eq. 12 and 13
Recall that, in Section 2.3 our goal is to make initialized GCN isometric.
Instead of exactly computesuch initialization by solving an optimization, we propose the following objective, which has aclosed-form solution:
[Equation]
In this section, we give the detailed derivation to reach the closed-form solution.
Notice that botha i a ⊤ j and e i e ⊤ j are sparsely structured.
Let
A ( i,j ) =
a i a ⊤ j and E ( i,j ) =
e i e ⊤ j , then we have:
A ( i,j ) m,n = (cid:26) 1 ( i, m ) ∈ E and ( j, n ) ∈
{+E 
+}
Otherwise ,
E ( i
[Equation]
m
[Equation]
n
[Equation]
(cid:26)
[Equation]
m
[Equation]
{+i and n = j 
+}
Otherwise
[Equation]
(18) where A ( i,j ) m,n , E ( i,j ) m,n denotes the m -th row and the n -th column entry of A ( i,j ) , E ( i,j ) , respectively.
Hence, for m, n ∈ [ N ] such that ( i, m ) ∈ E and ( j, n ) ∈ E but m
̸
= i or n ̸
= j , the block-wisedifference at location ( m, n ) remains W ⊤ W .
While for m =
i and n = j , the block differenceis W ⊤ W − I C
as A contains self-loop (i.e., ( i, i )
∈ E and ( j, j ) ∈ E ).
Then we can simply ourobjective Eq.
17 as below:
[Equation]
To solve Eq. 20, we decompose the diagonal and off-diagonal components from W ⊤ W .
Foroff-diagonal terms
, we can minimize them to zeros, thus we have w ⊤ k w l = 0 , where we notethat w k denotes the k -th column of W .
For on-diagonal terms, it is equivalent to minimizing 526 (cid:80) i,j ∈ [ N ]
[Equation]
d i + d
[Equation]
+ d i d
[Equation]
∥
[Equation]
k ∥ 42 + N
[Equation]
∥ w k ∥ 22 − 1) 2 for every k ∈
[ C ] .
Then combining botharguments above, the optimal solution should satisfy:
[Equation]
which reaches our final conclusion in Eq. 14.
To compute this magnitude given a graph, one needs tofirst compute the degree of each node, and plug them into our Eq. 12.
The complexity to computethis value is as small as O ( N 2 ) , which is significantly lower than directly optimizing Eq.
B Description of Methods in Table 3:
Table 3 presents comparison of our DE-Guided Rewiring dynamic rewiring with respect to variousfancy methods to improve the trainability of deep vanilla-GCNs [1] such as as skip-connections,regularization.
Our formal description of these methods are:
B.1 Skip Connections
Skip-connections [20, 21, 35, 17] helps in allevating the problem of over-smoothing and significantlyimprove the accuracy and training stability of deep GCNs.
For a L layer GCN, we can apply varioustype of skip-connections after certain graph convolutional layers with the current and precedingembeddings X l , 0 ≤ l ≤ L .
We have compared our method with following four representative typesof skip connections:
Residual Connection:
{+X l = (1 − α ) · X l + α · X l − 
+}
Initial Connection:
X l = (1 − α ) · X l + α · X 03.
Dense Connection: X l = COM ( { X k , 0 ≤ k ≤ l } )4.
Jumping Connection:
X L = COM ( { X k , 0 ≤ k ≤ L } )
where α is residual and initial connections is a hyperparameter to weight the contribution of a nodefeatures from the current layer l and previous layers.
Jumping connection is a simplified case ofdense connection and it is only applied at the end of whole forward propagation process to combinethe node features from all previous layers.
B.2 Graph Normalization
Graph normalization [25, 55, 56, 57, 58] re-scale node embeddings over an input graph to constrainpairwise node distance and thus alleviate over-smoothing.
Our investigated normalization mechanismsare formally depicted as follows:
BatchNorm: x ( k ) =
γ · x ( k ) − E
[ x ( k ) ]
std ( x ( k ) )
+ β2.
PairNorm:
˜ x i = x
i − 1 n (cid:80) ni
=1 x i , PairNorm ( x i ; s ) =
[Equation]
[Equation]
n (cid:80) ni
[Equation]
∥
[Equation]
∥
[Equation]
NodeNorm: NodeNorm ( x i ; p ) =
{+x i std ( x i ) 1 / 
+}
where s in PairNorm is a hperparameter controlling the average pair-wise variance and p in NodeNormdenotes the normalization order.
B.3 Dataset Details
[Figure]
Table 7 provided provides the detailed properties and download links for all adopted datasets.
Weadopt the following benchmark datasets since i) they are widely applied to develop and evaluate
GNN models, especially for deep GNNs studied in this paper; ii) they contains diverse graphs fromsmall-scale to large scale or from homogeneous to heterogeneous; iii) they are collected from differentapplications including citation network, social network, etc.
[Figure]
DownloadLinks
Cora 2, 5, 1,  https://github
[Table]
com/kimiyoung/planetoid/
[Table]
/master/data
Citeseer 3, 4, 3,  https://github.com/kimiyoung/planetoid/raw/master/data
PubMed 19,
[Table]
[Table]
https://github
[Table]
com/kimiyoung/planetoid/
[Table]
/master/data
OGBN
[Table]
[Table]
https://ogb.stanford.edu/
[Table]
[Table]
https://github
[Table]
com/shchur/gnn
[Table]
benchmark/raw/master/data/npz/
Computers 13,
[Table]
[Table]
[Table]
/shchur/
[Table]
[Table]
/master/
[Table]
/npz/
Photo 7,
[Table]
[Table]
[Table]
/shchur/
[Table]
benchmark/
[Table]
master/
[Table]
/npz/
[Table]
[Table]
[Table]
Wisconsin
[Table]
[Table]
[Table]