P AIR N ORM : T ACKLING O VERSMOOTHING IN GNN S
Anonymous authors
Paper under double-blind review
A BSTRACT
The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers.
This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable.
We take a closer look at two different interpretations, aiming to quantify oversmoothing.
Our main contribution is P AIR N ORM , a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar.
What is more,
P AIR N ORM is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN.
Experiments on real-world graphs demonstrate that P AIR N ORM makes deeper GCN, GAT, and
SGC models more robust against oversmoothing, and signiﬁcantly boosts performance for a new problem setting that beneﬁts from deeper GNNs.
I NTRODUCTION
Graph neural networks (GNNs) is a family of neural networks that can learn from graph structured data.
Starting with the success of GCN (Kipf & Welling, 2017) on achieving state-of-the-art performance on semi-supervised classiﬁcation, several variants of GNNs have been developed for this task; including GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018), SGC (Wu et al., 2019), and GMNN (Qu et al., 2019) to name a few most recent ones.
A key issue with GNNs is their depth limitations.
It has been observed that deeply stacking the layers often results in signiﬁcant drops in performance for GNNs, such as GCN and GAT, even beyond just a few (2–4) layers.
This drop is associated with a number of factors; including the vanishing gradients in back-propagation, overﬁtting due to the increasing number of parameters, as well as the phenomenon called oversmoothing.
Li et al.
(2018) was the ﬁrst to call attention to the oversmoothing problem.
Having shown that the graph convolution is a type of Laplacian smoothing , they proved that after repeatedly applying
Laplacian smoothing many times, the features of the nodes in the (connected) graph would converge to similar values—the issue coined as “ oversmoothing ”.
In effect, oversmoothing hurts classiﬁcation performance by causing the node representations to be indistinguishable across different classes.
Later, several others have alluded to the same problem (Xu et al., 2018; Klicpera et al., 2019;
Rong et al., 2019; Li et al., 2019) (See § 5 Related Work).
In this work, we address the oversmoothing problem in deep GNNs.
Speciﬁcally, we propose (to the best of our knowledge) the ﬁrst normalization layer for GNNs that is applied in-between intermediate layers during training.
Our normalization has the effect of preventing the output features of distant nodes to be too similar or indistinguishable, while at the same time allowing those of connected nodes in the same cluster become more similar.
We summarize our main contributions as follows
• Normalization to Tackle Oversmoothing in GNNs: We introduce a normalization scheme, called P AIR N ORM , that makes GNNs signiﬁcantly more robust to oversmoothing and as a result enables the training of deeper models without sacriﬁcing performance.
Our proposed scheme capitalizes on the understanding that most GNNs perform a special form of Laplacian smoothing, which makes node features more similar to one another.
The key idea is to ensure that the total pairwise feature distances remains a constant across layers, which in turn leads to distant pairs having less similar features, preventing feature mixing across clusters.
• Speed and Generality: P AIR N ORM is very straightforward to implement and introduces no additional parameters.
It is simply applied to the output features of each layer (except the last one) consisting of simple operations, in particular centering and scaling, that are linear in
the input size.
Being a simple normalization step between layers, P AIR N ORM is not speciﬁc to any particular GNN but rather applies broadly.
In this work we use P AIR N ORM to tackle oversmoothing for the GCN, GAT, and SGC models.
• Use Case for Deeper GNNs: While P AIR N ORM prevents performance from dropping signiﬁcantly with increasing number of layers, it does not necessarily yield increased performance in absolute terms.
We ﬁnd that this is because shallow architectures with no more than 2–4 layers is sufﬁcient for the often-used benchmark datasets in the literature.
In response, we motivate a real-world scenario wherein a notable portion of the nodes have no feature vectors.
In such settings, nodes beneﬁt from a larger range (i.e., neighborhood, hence a deeper GNN) to “recover” effective feature representations.
Through extensive experiments, we show that GNNs employing our P AIR N ORM signiﬁcantly outperform the ‘vanilla’ GNNs when deeper models are beneﬁcial to the classiﬁcation task.
All of our source code and datasets are shared publicly at http://bit.ly/PairNorm .
U NDERSTANDING O VERSMOOTHING
In this work, we consider the semi-supervised node classiﬁcation (SSNC) problem on a graph.
In the general setting, a graph G =
( V , E , X ) is given in which each node i ∈ V is associated with a feature vector x i ∈ R d where X
=
[ x 1 , . . .
, x n ] T denotes the feature matrix, and a subset V l ⊂ V of the nodes are labeled, i.e. y i ∈ { 1 , . . .
, c } for each i ∈ V l where c is the number of classes.
Let
A ∈ R n × n be the adjacency matrix and D = diag ( deg 1 , . .
, deg n ) ∈ R n × n be the degree matrix of G .
Let ˜A =
A + I and ˜D =
D
+ I denote the augmented adjacency and degree matrices with added self-loops on all nodes, respectively.
Let ˜A sym = ˜D − 1 / 2 ˜A ˜D − 1 / 2 and ˜A rw = ˜D − 1 ˜A denote symmetrically and nonsymmetrically normalized adjacency matrices with self-loops.
The task is to learn a hypothesis that predicts y i from x i that generalizes to the unlabeled nodes
V u = V\V l .
In Section 3.2, we introduce a variant of this setting where only a subset F ⊂ V of the nodes have feature vectors and the rest are missing.
We show that SSNC on such graphs necessitates deeper GNNs with a larger number of layers, where tackling the oversmoothing problem becomes particularly beneﬁcial.
2. T HE O VERSMOOTHING P ROBLEM
Although GNNs like GCN and GAT achieve state-of-the-art results in a variety of graph-based tasks, these models are not very well-understood, especially why they work for the SSNC problem where only a small amount of training data is available.
The success appears to be limited to shallow
GNNs, where the performance gradually decreases with the increasing number of layers.
This decrease is often attributed to three contributing factors: (1) overﬁtting due to increasing number of parameters, (2) difﬁculty of training due to vanishing gradients, and (3) oversmoothing due to many graph convolutions.
Among these, perhaps the least understood one is oversmoothing, which indeed lacks a formal deﬁnition.
In their analysis of GCN’s working mechanism, Li et al.
(2018) showed that the graph convolution of GCN is a special form of Laplacian smoothing.
The standard form being ( I − γ I ) X + γ ˜A rw X , the graph convolution lets γ = 1 and uses the symmetrically normalized Laplacian to obtain ˜X
= ˜A sym X , where the new features ˜x of a node is the weighted average of its own and its neighbors’ features.
This smoothing allows the node representations within the same cluster become more similar, and in turn helps improve SSNC performance under the cluster assumption (Chapelle et al., 2006).
However when GCN goes deep, the performance can suffer from oversmoothing where node representations from different clusters become mixed up.
Let us refer to this issue of node representations becoming too similar as node-wise oversmoothing.
Another way of thinking about oversmoothing is as follows.
Repeatedly applying Laplacian smoothing too many times would drive node features to a stationary point, washing away all the information from these features.
Let x · j ∈ R n denote the j -th column of X .
Then, for any x · j ∈ R n :
[Equation]
and
[Equation]
where the normalized solution π ∈ R n satisﬁes π i =
[Equation]
i
[Equation]
i for all i ∈ [ n ] .
Notice that π is independent of the values x · j of the input feature and is only a function of the graph structure (i.e., degree).
In other words, (Laplacian) oversmoothing washes away the signal from all the features, making them indistinguishable.
We will refer to this viewpoint as feature-wise oversmoothing.
To this end we propose two measures, row-diff and col-diff, to quantify these two types of oversmoothing.
Let H ( k ) ∈
R n × d be the representation matrix after k graph convolutions, i.e.
H ( k ) =
˜A k sym X .
Let h ( k ) i ∈
R d be the i -th row of H ( k ) and h ( k ) · i ∈ R n be the i -th column of
H ( k ) .
Then we deﬁne row-diff( H ( k ) ) and col-diff( H ( k ) ) as follows.
[Equation]
The row-diff measure is the average of all pairwise distances between the node features (i.e., rows of the representation matrix) and quantiﬁes node-wise oversmoothing, whereas col-diff is the average of pairwise distances between ( L 1 -normalized 1 ) columns of the representation matrix and quantiﬁes feature-wise oversmoothing.
2. S TUDYING O VERSMOOTHING WITH SGC
Although oversmoothing can be a cause of performance drop with increasing number of layers in
GCN, adding more layers also leads to more parameters (due to learned linear projections W ( k ) at each layer k ) which magnify the potential of overﬁtting.
Furthermore, deeper models also make the training harder as backpropagation suffers from vanishing gradients.
In order to decouple the effect of oversmoothing from these other two factors, we study the oversmoothing problem using the SGC model (Wu et al., 2019).
(Results on other GNNs are presented in § 4.)
SGC is simpliﬁed from GCN by removing all projection parameters of graph convolution layers and all nonlinear activations between layers.
The estimation of SGC is simply written as:
[Equation]
where K is the number of graph convolutions, and W ∈ R d × c denote the learnable parameters of a logistic regression classiﬁer.
Note that SGC has a ﬁxed number of parameters that does not depend on the number of graph convolutions (i.e. layers).
In effect, it is guarded against the inﬂuence of overﬁtting and vanishing gradient problem with more layers.
This leaves us only with oversmoothing as a possible cause of performance degradation with increasing K .
Interestingly, the simplicity of SGC does not seem to be a sacriﬁce; it has been observed that it achieves similar or better accuracy in various relational classiﬁcation tasks (Wu et al., 2019).
[Figure]
in color) SGC’s performance (dashed lines) with increasing graph convolutions ( K ) on Cora dataset (train/val/test split is 3%/10%/87%).
For each K , we train SGC in 500 epochs, save the model with the best validation accuracy, and report all measures based on the saved model.
Measures row-diff and col-diff are computed based on the ﬁnal layer representation of the saved model.
(Solid lines depict after applying our method P AIR N ORM , which we discuss in § 3.2.)
Dashed lines in Figure 1 illustrate the performance of SGC on the Cora dataset as we increase the number of layers ( K ).
The training (cross-entropy) loss monotonically increases with larger K , potentially because graph convolution mixes node representations with their neighbors’ and makes them less distinguishable (training becomes harder).
On the other hand, graph convolutions (i.e., smoothing) improve generalization ability, reducing the gap between training and validation/test loss up to K = 4 , after which (over)smoothing begins to hurt performance.
The row-diff and col-diff both continue decreasing monotonically with K , providing supporting evidence for oversmoothing.
T ACKLING O VERSMOOTHING 3. P ROPOSED P AIR N ORM
We start by establishing a connection between graph convolution and an optimization problem, that is graph-regularized least squares (GRLS), as shown by NT & Maehara (2019).
Let ¯X ∈ R n × d be a new node representation matrix, with ¯x i ∈
R d depicting the i -th row of ¯X .
Then the GRLS problem is given as
[Equation]
where (cid:107) z i (cid:107) 2 ˜D =
z Ti ˜Dz i .
The ﬁrst term can be seen as total degree-weighted least squares.
The second is a graph-regularization term that measures the variation of the new features over the graph structure.
The goal of the optimization problem can be stated as estimating new “denoised” features ¯x i ’s that are not too far off of the input features x i ’s and are smooth over the graph structure.
The GRLS problem has a closed form solution ¯X = (2 I − ˜A rw ) − 1 X , for which ˜A rw X is the ﬁrstorder Taylor approximation, that is ˜A rw X ≈ ¯X .
By exchanging ˜A rw with ˜A sym we obtain the same form as the graph convolution, i.e., ˜X
=
˜A sym X ≈ ¯X .
As such, graph convolution can be viewed as an approximate solution of (5), where it minimizes the variation over the graph structure while keeping the new representations close to the original.
The optimization problem in (5) facilitates a closer look to the oversmoothing problem of graph convolution.
Ideally, we want to obtain smoothing over nodes within the same cluster, however avoid smoothing over nodes from different clusters.
The objective in (5) dictates only the ﬁrst goal via the graph-regularization term.
It is thus prone to oversmoothing when convolutions are applied repeatedly.
To circumvent the issue and fulﬁll both goals simultaneously, we can add a negative term such as the sum of distances between disconnected pairs as follows.
[Equation]
where λ is a balancing scalar to account for different volume and importance of the two goals.
By deriving the closed-form solution of (6) and approximating it with ﬁrst-order Taylor expansion, one can get a revised graph convolution operator with hyperparameter λ .
In this paper, we take a different route.
Instead of a completely new graph convolution operator, we propose a general and efﬁcient “patch”, called P AIR N ORM , that can be applied to any form of graph convolution having the potential of oversmoothing.
Let ˜X (the output of graph convolution) and ˙X respectively be the input and output of P AIR N ORM .
Observing that the output of graph convolution ˜ X = ˜
A sym X only achieves the ﬁrst goal, P AIR N ORM serves as a normalization layer that works on ˜X to achieve the second goal of keeping disconnected pair representations farther off.
Speciﬁcally, P AIR N ORM normalizes ˜X such that the total pairwise squared distance TPSD ( ˙X ) :=
[Equation] i,j ∈ [ n ] [Equation]
˙x i − ˙x
j (cid:107) 22 is the same as TPSD ( X ) .
That is,
[Equation]
By keeping the total pairwise squared distance unchanged , the term (cid:80) ( i,j ) / ∈E (cid:107) ˙x
i − ˙x
j (cid:107) 22 is guaranteed to be at least as large as the original value (cid:80) ( i,j ) / ∈E (cid:107) x
i − x j (cid:107) 22 since the other term
[Equation] ˜x
[Equation] ˜x j[-(cid:107) 
 ≈ (cid:80) ( i,-]
[Equation]
22 is shrunk through the graph convolution.
In practice, instead of always tracking the original value TPSD ( X ) , we can maintain a constant TPSD value C across all layers, where C is a hyperparameter that could be tuned per dataset.
To normalize ˜ X to constant TPSD, we need to ﬁrst compute TPSD ( ˜ X ) .
Directly computing TPSD involves n 2 pairwise distances that is O ( n 2 d ) , which can be time consuming for large datasets.
Equivalently, normalization can be done via a two-step approach where TPSD is rewritten as
[Equation]
The ﬁrst term (ignoring the scale 2 n 2 ) in Eq.
(8) represents the mean squared length of node representations, and the second term depicts the squared length of the mean of node representations.
To simplify the computation of (8), we subtract the row-wise mean from each ˜x
i , i.e., ˜x ci =
˜x i − 1 n (cid:80) ni
˜x i where ˜x ci denotes the centered representation.
Note that this shifting does not affect the TPSD, and furthermore drives the term (cid:107) 1 n (cid:80) ni
=
˜x i (cid:107) 22 to zero, where computing TPSD ( ˜X ) boils down to calculating the squared Frobenius norm of ˜X c and overall takes O ( nd ) .
That is,
[Equation]
In summary, our proposed P AIR N ORM (with input ˜X and output ˙X ) can be written as a two-step, center-and-scale, normalization procedure:
[Equation]
(Scale) (11) After scaling the data remains centered, that is, (cid:107) (cid:80) ni
=
˙x i (cid:107) 22 = 0 .
In Eq.
(11), s is a hyperparameter that determines C .
Speciﬁcally,
[Equation]
Then, ˙X := P AIR N ORM ( ˜X ) has row-wise mean
(i.e., is centered) and constant total pairwise squared distance C = 2 n 2 s 2 .
An illustration of P AIR N ORM is given in Figure 2.
The output of P AIR N ORM is input to the next convolution layer.
[Figure]
[Figure]
We also derive a variant of P AIR N ORM by replacing (cid:80) ni =1 (cid:107)
˜x ci (cid:107) 22 in Eq.
[Equation]
[Equation] with n (cid:107) ˜x ci (cid:107) 22 , such that the scaling step computes ˙x
i
[Equation]
We call it P AIR N ORM - SI (for S cale
I ndividually), which imposes more restriction on node representations, such that all have the same L 2 -norm s .
In practice we found that both P AIR N ORM and P AIR N ORM - SI work well for SGC, whereas P AIR N ORM - SI provides better and more stable results for GCN and
GAT.
The reason why GCN and GAT require stricter normalization may be because they have more parameters and are more prone to overﬁtting.
In all experiments, we employ P AIR N ORM for SGC and P AIR N ORM - SI for both GCN and GAT.
P AIR N ORM is effective and efﬁcient in solving the oversmoothing problem of GNNs.
As a general normalization layer, it can be used for any GNN.
Solid lines in Figure 1 present the performance of SGC on Cora with increasing number of layers, where we employ P AIR N ORM after each graph 3 See Appendix A 1 for the detailed derivation. convolution layer, as compared to ‘vanilla’ versions.
Similarly, Figure 3 is for GCN and GAT (P AIR N ORM is applied after the activation of each graph convolution).
Note that the performance decay with P AIR N ORM -at-work is much slower.
(See Fig.s 5–6 in Appx.
A.3 for other datasets.)
While P AIR N ORM enables deeper models that are more robust to oversmoothing, it may seem odd that the overall test accuracy does not improve.
In fact, the benchmark graph datasets often used in the literature require no more than 4 layers, after which performance decays (even if slowly).
In the next section, we present a realistic use case setting for which deeper models are more likely to provide higher performance, where the beneﬁt of P AIR N ORM becomes apparent.
3. A C ASE W HERE D EEPER GNN S ARE B ENEFICIAL
In general, oversmoothing gets increasingly more severe as the number of layers goes up.
A task would beneﬁt from employing P AIR N ORM more if it required a large number of layers to achieve its best performance.
To this effect we study the “missing feature setting”, where a subset of the nodes lack feature vectors.
Let M ⊆
V u be the set where ∀ m ∈ M , x m = ∅ , i.e., all of their features are missing.
We denote with p = |M| /
|V u | the missing fraction.
We call this variant of the task as semi-supervised node classiﬁcation with missing vectors (SSNC-MV).
Intuitively, one would require a larger number of propagation steps (hence, a deeper GNN) to be able to “recover” effective feature representations for these nodes.
SSNC-MV is a general and realistic problem that ﬁnds several applications in the real world.
For example, the credit lending problem of identifying low- vs. high-risk customers (nodes) can be modeled as SSNC-MV where a large fraction of nodes do not exhibit any meaningful features (e.g., due to low-volume activity).
In fact, many graph-based classiﬁcation tasks with the cold-start issue (entity with no history) can be cast into SSNC-MV.
To our knowledge, this is the ﬁrst work to study the SSNC-MV problem using GNN models.
Figure 4 presents the performance of SGC, GCN
[Figure]
and GAT models on Cora with increasing number of layers, where we remove feature vectors from all the unlabeled nodes,
i
[Figure]
e. p = 1 .
The models with P AIR N ORM achieve a higher test accuracy compared to those without, which they typically reach at a larger number of layers.
(See Fig.
7 in Appx.
A.4 for results on other datasets.)
Note that we have not explored the full range of possible settings that could beneﬁt from deeper GNN models.
We leave the exploration of other use cases for which P AIR N ORM could yield performance boost as a possible future direction
[Figure]
E XPERIMENTS
In section 3 we have shown the robustness of P AIR N ORM -enhanced models against increasing number of layers in SSNC problem.
In this section we design extensive experiments to evaluate the effectiveness of P AIR N ORM under the SSNC-MV setting, over SGC, GCN and GAT models.
4. E XPERIMENT
S ETUP
Datasets.
We use 4 well-known benchmark datasets in GNN domain: Cora , Citeseer , Pubmed (Sen et al., 2008), and CoauthorCS (Shchur et al., 2018).
Their statistics are reported in Appx.
A.2.
For Cora , Citeseer and Pubmed , we use the same dataset splits as Kipf & Welling (2017), where all nodes outside train and validation are used as test set.
For CoauthorCS , we randomly split all nodes into train/val/test as 3% / 10% / 87% , and keep the same split for all experiments.
Models.
We use three different GNN models as our base model: SGC (Wu et al., 2019), GCN (Kipf & Welling, 2017), and GAT (Velickovic et al., 2018).
We compare our P AIR N ORM with residual connection method (He et al., 2016) over base models (except SGC since there is no “residual connected” SGC), as we surprisingly ﬁnd it can slow down oversmoothing and beneﬁt SSNCMV problem.
Similar to us, residual connection is a general technique that can be applied to any model without changing its architecture.
We focus on the comparison between the base models and P AIR N ORM -enhanced models, rather than achieving the state of the art performance for SSNC and SSNC-MV.
There exist a few other work addressing oversmoothing (Klicpera et al., 2019; Li et al., 2018;
Rong et al., 2019; Xu et al., 2018) however they design specialized architectures and not simple “patch” procedures like P AIR N ORM that can be applied on top of any GNN.
Hyperparameters.
We choose the hyperparameter s of P AIR N ORM from { 0 . 1 , 1 , 10 , 50 , 100 } over validation set for SGC, while keeping it ﬁxed at s = 1 for both GCN and GAT due to resource limitations.
We set the #hidden units of GCN and GAT (#attention heads is set to 1) to 32 andrespectively for all datasets.
Dropout with rate 0 .
6 and L 2 regularization with penalty 5 · 10 − 4 are applied to GCN and GAT.
For SGC, we vary number of layers in { 1 , 2 , . . .
. , 60 } and for
GCN and GAT in { 2 , 4 , . .
Conﬁgurations.
For P AIR N ORM -enhanced models, we apply P AIR N ORM after each graph convolution layer (i.e., after activation if any) in the base model.
For residual-connected models with t skip steps, we connect the output of l -th layer to ( l + t ) -th, that is, H ( l + t ) new
= H ( l + t )
+ H ( l ) where
H ( l ) denotes the output of l -th graph convolution (after activation).
For the SSNC-MV setting, we randomly erase p fraction of the feature vectors from nodes in validation and test sets (for which we input vector 0 ∈ R d ), whereas all training (labeled) nodes keep their original features (See 3.2).
We run each experiment within 1000 epochs 5 times and report the average performance.
We mainly use a single GTX-1080ti GPU, with some SGC experiments ran on an Intel i7-8700k CPU.
4. E XPERIMENT R ESULTS
We ﬁrst show the global performance gain of applying P AIR N ORM to SGC for SSNC-MV under varying feature missing rates as shown in Table 1.
P AIR N ORM -enhanced SGC performs similar or better over 0% missing, while it signiﬁcantly outperforms vanilla SGC for most other settings, especially for larger missing rates.
#L denotes the best number of layers for the model that yields the largest average validation accuracy (over 5 runs), for which we report the average test accuracy (Acc).
Notice the larger #L values for SGC-PN compared to vanilla SGC, which shows the power of P AIR N ORM for enabling “deep” SGC models by effectively tackling oversmoothing.
Similar to Wu et al.
(2019) who showed that the simple SGC model achieves comparable or better performance as other GNNs for various tasks, we found P AIR N ORM -enhanced SGC to follow the same trend when compared with P AIR N ORM -enhanced GCN and GAT, for all SSNC-MV settings.
Due to its simplicity and extreme efﬁciency, we believe P AIR N ORM -enhanced SGC sets a strong baseline for the SSNC-MV problem. problem, with missing rate ranging from 0% to 100% .
Showing test accuracy at # L ( K in Eq. 4) layers, at which model achieves best validation accuracy
[Table]
We next employ P AIR N ORM - SI for GCN and GAT under the same setting [Table] comparing it with the residual (skip) connections technique.
Results are shown in Table 2 and Table 3 respectively for
GCN and GAT.
Due to space and resource limitations, we only show results for 0% and 100% missing rate scenarios.
(We provide results for other missing rates ( 70 , 80 , 90 %) over 1 run only in Appx.
A.5.)
We observe similar trend for GCN and GAT: (1) vanilla model suffers from performance drop under SSNC-MV with increasing missing rate; (2) both residual connections and P AIR N ORM - SI enable deeper models and improve performance (note the larger #L and Acc); (3) GCN-PN and
GAT-PN achieve performance that is comparable or better than just using skips; (4) performance can be further improved (albeit slightly) by using skips along with P AIR N ORM - SI .
[Table] [Table]
R ELATED W ORK
Oversmoothing in GNNs: Li et al.
(2018) was the ﬁrst to call attention to the oversmoothing problem.
Xu et al.
(2018) introduced Jumping Knowledge Networks, which employ skip connections for multi-hop message passing and also enable different neighborhood ranges.
Fey (2019) proposed
Just-Jump that makes a improvement based on Xu et al.
Klicpera et al.
(2019) proposed a propagation scheme based on personalized Pagerank that ensures locality (via teleports) which in turn prevents oversmoothing.
Li et al.
(2019) built on ideas from ResNet to use residual as well as dense connections to train deep GCNs.
DropEdge Rong et al.
(2019) proposed to alleviate oversmoothing through message passing reduction via removing a certain fraction of edges at random from the input graph.
These are all specialized solutions that introduce additional parameters and/or a different network architecture.
Normalization Schemes for Deep-NNs:
There exist various normalization schemes proposed for deep neural networks, including batch normalization Ioffe & Szegedy (2015), weight normalization
Salimans & Kingma (2016), layer normalization Ba et al.
(2016), and so on.
Conceptually these have substantially different goals (e.g., reducing training time), and were not proposed for graph neural networks nor the oversmoothing problem therein.
Important difference to note is that larger depth in regular neural-nets does not translate to more hops of propagation on a graph structure.
C ONCLUSION
We investigated the oversmoothing problem in GNNs and proposed P AIR N ORM , a novel normalization layer that boosts the robustness of deep GNNs against oversmoothing.
P AIR N ORM is fast to compute, requires no change in network architecture nor any extra parameters, and can be applied to are and layers improves generalization ability problem Figure lines any GNN.
Experiments on real-world classiﬁcation tasks showed the effectiveness of P AIR N ORM , where it provides performance gains when the task beneﬁts from more layers.
Future work will explore other use cases of deeper GNNs that could further showcase P AIR N ORM ’s advantages.
A A PPENDIX A. D ERIVATION OF E Q .
TPSD ( ˜X ) = (cid:88) i,j ∈ [ n ] (cid:107)
˜x i − ˜x j (cid:107) {+
+}
= (cid:88) i,j ∈ [ n ] ( ˜x i − ˜x j ) T
( ˜x i − ˜x j ) (13) =
(cid:88) i,j ∈ [ n ] ( ˜x Ti ˜x i + ˜x
Tj ˜x j − 2 ˜x
Ti ˜x j ) (14) =
2 n (cid:88) i ∈ [ n ] ˜x Ti ˜x i − 2 (cid:88)
i,j ∈ [ n ] ˜x Ti ˜x j (15) = 2 n (cid:88) i ∈ [ n ] (cid:107)
˜x i (cid:107) 22 − 2 1 T ˜X˜X T(16)
= 2 n (cid:88) i ∈ [ n ] (cid:107)
˜x i (cid:107) 22 − 2 (cid:107) 1 T ˜X (cid:107)(17) =
2 n 2 (cid:18)n n (cid:88) i =
(cid:107)
˜x i (cid:107) 22 − (cid:107) 1 n n (cid:88) i
=
˜x i (cid:107)(cid:19) .
A. D ATASET S TATISTICS
[Table]
A. A DDITIONAL P ERFORMANCE P LOTS WITH I NCREASING N UMBER OF L AYERS
[Figure] [Figure]
improved robustness to performance decay due to oversmoothing with increasing number of layers.
[Figure]
A. A DDITIONAL P ERFORMANCE P LOTS WITH I NCREASING N UMBER OF L AYERS UNDER SSNC-MV WITH p =
[Figure]
A. A DDITIONAL E XPERIMENTS UNDER SSNC-MV WITH I NCREASING M ISSING F RACTION p
In this section we report additional experiment results under the SSNC-MV setting with varying missing fraction, in particular p = { 0 . 7 , 0 . 8 , 0 .
9 , 1 } and also report the base case where p = 0 for comparison.
Figure 8 presents results on all four datasets for GCN vs. P AIR N ORM -enhanced GCN (denoted
PN for short).
The models without any skip connections are denoted by *-0, with one-hop skip connection by *-1, and with one and two-hop skip connections by *-2.
Barcharts on the right report the best layer that each model produced the highest validation accuracy, and those on the left report the corresponding test accuracy.
Figure 9 presents corresponding results for GAT.
We discuss the take-aways from these ﬁgures on the following page.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
We make the following observations based on Figures 8 and 9:
• Performance of ‘vanilla’ GCN and GAT models without skip connections (i.e., GCN-0 and GAT-0) drop monotonically as we increase missing fraction p .
• P AIR N ORM -enhanced ‘vanilla’ models (PN-0, no skips) perform comparably or better than GCN-0 and GAT-0 in all cases, especially as p increases.
In other words, with P AIR N ORM at work, model performance is more robust against missing data.
• Best number of layers for GCN-0 as we increase p only changes between 2-4.
For GAT-0, it changes mostly between 2-6. • P AIR N ORM -enhanced ‘vanilla’ models (PN-0, no skips) can go deeper,
i
e., they can leverage a larger range of #layers (2-12) as we increase p .
Speciﬁcally, GCN-PN-0 (GAT-PN-0) uses equal number or more layers than GCN-0 (GAT-0) in almost all cases.
• Without any normalization, adding skip connections helps—GCN/GAT-1 and GCN/GATare better than GCN/GAT-0, especially as we increase p .
• With P AIR N ORM but no-skip, performance is comparable or better than just adding skips.
• Adding skips on top of P AIR N ORM does not seem to introduce any notable gains.
In summary, simply employing our P AIR N ORM for GCN and GAT provides robustness against oversmoothing that allows them to go deeper and achieve improved performance under SSNC-MV.
[Table]
[Figure]
A. C ASE STUDY : ADDITIONAL M EASURES FOR P AIR N ORM AND P AIR N ORM -SI WITH SGC AND GCN
To better understand why P AIR N ORM and P AIR N ORM -SI are helpful for training deep GNNs, we report additional measures for (SGC and GCN) with (P AIR N ORM and P AIR N ORM -SI) over the
Cora dataset.
In the main text, we claim TPSD (total pairwise squared distances) is constant across layers for SGC with P AIR N ORM (for GCN/GAT this is not guaranteed because of the inﬂuence of activation function and dropout layer).
In this section we empirically measure pairwise (squared) distances for both SGC and GCN, with P AIR N ORM and P AIR N ORM -SI.
A.6. SGC WITH P AIR N ORM AND P AIR N ORM -SI
To verify our analysis of P AIR N ORM for SGC, and understand how the variant of P AIR N ORM (P AIR N ORM -SI) works, we measure the average pairwise squared distance (APSD) as well as the average pairwise distance (APD) between the representations for two categories of node pairs: (1) connected pairs (nodes that are directly connected in graph) and (2) random pairs (uniformly randomly chosen among the node set).
APSD of random pairs reﬂects the TPSD, and APD of random pairs reﬂects the total pairwise distance (TPD).
Under the homophily assumption of the labels w.r.t.
the graph structure, we want APD or APSD of connected pairs to be small while keeping APD or
APSD of random pairs relatively large.
The results are shown in Figure 10.
Without normalization, SGC suffers from fast diminishing APD and APSD of random pairs.
As we have proved, P AIR N ORM normalizes APSD to be constant across layers, however it does not normalize APD, which appears to decrease linearly with increasing number of layers.
Surprisingly, although P AIR N ORM -SI is not theoretically proved to have a constant
APSD and APD, empirically it achieves more stable APSD and APD than P AIR N ORM .
We were not able to prove this phenomenon mathematically, and leave it for further investigation
[Figure]
APD does not capture the full information of the distribution of pairwise distances.
To show how the distribution changes by increasing number of layers, we use Tensorboard to plot the histograms of pairwise distances, as shown in Figure 11.
Comparing SGC and SGC with P AIR N ORM , adding
P AIR N ORM keeps the left shift (shrinkage) of the distribution of random pair distances much slower than without normalization, while still sharing similar behavior of the distribution of connected pairwise distances.
P AIR N ORM -SI seems to be more powerful in keeping the median and mean of the distribution of random pair distances stable, while “spreading” the distribution out by increasing the variance.
The performance of P AIR N ORM and P AIR N ORM -SI are similar, however it seems that
P AIR N ORM -SI is more powerful in stabilizing TPD and TPSD.
[Figure]
A.6. GCN WITH P AIR N ORM AND P AIR N ORM -SI
[Figure]
dropout=0.6 in 1000 epochs.
Respective test set accuracies are 31.09%, 77.77%, 75.09%.
Note that the scale of distances is not comparable across models, since they have learnable parameters that scale these distances differently.
The formal analysis for P AIR N ORM and P AIR N ORM -SI is based on SGC.
GCN (and other GNNs) has learnable parameters, dropout layers, and activation layers, all of which complicate direct mathematical analyses.
Here we perform similar empirical measurements for pairwise distances to get a rough sense of how P AIR N ORM and P AIR N ORM -SI work with GCN based on the Cora dataset.
Figures 12 and 13 demonstrate how P AIR N ORM and P AIR N ORM -SI can help train a relatively deep (12 layers) GCN.
Notice that oversmoothing occurs very quickly for GCN without any normalization, where both connected and random pair distances reach zero (!).
In contrast, GCN with P AIR N ORM or P AIR N ORMSI is able to keep random pair distances relatively apart while allowing connected pair distances to shrink.
As also stated in main text, using P AIR N ORM -SI for GCN and GAT is relatively more stable than using P AIR N ORM in general cases (notice the near-constant random pair distances in the rightmost subﬁgures).
There are several possible explanations for why P AIR N ORM -SI is more stable.
First, as shown in Figure 10 and Figure 12, P AIR N ORM -SI not only keeps APSD stable but also APD, further, the plots of distributions of pairwise distances (Figures 11 and 13) also show the power of P AIR N ORM -SI (notice the large gap between smaller connected pairwise distances and the larger random pairwise distances).
Second, we conjecture that restricting representations to reside on a sphere can make training stable and faster, which we also observe empirically by studying the training curves.
Third, GCN and GAT tend to overﬁt easily for the SSNC problem, due to many learnable parameters across layers and limited labeled input data, therefore it is possible that adding more restriction on these models helps reduce overﬁtting.
[Figure]
All in all
, these empirical measurements as illustrated throughout the ﬁgures in this section demonstrates that P AIR N ORM and P AIR N ORM -SI successfully address the oversmoothing problem for deep GNNs.
Our work is the ﬁrst to propose a normalization layer speciﬁcally designed for graph neural networks, which we hope will kick-start more work in this area toward training more robust and effective GNNs.