D ROP E DGE : T OWARDS D EEP G RAPH C ONVOLUTIONAL N ETWORKS ON N ODE C LASSIFICATION
A BSTRACT
Over-ﬁtting and over-smoothing are two main obstacles of developing deep Graph
Convolutional Networks (GCNs) for node classiﬁcation.
In particular, over-ﬁtting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth.
This paper proposes DropEdge, a novel and ﬂexible technique to alleviate both issues.
At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer.
Furthermore, we theoretically demonstrate that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it.
More importantly, our DropEdge is a general skill that can be equipped with many other backbone models ( e.g.
GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance.
Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs.
The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well.
Codes are released on https://github.com/DropEdge/DropEdge .
I NTRODUCTION
Graph Convolutional Networks (GCNs), which exploit message passing or equivalently certain neighborhood aggregation function to extract high-level features from a node as well as its neighborhoods, have boosted the state-of-the-arts for a variety of tasks on graphs, such as node classiﬁcation (Bhagat et al., 2011;
Zhang et al., 2018), social recommendation (Freeman, 2000; Perozzi et al., 2014), and link prediction (Liben-Nowell & Kleinberg, 2007) to name some.
In other words, GCNs have been becoming one of the most crucial tools for graph representation learning.
Yet, when we revisit typical
GCNs on node classiﬁcation (Kipf & Welling, 2017), they are usually shallow ( e.g. the number of the layers is 2 1 ).
Inspired from the success of deep CNNs on image classiﬁcation, several attempts have been proposed to explore how to build deep GCNs towards node classiﬁcation (Kipf & Welling, 2017;
Li et al., 2018a; Xu et al., 2018a; Li et al., 2019); nevertheless, none of them delivers sufﬁciently expressive architecture.
The motivation of this paper is to analyze the very factors that impede deeper
GCNs to perform promisingly, and develop method to address them.
We begin by investigating two factors: over-ﬁtting and over-smoothing .
Over-ﬁtting comes from the case when we utilize an over-parametric model to ﬁt a distribution with limited training data, where the model we learn ﬁts the training data very well but generalizes poorly to the testing data.
It does exist if we apply a deep GCN on small graphs (see 4-layer GCN on Cora in Figure 1).
Over-smoothing, towards the other extreme, makes training a very deep GCN difﬁcult.
As ﬁrst introduced by Li et al. (2018a) and further explained in Wu et al.
; Xu et al. (2018a); Klicpera et al.
(2019), graph convolutions essentially push representations of adjacent nodes mixed with each [Figure] other, such that, if extremely we go with an inﬁnite number of layers, all nodes’ representations will converge to a stationary point, making them unrelated to the input features and leading to vanishing gradients.
We call this phenomenon as over-smoothing of node features.
To illustrate its inﬂuence, we have conducted an example experiment with 8-layer GCN in Figure [Figure] in which the training of such a deep GCN is observed to converge poorly.
Both of the above two issues can be alleviated, using the proposed method, DropEdge.
The term “DropEdge” refers to randomly dropping out certain rate of edges of the input graph for each training time.
There are several beneﬁts in applying DropEdge for the GCN training (see the experimental improvements by DropEdge in Figure 1).
First, DropEdge can be considered as a data augmentation technique.
By DropEdge, we are actually generating different random deformed copies of the original graph; as such, we augment the randomness and the diversity of the input data, thus better capable of preventing over-ﬁtting.
Second, DropEdge can also be treated as a message passing reducer.
In
GCNs, the message passing between adjacent nodes is conducted along edge paths.
Removing certain edges is making node connections more sparse, and hence avoiding over-smoothing to some extent when GCN goes very deep.
Indeed, as we will draw theoretically in this paper, DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it.
We are also aware that the dense connections employed by JKNet (Xu et al., 2018a) are another kind of tools that can potentially prevent over-smoothing.
In its formulation, JKNet densely connects each hidden layer to the top one, hence the feature mappings in lower layers that are hardly affected by over-smoothing are still maintained.
Interestingly and promisingly, we ﬁnd that the performance of
JKNet can be promoted further if it is utilized along with our DropEdge.
Actually, our DropEdge—as a ﬂexible and general technique—is able to enhance the performance of various popular backbone networks on several benchmarks, including GCN (Kipf & Welling, 2017), ResGCN (Li et al., 2019),
JKNet (Xu et al., 2018a), and GraphSAGE (Hamilton et al., 2017).
We provide detailed evaluations in the experiments.
R ELATED W ORK
GCNs Inspired by the huge success of CNNs in computer vision, a large number of methods come redeﬁning the notion of convolution on graphs under the umbrella of GCNs.
The ﬁrst prominent research on GCNs is presented in Bruna et al.
(2013), which develops graph convolution based on spectral graph theory.
Later, Kipf & Welling (2017); Defferrard et al. (2016); Henaff et al.
; Li et al.
(2018b); Levie et al.
(2017) apply improvements, extensions, and approximations on spectralbased GCNs.
To address the scalability issue of spectral-based GCNs on large graphs, spatial-based
GCNs have been rapidly developed (Hamilton et al., 2017; Monti et al., 2017; Niepert et al., 2016;
Gao et al., 2018).
These methods directly perform convolution in the graph domain by aggregating the information from neighbor nodes.
Recently, several sampling-based methods have been proposed for fast graph representation learning, including the node-wise sampling methods (Hamilton et al., 2017), the layer-wise approach (Chen et al., 2018) and its layer-dependent variant (Huang et al., 2018).
Speciﬁcally, GAT (Velickovic et al., 2018) has discussed applying dropout on edge attentions.
While it actually is a post-conducted version of DropEdge before attention computation, the relation to over-smoothing is never explored in Velickovic et al.
In our paper, however, we have formally presented the formulation of DropEdge and provided rigorous theoretical justiﬁcation of its beneﬁt in alleviating over-smoothing.
We also carried out extensive experiments by imposing
DropEdge on several popular backbones.
One additional point is that we further conduct adjacency normalization after dropping edges, which, even simple, is able to make it much easier to converge during training and reduce gradient vanish as the number of layers grows.
Deep GCNs Despite the fruitful progress, most previous works only focus on shallow GCNs while the deeper extension is seldom discussed.
The attempt for building deep GCNs is dated back to the GCN paper (Kipf & Welling, 2017), where the residual mechanism is applied; unexpectedly, as shown in their experiments, residual GCNs still perform worse when the depth is 3 and beyond.
The authors in Li et al.
(2018a)
ﬁrst point out the main difﬁculty in constructing deep networks lying in over-smoothing, but unfortunately, they never propose any method to address it.
The follow-up study (Klicpera et al., 2019) solves over-smoothing by using personalized PageRank that additionally involves the rooted node into the message passing loop; however, the accuracy is still observed to decrease when the depth increases from 2.
JKNet (Xu et al., 2018a) employs dense connections for multi-hop message passing which is compatible with DropEdge for formulating deep GCNs.
Oono & Suzuki (2019) theoretically prove that the node features of deep GCNs will converge to a subspace and incur information loss.
It generalizes the conclusion in Li et al.
(2018a)
by further considering the ReLu function and convolution ﬁlters.
Our interpretations on why DropEdge can impede over-smoothing is based on the concepts proposed by Oono & Suzuki (2019).
A recent method (Li et al., 2019) has incorporated residual layers, dense connections and dilated convolutions into GCNs to facilitate the development of deep architectures.
Nevertheless, this model is targeted on graph-level classiﬁcation ( i.e. point cloud segmentation), where the data points are graphs and naturally disconnected from each other.
In our task for node classiﬁcation, the samples are nodes and they all couple with each other, thus the over-smoothing issue is more demanded to be addressed.
By leveraging DropEdge, we are able to relieve over-smoothing, and derive more enhanced deep GCNs on node classiﬁcation.
N OTATIONS AND P RELIMINARIES
Notations.
Let G =
( V , E ) represent the input graph of size N with nodes v i ∈ V and edges ( v i , v j ) ∈ E .
The node features are denoted as X = { x 1 , · · · , x N } ∈ R N × C , and the adjacency matrix is deﬁned as A ∈ R N × N which associates each edge ( v i , v j ) with its element A ij .
The node degrees are given by d = { d 1 , · · · , d N } where d i computes the sum of edge weights connected to node i .
We deﬁne D as the degree matrix whose diagonal elements are obtained from d .
GCN is originally developed by Kipf & Welling (2017).
The feed forward propagation in GCN is recursively conducted as
[Equation]
(1) where H ( l +1) = { h ( l +1), · · · , h ( l +1) N } are the hidden vectors of the l -th layer with h ( l ) i as the hidden feature for node i ; ˆ A =
ˆ D − 1 /
( A + I )
ˆ D − 1 / 2 is the re-normalization of the adjacency matrix, and ˆ D is the corresponding degree matrix of A
+ I ; σ ( · ) is a nonlinear function, i.e. the ReLu function; and
W ( l ) ∈ R C
l × C l − 1 is the ﬁlter matrix in the l -th layer with C l refers to the size of l -th hidden layer.
We denote one-layer GCN computed by Equation 1 as Graph Convolutional Layer ( GCL ) in what follows.
[Equation]
This section ﬁrst introduces the methodology of the DropEdge technique as well as its layer-wise variant where the adjacency matrix for each GCN layer is perturbed individually.
We also explain how the proposed DropEdge can prevent over-ﬁtting and over-smoothing in generic GCNs.
Particularly for over-smoothing, we provide its mathematical deﬁnition and theoretical derivations on showing the beneﬁts of DropEdge.
4. M ETHODOLOGY
At each training epoch, the DropEdge technique drops out a certain rate of edges of the input graph by random.
Formally, it randomly enforces V p non-zero elements of the adjacency matrix A to be zeros, where V is the total number of edges and p is the dropping rate.
If we denote the resulting adjacency matrix as A drop , then its relation with A becomes
[Equation]
where A (cid:48) is a sparse matrix expanded by a random subset of size V p from original edges E .
Following the idea of Kipf & Welling (2017), we also perform the re-normalization trick on A drop , leading to ˆ A drop .
We replace ˆ A with ˆ A drop in Equation 1 for propagation and training.
When validation and testing, DropEdge is not utilized.
Preventing over-ﬁtting.
DropEdge produces varying perturbations of the graph connections.
As a result, it generates different random deformations of the input data and can be regarded as a data augmentation skill for graphs.
To explain why this is valid, we provide an intuitive understanding here.
The key in GCNs is to aggregate neighbors’ information for each node, which can be understood as a weighted sum of the neighbor features (the weights are associated with the edges).
From the perspective of neighbor aggregation, DropEdge enables a random subset aggregation instead of the full aggregation during GNN training.
Statistically, DropEdge only changes the expectation of the neighbor aggregation up to a multiplier p , if we drop edges with probability p .
This multiplier will be actually removed after weights normalization, which is often the case in practice.
Therefore, DropEdge does not change the expectation of neighbor aggregation and is an unbiased data augmentation technique for GNN training, similar to typical image augmentation skills ( e.g. rotation, cropping and ﬂapping) that are capable of hindering over-ﬁtting in training CNNs.
We will provide experimental validations in § 5.1.
Layer-Wise DropEdge.
The above formulation of DropEdge is one-shot with all layers sharing the same perturbed adjacency matrix.
Indeed, we can perform DropEdge for each individual layer.
Speciﬁcally, we obtain ˆ A ( l ) drop by independently computing Equation 2 for each l -th layer.
Different layer could have different adjacency matrix ˆ A ( l ) drop .
Such layer-wise version brings in more randomness and deformations of the original data, and we will experimentally compare its performance with the original DropEdge in § 5.2.
Over-smoothing is another obstacle of training deep GCNs, and we will detail how DropEdge can address it to some extent in the next section.
For simplicity, the following derivations assume all
GCLs share the same perturbed adjacency matrix, and we will leave the discussion on layer-wise
DropEdge for future exploration.
4. T OWARDS PREVENTING OVER - SMOOTHING
By its original deﬁnition in Li et al.
(2018a)
, the over-smoothing phenomenon implies that the node features will converge to a ﬁxed point as the network depth increases.
This unwanted convergence restricts the output of deep GCNs to be only relevant to the graph topology but independent to the input node features, which as a matter of course incurs detriment of the expressive power of GCNs.
Oono & Suzuki (2019) has generalized the idea in Li et al.
(2018a)
by taking both the non-linearity ( i.e. the ReLu function) and the convolution ﬁlters into account; they explain over-smoothing as convergence to a subspace rather than convergence to a ﬁxed point.
This paper will use the concept of subspace by Oono & Suzuki (2019) for more generality.
We ﬁrst provide several relevant deﬁnitions that facilitate our later presentations.
Deﬁnition 1 (subspace) .
Let M :=
{ EC | C ∈ R M × C } be an M -dimensional subspace in R N × C , where E ∈ R N × M is orthogonal, i.e. E T E
= I M , and M ≤ N .
Deﬁnition 2 ( (cid:15) -smoothing) .
We call the (cid:15) -smoothing of node features happens for a GCN, if all its hidden vectors H ( l ) beyond a certain layer L have a distance no larger than (cid:15) ( (cid:15) > 0 ) with respect to a subspace M that is independent to the input features, namely,
[Equation]
where d M ( · ) computes the distance between the input matrix and the subspace M .
Deﬁnition 3 (the (cid:15) -smoothing layer) .
Given the subspace M and (cid:15) , we call the minimal value of the layers that satisfy Equation 3 as the (cid:15) -smoothing layer, that is, l ∗ ( M , (cid:15) ) :
= min l { d M ( H ( l ) ) < (cid:15) } .
Since conducting analysis exactly based on the (cid:15) -smoothing layer is difﬁcult, we instead deﬁne the relaxed (cid:15) -smoothing layer which is proved to be an upper bound of l ∗ .
Deﬁnition 4 (the relaxed (cid:15) -smoothing layer) .
Given the subspace M and (cid:15) , we call ˆ l ( M , (cid:15) )
= (cid:100) log( (cid:15)/d M ( X ))
log sλ (cid:101) as the relaxed smoothing layer, where, (cid:100)·(cid:101) computes the ceil of the input, s is the supremum of the ﬁlters’ singular values over all layers, and λ is the second largest eigenvalue of ˆ A .
Besides, we have ˆ l ≥ l ∗ 4 . According to the conclusions by the authors in Oono & Suzuki (2019), a sufﬁciently deep GCN will certainly suffer from the (cid:15) -smoothing issue for any small value of (cid:15) under some mild conditions (the details are included in the supplementary material).
Note that they only prove the existence of (cid:15) -smoothing in deep GCN without developing any method to address it.
Here, we will demonstrate that adopting DropEdge alleviates the (cid:15) -smoothing issue in two aspects: 1.
By reducing node connections, DropEdge is proved to slow down the convergence of over-smoothing; in other words, the value of the relaxed (cid:15) -smoothing layer will only increase if using DropEdge. 2.
The gap between the dimensions of the original space and the converging subspace, i.e. N − M measures the amount of information loss; larger gap means more severe information loss.
As shown by our derivations, DropEdge is able to increase the dimension of the converging subspace, thus capable of reducing information loss.
We summarize our conclusions as follows.
Theorem 1.
We denote the original graph as G and the one after dropping certain edges out as G (cid:48) .
Given a small value of (cid:15) , we assume G and G (cid:48) will encounter the (cid:15) -smoothing issue with regard to subspaces M and M (cid:48) , respectively.
Then, either of the following inequalities holds after sufﬁcient edges removed.
• The relaxed smoothing layer only increases: ˆ l ( M , (cid:15) ) ≤ ˆ l ( M (cid:48) , (cid:15) ) ; • The information loss is decreased: N − dim ( M ) > N − dim ( M (cid:48) ) .
The proof of Theorem 1 is based on the derivations in Oono & Suzuki (2019) as well as the concept of mixing time that has been studied in the random walk theory (Lovász et al., 1993).
We provide the full details in the supplementary material.
Theorem 1 tells that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it.
In this way, DropEdge enables us to train deep GCNs more effectively.
4. DISCUSSIONS
This sections contrasts the difference between DropEdge and other related concepts including Dropout,
DropNode, and Graph Sparsiﬁcation.
DropEdge vs. Dropout
The Dropout trick (Hinton et al., 2012) is trying to perturb the feature matrix by randomly setting feature dimensions to be zeros, which may reduce the effect of over-ﬁtting but is of no help to preventing over-smoothing since it does not make any change of the adjacency
All detailed deﬁnitions and proofs are provided in the appendix.
matrix. As a reference, DropEdge can be regarded as a generation of Dropout from dropping feature dimensions to dropping edges, which mitigates both over-ﬁtting and over-smoothing.
In fact, the impacts of Dropout and DropEdge are complementary to each other, and their compatibility will be shown in the experiments.
DropEdge vs. DropNode
Another related vein belongs to the kind of node sampling based methods, including GraphSAGE (Hamilton et al., 2017), FastGCN
(Chen et al., 2018), and ASGCN (Huang et al., 2018).
We name this category of approaches as DropNode.
For its original motivation, DropNode samples sub-graphs for mini-batch training, and it can also be treated as a speciﬁc form of dropping edges since the edges connected to the dropping nodes are also removed.
However, the effect of DropNode on dropping edges is node-oriented and indirect.
By contrast,
DropEdge is edge-oriented, and it is possible to preserve all node features for the training (if they can be ﬁtted into the memory at once), exhibiting more ﬂexibility.
Further, to maintain desired performance, the sampling strategies in current DropNode methods are usually inefﬁcient, for example, GraphSAGE suffering from the exponentially-growing layer size, and AS-GCN requiring the sampling to be conducted recursively layer by layer.
Our DropEdge, however, neither increases the layer size as the depth grows nor demands the recursive progress because the sampling of all edges are parallel.
DropEdge vs. Graph-Sparsiﬁcation Graph-Sparsiﬁcation (Eppstein et al., 1997) is an old research topic in the graph domain.
Its optimization goal is removing unnecessary edges for graph compressing while keeping almost all information of the input graph.
This is clearly district to the purpose of DropEdge where no optimization objective is needed.
Speciﬁcally, DropEdge will remove the edges of the input graph by random at each training time, whereas Graph-Sparsiﬁcation resorts to a tedious optimization method to determine which edges to be deleted, and once those edges are discarded the output graph keeps unchanged.
E XPERIMENTS
Datasets Joining the previous works’ practice, we focus on four benchmark datasets varying in graph size and feature type: (1) classifying the research topic of papers in three citation datasets:
Cora, Citeseer and Pubmed (Sen et al., 2008); (2) predicting which community different posts belong to in the Reddit social network (Hamilton et al., 2017).
Note that the tasks in Cora, Citeseer and
Pubmed are transductive underlying all node features are accessible during training, while the task in
Reddit is inductive meaning the testing nodes are unseen for training.
We apply the full-supervised training fashion used in Huang et al.
and Chen et al.
(2018) on all datasets in our experiments.
The statics of all datasets are listed in the supplemental materials.
5. C AN D ROP E DGE GENERALLY IMPROVE THE PERFORMANCE OF DEEP GCN S ?
In this section, we are interested in if applying DropEdge can promote the performance of current popular GCNs (especially their deep architectures) on node classiﬁcation.
Implementations We consider ﬁve backbones: GCN (Kipf & Welling, 2017), ResGCN (He et al., 2016; Li et al., 2019), JKNet (Xu et al., 2018a), IncepGCN 5 and GraphSAGE (Hamilton et al., 2017) with varying depth from 2 to 64. 6 Since different structure exhibits different training dynamics on different dataset, to enable more robust comparisons, we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark.
The searching space of hyper-parameters and more details are provided in Table 4 in the supplementary material.
Regarding the same architecture w or w/o DropEdge, we apply the same set of hyperparameters except the drop rate p for fair evaluation.
Overall Results Table 1 summaries the results on all datasets.
We only report the performance of the model with 2/8/32 layers here due to the space limit, and provide the accuracy under other different depths in the supplementary material.
It’s observed that DropEdge consistently improves the the memory bottleneck. [Table] testing accuracy for all cases.
The improvement is more clearly depicted in Figure 2a, where we have computed the average absolute improvement over all backbones by DropEdge on each dataset under different numbers of layers.
On Citeseer, for example, DropEdge yields further improvement for deeper architecture; it gains 0.9% average improvement for the model with 2 layers while achieving a remarkable 13.5% increase for the model with 64 layers.
In addition, the validation losses of all 4-layer models on Cora are shown in Figure 2b.
The curves along the training epoch are dramatically pulled down after applying DropEdge, which also explains the effect of DropEdge on alleviating over-ﬁtting.
Another valuable observation in Table 1 is that the 32-layer IncepGCN without DropEdge incurs the Out-Of-Memory (OOM) issue while the model with DropEdge survives, showing the advantage of DropEdge to save memory consuming by making the adjacency matrix sparse.
Comparison with SOTAs We select the best performance for each backbone with DropEdge, and contrast them with existing State of the Arts (SOTA), including GCN, FastGCN, AS-GCN and
GraphSAGE in Table 2; for the SOTA methods, we reuse the results reported in Huang et al.
We have these ﬁndings: (1) Clearly, our DropEdge obtains signiﬁcant enhancement against SOTAs; particularly on Reddit, the best accuracy by our method is 97.02%, and it is better than the previous best by AS-GCN (96.27%), which is regarded as a remarkable boost considering the challenge on this benchmark.
(2) For most models with DropEdge, the best accuracy is obtained under the depth beyond 2, which again veriﬁes the impact of DropEdge on formulating deep networks.
(3) As mentioned in § 4.3, FastGCN, AS-GCN and GraphSAGE are considered as the DropNode extensions of GCNs.
The DropEdge based approaches outperform the DropNode based variants as shown in
Table 2, which somehow conﬁrms the effectiveness of DropEdge.
Actually, employing DropEdge upon the DropNode methods further delivers promising enhancement, which can be checked by revisiting the increase by DropEdge for GraphSAGE in Table 1.
5. H OW DOES D ROP E DGE HELP ?
This section continues a more in-depth analysis on DropEdge and attempts to ﬁgure out why it works.
Due to the space limit, we only provide the results on Cora, and defer the evaluations on other datasets to the supplementary material.
Note that this section mainly focuses on analyzing DropEdge and its variants, without the concern with pushing state-of-the-art results.
So, we do not perform delicate hyper-parameter selection.
We employ GCN as the backbone in this section.
Here, GCN- n denotes GCN of depth n .
The hidden dimension, learning rate and weight decay are ﬁxed to 256, 0.005 and 0.0005, receptively. The [Figure] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
ResGCN+DropEdge 87.00(4) 79.
[Table]
JKNet+DropEdge 88.00(16) 80.20(8) 91.60(64) 97.02(8)
IncepGCN+DropEdge 88.20(8) 80.50(8) 91.60(4) 96.87(8)
GraphSAGE+DropEdge 88.10(4) 80.00(2) 91.70(8) 96.54(4) random seed is ﬁxed.
We train all models with 200 epochs.
Unless otherwise mentioned, we do not utilize the “withloop” and “withbn” operation (see their deﬁnitions in Table 4 in the appendix).
5.2. O N PREVENTING OVER - SMOOTHING
As discussed in § 4.2, the over-smoothing issue exists when the top-layer outputs of GCN converge to a subspace and become unrelated to the input features with the increase in depth.
Since we are unable to derive the converging subspace explicitly, we measure the degree of over-smoothing by instead computing the difference between the output of the current layer and that of the previous one.
We adopt the Euclidean distance for the difference computation.
Lower distance means more serious over-smoothing.
Experiments are conducted on GCN-8.
Figure 3 (a) shows the distances of different intermediate layers (from 2 to 6) under different edge dropping rates (0 and 0.8).
Clearly, over-smoothing becomes more serious in GCN as the layer grows, which is consistent with our conjecture.
Conversely, the model with DropEdge ( p = 0 . 8 ) reveals higher distance and slower convergent speed than that without DropEdge ( p = 0 ), implying the importance of DropEdge to alleviating over-smoothing.
We are also interested in how the oversmoothing will act after training.
For this purpose, we display the results after 150-epoch training in
Figure 3 (b).
For GCN without DropEdge, the difference between outputs of the 5-th and 6-th layers is equal to 0, indicating that the hidden features have converged to a certain stationary point. On the contrary, GCN with DropEdge performs promisingly, as the distance does not vanish to zero when
[Figure] [Figure] the number of layers grows; it probably has successfully learned meaningful node representations after training, which could also be validated by the training loss in Figure 3 (c).
5.2. O N C OMPATIBILITY WITH D ROPOUT
§ 4.3 has discussed the difference between DropEdge and Dropout.
Hence, we conduct an ablation study on GCN-4, and the validation losses are demonstrated in Figure 4a.
It reads that while both
Dropout and DropEdge are able to facilitate the training of GCN, the improvement by DropEdge is more signiﬁcant, and if we adopt them concurrently, the loss is decreased further, indicating the compatibility of DropEdge with Dropout.
5.2. O N LAYER - WISE D ROP E DGE
§ 4.1 has descried the Layer-Wise (LW) extension of DropEdge.
Here, we provide the experimental evaluation on assessing its effect.
As observed from Figure 4b, the LW DropEdge achieves lower training loss than the original version, whereas the validation value between two models is comparable.
It implies that LW DropEdge can facilitate the training further than original DropEdge.
However, we prefer to use DropEdge other than the LW variant so as to not only avoid the risk of over-ﬁtting but also reduces computational complexity since LW DropEdge demands to sample each layer and spends more time.
C ONCLUSION
We have presented DropEdge, a novel and efﬁcient technique to facilitate the development of deep
Graph Convolutional Networks (GCNs).
By dropping out a certain rate of edges by random, DropEdge includes more diversity into the input data to prevent over-ﬁtting, and reduces message passing in graph convolution to alleviate over-smoothing.
Considerable experiments on Cora, Citeseer, Pubmed and Reddit have veriﬁed that DropEdge can generally and consistently promote the performance of current popular GCNs, such as GCN, ResGCN, JKNet, IncepGCN, and GraphSAGE.
It is expected that our research will open up a new venue on a more in-depth exploration of deep GCNs for broader potential applications.
A CKNOWLEDGEMENTS
This research was funded by National Science and Technology Major Project of the Ministry of
Science and Technology of China (No. 2018AAA0102900).
Finally, Yu Rong wants to thank, in particular, the invaluable love and support from Yunman Huang over the years.
Will you marry me?
A A PPENDIX : P ROOF OF T HEOREM
To prove Theorem 1, we need to borrow the following deﬁnitions and corollaries from Oono & Suzuki (2019).
First, we denote the maximum singular value of W l by s l and set s :=
sup l ∈ N + s l .
We assume that W l of all layers are initialized so that s ≤ 1 .
Second, we denote the distance that induced as the Frobenius norm from X to M by d M ( X ) :=
inf Y ∈M || X − Y
|| F .
Then, we recall Corollary 3 and Proposition 1 in Oono & Suzuki (2019) as Corollary 1 below.
Corollary 1.
Let λ 1 ≤ · · · ≤ λ N be the eigenvalues of ˆ A , sorted in ascending order.
Suppose the multiplicity of the largest eigenvalue
λ
N is M ( ≤ N ) , i.e., λ N − M < λ N {+− M +
+}
= · · · = λ N and the second largest eigenvalue is deﬁned as
[Equation]
Let E to be the eigenspace associated with λ N − M +1 , · · · , λ N .
Then we have λ <
λ N = 1 , and
[Equation]
where M := { EC | C ∈ R M × C } .
Besides, s l λ < 1 , implying that the output of the l -th layer of
GCN on G exponentially approaches M[-} be M connected graphs in G . Since λ N = 1 for ˆ A in GCN, ˆ A contains M multiplicity of the largest eigenvalue λ N that λ N − M +1 = · · · = λ N = 
 and λ < 1 . Therefore, we have s l λ < 1-] .
We also need to adopt some concepts from Lovász et al.
in proving Theorem 1.
Consider the graph G as an electrical network, where each edge represents an unit resistance.
Then the effective resistance, R st from node s to node t is deﬁned as the total resistance between node s and t .
According to Corollary 3.3 and Theorem 4.1 (i) in Lovász et al.
(1993), we can build the connection between λ and R st for each connected component via commute time as the following inequality.
[Equation]
Prior to proving Theorem 1, we ﬁrst derive the lemma below.
Lemma 2.
The (cid:15) -smoothing happens whenever the layer number satisﬁes
[Equation]
where (cid:100)·(cid:101) computes the ceil of the input.
It means ˆ l ≥ l ∗ .
Proof.
We start our proof from Inequality 5, leading to
[Equation]
When it reaches (cid:15) -smoothing, the following inequality should be satisﬁed as
[Equation]
Since 0 ≤ sλ < 1 , then log sλ < 0 .
Therefore, the Inequality 8 becomes l >
[Equation]
[Equation]
Clearly, we have ˆ l ≥ l ∗ since l ∗ is deﬁned as the minimal layer that satisﬁes (cid:15) -smoothing. The proof is concluded. Now, we prove Theorem 1.
Proof.
Our proof relies basically on the connection between λ and R st in Equation (6). We recall
Corollary 4.3 in Lovász et al. (1993) that removing any edge from G can only increase any R st , then according to (6), the lower bound of λ only increases if the removing edge is not connected to either s or t ( i.e. the degree d s and d t keep unchanged).
Since there must exist a node pair satisfying
R st = ∞ after sufﬁcient edges (except self-loops) are removed from one connected component of G , we have the inﬁnite case λ = 1 given in Equation (6) that both 1 /d s and 1 /d t are consistently bounded by a ﬁnite number, i.e. 1 .
It implies λ does increase before it reaches λ
= 1 .
As ˆ l is positively related to λ (see the right side of Equation (7) where log ( sλ ) <0), we have proved the ﬁrst part of
Theorem 1, i.e. , ˆ l ( M , (cid:15) ) ≤ ˆ l ( M (cid:48) , (cid:15) ) after removing sufﬁcient edges. When there happens R st = ∞ , the connected component is disconnected into two parts, which leads to the increment of the dimension of M by 1 and proves the second part of Theorem 1.
i.e., the information loss is decreased: N − dim ( M ) > N − dim ( M (cid:48) ) .
B A PPENDIX : M ORE D ETAILS IN E XPERIMENTS B. D ATASETS S TATISTICS
Datasets The statistics of all datasets are summarized in Table 3.
[Table]
B. M ODELS AND B ACKBONES
Backbones Other than the multi-layer GCN, we replace the CNN layer with graph convolution layer to implement three popular backbones recasted from image classiﬁcation.
They are residual network (ResGCN)(He et al., 2016; Li et al., 2019), inception network (IncepGCN)(Szegedy et al., 2016) and dense network (JKNet) (Huang et al., 2017; Xu et al., 2018b).
Figure 5 shows the detailed architectures of four backbones.
Furthermore, we employ one input GCL and one output GCL on these four backbones.
Therefore, the layers in ResGCN, JKNet and InceptGCN are at least 3 layers.
All backbones are implemented in Pytorch (Paszke et al., 2017).
For GraphSAGE, we utilize the
Pytorch version implemented by DGL(Wang et al., 2019).
Aggregation
Input
GCL GCL
GCL GCL
GCL GCL
Output (d) IncepGCN
Aggregation
Input
GCL GCL
GCL
Output (c) JKNet
Input
GCL GCL
GCL
Output (a) GCN Input
GCL GCL GCL Output (
[Figure]
Self Feature Modeling We also implement a variant of graph convolution layer with self feature modeling (Fout et al., 2017):
[Equation]
where W ( l ) self ∈ R C l × C l − 1 .
Hyper-parameter Optimization We adopt the Adam optimizer for model training.
To ensure the re-productivity of the results, the seeds of the random numbers of all experiments are set to the same.
We ﬁx the number of training epoch to 400 for all datasets.
All experiments are conducted on a
NVIDIA Tesla P40 GPU with 24GB memory.
Given a model with n ∈ { 2 , 4 , 8 , 16 , 32 , 64 } layers, the hidden dimension is 128 and we conduct a random search strategy to optimize the other hyper-parameter for each backbone in § 5.1.
The decryptions of hyper-parameters are summarized in Table 4.
Table 5 depicts the types of the normalized adjacency matrix that are selectable in the “normalization” hyper-parameter.
For GraphSAGE, the aggregation type like GCN, MAX, MEAN, or LSTM is a hyper-parameter as well.
For each model, we try 200 different hyper-parameter combinations via random search and select the best test accuracy as the result.
Table 6 summaries the hyper-parameters of each backbone with the best accuracy on different datasets and their best accuracy are reported in Table 2.
B.
Figure 6 depicts the additional results of validation loss on different backbones w and w/o DropEdge. [Figure]
B. T HE A BLATION S TUDY ON C ITESEER
Figure 7a shows the ablation study of Dropout vs. DropEdge and Figure 4b depicts a comparison between the proposed DropEdge and the layer-wise DropEdge on Citeseer.
[Figure]
[Figure]
[Table] [Table]
[Table] [Table]
T a b l e 7 :
A cc u r ac y ( % ) c
o m p
a
r
i s on s on d i ff e r e n t b ac kbon e s w it h a nd w it
hou t
D r op
E dg e
D a t a s e t
B ac kbon e
O r i gn a l
D r op
E dg e
O r i gn a l
D r op
E dg e
O r i gn a l
D r op
E dg e
O r i gn a l
D r op
E dg e
O r i gn a l
[Table]
r op
E dg e
O r i gn a
[Table]
o r a
[-G C N 86 .86 .
-]
[Table]
C N
[Table]
[Table]
[Table]
KN e t - -[-86 .87 .86 .87 .86 .88 . 
 [Table] [Table] [Table]-]
[Table]
I n c e p G C
N - -
[Table]
[Table]
AG[-E 87 .
-]
[Table]
C it e s ee r
G C N 75 .78 .76 .79 [-.
-].74 .77 .65 .76 .
[Table] [Table]
[Table] [Table]
R e s G
C N - - 78 .78 .
[Table] [Table]
[Table] [Table]
J KN e t - - 79 .
[Table] [Table]
[Table]
I n c e p G C
N - - 79 {+.
+}
[Table] [Table]
[Table]
[Table]
G r a ph S
AG E 78 .
[Table] [Table]
[Table] [Table]
P ub m e d
G C N 90 .
[Table]
[Table]
[Table] [Table]
R e s G
C N - - 90 .90 .89 .
[Table] [Table]
J KN e t - - 90 .91 .
[Table]
[Table]
I n c e p G C
N - - 89 .91 .
OO
M 90 .
OO
M
[Table]
G r a ph S
[Table]