GR ES N ET : G RAPH R ESIDUAL N ETWORK FOR R EVIVING D EEP GNN S FROM S USPENDED A NIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The existing graph neural networks (GNNs) based on the spectral graph convolutional operator have been criticized for its performance degradation, which is especially common for the models with deep architectures.
In this paper, we further identify the suspended animation problem with the existing GNNs.
Such a problem happens when the model depth reaches the suspended animation limit, and the model will not respond to the training data any more and become not learnable.
Analysis about the causes of the suspended animation problem with existing
GNNs will be provided in this paper, whereas several other peripheral factors that will impact the problem will be reported as well.
To resolve the problem, we introduce the GR ES N ET (Graph Residual Network) framework in this paper, which creates extensively connected highways to involve nodes’ raw features or intermediate representations throughout the graph for all the model layers.
Different from the other learning settings, the extensive connections in the graph data will render the existing simple residual learning methods fail to work.
We prove the effectiveness of the introduced new graph residual terms from the norm preservation perspective, which will help avoid dramatic changes to the node’s representations between sequential layers.
Detailed studies about the GR ES N ET framework for many existing GNNs, including GCN, GAT and L OOPY N ET , will be reported in the paper with extensive empirical experiments on real-world benchmark datasets.
I NTRODUCTION
Graph neural networks (GNN), e.g., graph convolutional network (GCN) Kipf & Welling (2016) and graph attention network (GAT) Veliˇckovi´c et al.
(2018), based on the approximated spectral graph convolutional operator Hammond et al.
(2011), can learn the representations of the graph data effectively.
Meanwhile, such GNNs have also received lots of criticism, since as these GNNs’ architectures go deep, the models’ performance will get degraded, which is similar to observations on other deep models (e.g., convolutional neural network) as reported in He et al. (2015).
Meanwhile, different from the existing deep models, when the GNN model depth reaches a certain limit (e.g., depth ≥ 5 for GCN with the bias term disabled or depth ≥ 8 for GCN with the bias term enabled on the Cora dataset), the model will not respond to the training data any more and become not learnable.
Formally, we name such an observation as the GNNs’ suspended animation problem , whereas the corresponding model depth is named as the suspended animation limit of GNNs.
Here, we need to add a remark: to simplify the presentations in this paper, we will ﬁrst take vanilla GCN as the base model example to illustrate our discoveries and proposed solutions in the method sections.
Meanwhile, empirical tests on several other existing GNNs, e.g., GAT Veliˇckovi´c et al.
and
L OOPY N ET Zhang et al.
(2018), will also be studied in the experiment section of this paper.
As illustrated in Figure 1, we provide the learning performance of the GCN model on the Cora dataset, where the learning settings (including train/validation/test sets partition, algorithm implementation and ﬁne-tuned hyper-parameters) are identical to those introduced in Kipf & Welling (2016).
The GCN model with the bias term disable of seven different depths, i.e., GCN(1-layer)-
GCN(7-layer), are compared.
Here, the layer number denotes the sum of hidden and output layers, which is also equal to the number of spectral graph convolutional layers involved in the model.
For instance, besides the input layer, GCN(7-layer) has 6 hidden layer and 1 output layer, both of which involve the spectral graph convolutional operations.
According to the plots, GCN(2-layer) and GCN(3-layer) have comparable performance, which both outperform GCN(1-layer).
Meanwhile, as the model depth increases from 3 to 7 , its learning performance on both the training set
[Figure]
and the testing set degrades greatly.
It is easy to identify that such degradation is not caused by overﬁtting the training data.
What’s more, much more surprisingly, as the model depth goes deeper toor more, it will suffer from the suspended animation problem and does not respond to the training data anymore.
(Similar phenomena can be observed for GCN (bias enabled) and GAT as illustrated by Figures 9 and 10 in the appendix of this paper, whose suspended animation limits are 8 and 5 , respectively.
Meanwhile, on L OOPY N ET , we didn’t observe such a problem as shown in Figurein the appendix, and we will state the reasons in Section 6 in detail.)
In this paper, we will investigate the causes of the GNNs’ suspended animation problem , and analyze if such a problem also exists in all other GNN models or not.
GNNs are very different from the traditional deep learning models, since the extensive connections among the nodes render their learning process no longer independent but strongly correlated.
Therefore, the existing solutions proposed to resolve such problems, e.g., residual learning methods used in ResNet for CNN He et al.
(2015), cannot work well for GNNs actually.
In this paper, several different novel graph residual terms will be studied for GNNs specially.
Equipped with the new graph residual terms , we will further introduce a new graph neural network architecture, namely graph residual neural network (GR ES N ET ), to resolve the observed problem.
Instead of merely stacking the spectral graph convolution layers on each other, the extensively connected high-ways created in GR ES N ET allow the raw features or intermediate representations of the nodes to be fed into each layer of the model.
We will study the effectiveness of the GR ES N ET architecture and those different graph residuals for several existing vanilla GNNs.
In addition, theoretic analyses on GR ES N ET will be provided in this paper as well to demonstrate its effectiveness from the norm-preservation perspective.
The remaining parts of this paper are organized as follows.
In Section 2, we will introduce the related work of this paper.
The suspended animation problem with the spectral graph convolutional operator will be discussed in Section 3, and the suspended animation limit will be analyzed in
Section 4.
Graph residual learning will be introduced in Section 5, whose effectiveness will be tested in Section 6.
Finally, we will conclude this paper in Section 7.
R ELATED W ORK
Graph Neural Network : Graph neural networks Monti et al.
; Atwood & Towsley (2016);
Masci et al.
; Kipf & Welling (2016); Battaglia et al.
; Bai et al.
; Scarselli et al.
; Zhou et al.
; Niepert et al.
(2016) have become a popular research topic in recent years.
Traditional deep models cannot be directly applied to graph data due to the graph interconnected structures.
Many efforts have been devoted to extend deep neural networks on graphs for representation learning.
GCN proposed in Kipf & Welling (2016) feeds the generalized spectral features into the convolutional layer for representation learning.
Similar to GCN, deep loopy graph neural network Zhang (2018) proposes to update the node states in a synchronous manner, and it introduces a spanning tree based learning algorithm for training the model.
L OOPY N ET accepts nodes’ raw features into each layer of the model, and it can effectively ﬁght against the suspended animation problem according to the studied in this paper.
GAT Veliˇckovi´c et al.
(2018) leverages masked self-attentional layers to address the shortcomings of GCN.
In this year, we have also witnessed some preliminary works on heterogeneous graph neural networks Wang et al.
Liu et al.
Similar to GCN, GEM Liu et al.
(2018) utilizes one single layer of attention to capture the impacts of both neighbors and network heterogeneity, which cannot work well on real-world complex networks.
Based on GAT, HAN Wang et al.
(2019) learns the attention coefﬁcients between the neighbors based on a set of manually crafted meta paths Sun et al.
(2011), which may require heavy human involvements.
D IF NN Zhang et al.
(2018) introduce a diffusive neural network for the graph structured data speciﬁcally, which doesn’t suffer from the oversmoothing problem due to the involvement of the neural gates and residual inputs for all the layers.
Due to the limited space, we can only name a few number of the representative graph neural network here.
The readers are also suggested to refer to page 1 , which provides a summary of the latest graph neural network research papers with code on the node classiﬁcation problem.
Residual Network : Residual learning Srivastava et al.
; He et al.
; Bae et al.
Han et al.
; Gomez et al.
; Tai et al.
Yu et al.
; Ahn et al.
; Li et al.
(2018a)
; Behrmann et al.
(2019) has been utilized to improve the learning performance, especially for the neural network models with very deep architectures.
To ease gradient-based training of very deep networks, Srivastava et al. (2015) introduces the highway network to allow unimpeded information ﬂow across several layers.
Innovated by the high-way structure,
He et al.
(2015) introduces the residual network to simplify highway network by removing the fusion gates.
After that, residual learning has been widely adopted for deep model training and optimization.
Bae et al.
(2016) introduces residual learning for image restoration via persistent homology-guided manifold simpliﬁcation; Tai et al.
; Li et al.
(2018a) introduces a recursive residual network for image resolution adjustment.
Some improvement of residual network has also been proposed in recent years.
A reversible residual network is introduced in Gomez et al.
(2017), where each layer’s activations can be reconstructed exactly from the next layer’s;
Han et al.
(2016) improves the conventional model shape with a pyramidal residual network instead; Yu et al.
(2017) introduce the dilated residual network to increases the resolution of output feature maps without reducing the receptive ﬁeld of individual neurons; and Ahn et al.
(2018) studies the cascading residual network as an accurate and lightweight deep network for image super-resolution.
Readers can also refer to He (2016) for a detailed tutorial on residual learning and applications in neural network studies.
S USPENDED A NIMATION P ROBLEM WITH GCN M ODEL
In this part, we will provide an analysis about the suspended animation problem of the spectral graph convolutional operator used in GCN to interpret the causes of the observations illustrated in Figure 1.
In addition, given an input network data, we will provide the theoretic bound of the suspended animation limit for the GCN model.
3. V ANILLA G RAPH C ONVOLUTIONAL N ETWORK R EVISIT
To make this paper self-contained, we will provide a brief revisit of the vanilla GCN model in this part.
Formally, given an input network G =
( V , E ) , its network structure information can be denoted as an adjacency matrix A
= { 0 , 1 } n × n (where |V| = n ).
GCN deﬁnes the normalized adjacency matrix of the input network as ˆ A = ˜ D − 12 ˜
A ˜ D − 12 , where
˜ A =
A + I n × n and ˜ D is the diagonal matrix of ˜ A with entry ˜ D
( i, i ) =
(cid:80)
j ˜ A ( i, j ) .
Given all the nodes in V together with their raw feature inputs X ∈ R n × d x ( d x denotes the node raw feature length), GCN deﬁnes the spectral graph convolutional operator to learn the nodes’ representations as follows:
[Equation]
(1) where W ∈ R d x × d h is the variable involved in the operator.
Furthermore, GCN can involve a deep architecture by stacking multiple spectral graph convolutional layers on each other, which will be able to learn very complex high-level representations of the nodes.
Here, let’s assume the model depth to be K (i.e., the number of hidden layers and output layer), and the corresponding node representation updating equations can be denoted as:
[Equation]
3. S USPENDED A NIMATION P ROBLEM WITH GCN
By investigating the spectral graph convolutional operator deﬁned above, we observe that it actually involves two sequential steps:
[Equation]
where the ﬁrst term on the right-hand-side deﬁnes a 1-step Markov chain (MC or a random walk) based on the graph and the second term is a fully-connected (FC) layer parameterized by variable
W ( k − 1) .
Similar observations have been reported in Li et al.
(2018b) as well,
but it interprets the spectral graph convolutional operator in a different way as the Laplacian smoothing operator used in mesh smoothing in graphics instead.
Therefore, stacking multiple spectral graph convolutional layers on top of each other is equivalent to the stacking of multiple 1-step Markov chain layers and fully-connected layers in a crosswise way.
Considering that the variables W ( k − 1) for the vector dimension adjustment are shared among all the nodes, given two nodes with identical representations, the fully-connected layers (parameterized by W ( k − 1) ) will still generate identical representations as well.
In other words, the fully-connected layers with shared variables for all the nodes will not have signiﬁcant impacts on the convergence of Markov chain layers actually.
Therefore, in the following analysis, we will simplify the model structure by assuming the mapping deﬁned by fully-connected layers to be the identity mapping.
We will investigate the Markov chain layers closely by picking them out of the model to compose the Markov chain of multiple steps:
[Equation]
Meanwhile, the Markov chain layers may converge with k layers iff T ( k )
=
T ( k − 1) , i.e., the representations before and after the updating are identical (or very close), which is highly dependent on the input network structure, i.e., matrix ˆ A , actually.
D EFINITION 1.
(Irreducible and Aperiodic Network):
Given an input network G =
( V , E ) , G is irreducible iff for any two nodes v i , v j ∈ V , node v
i is accessible to v j .
Meanwhile, G is aperiodic iff G is not bipartite.
L EMMA 1.
Given an unweighted input graph G , which is irreducible, ﬁnite and aperiodic, if its corresponding matrix is asymmetric, starting from any initial distribution vector x ∈ R n × 1 ( x ≥and (cid:107) x (cid:107) 1 = 1 ), the Markov chain operating on the graph has one unique stationary distribution vector π ∗ such that lim t →∞ ˆ A t x = π ∗ , where π ∗ ( i ) =
d ( v i ) 2 |E| .
Meanwhile, if matrix ˆ A is symmetric, the stationary distribution vector π ∗ will be a uniform distribution over the nodes, i.e., π ∗ ( i ) = 1 n .
Based on the above Lemma 1, we can derive similar results for the multiple Markov chain layers in the GCN model based on the nodes’ feature inputs, which will reduce the learned nodes’ representations to the stationary representation matrix.
T HEOREM 1.
Given a input network G =
( V , E ) , which is unweighted, irreducible, ﬁnite and aperiodic, if there exist enough nested Markov chain layers in the GCN model, it will reduce the nodes’ representations from the column-normalized feature matrix X ∈ R n × d x to the stationary representation Π ∗ =
[ π ∗ , π ∗ , · · · , π ∗ ] ∈ R n × d x .
Furthermore, if G is undirected, then the stationary representation will become Π ∗ = 1 n · 1 n × d x .
Theorem 1 illustrates the causes of the GNNs’ suspended animation problem.
Proofs of Lemmaand Theorem 1 are provided in the appendix attached to this paper at the end.
S USPENDED A NIMATION L IMIT A NALYSIS
Here, we will study the suspended animation limit of the GCN model based on its spectral convolutional operator analysis, especially the Markov chain layers.
4. S USPENDED A NIMATION L IMIT BASED I NPUT N ETWORK S TRUCTURE
Formally, we deﬁne the suspended animation limit of GCN as follows:
D EFINITION 2.
(Suspended Animation Limit): The suspended animation limit of GCN on network
G is deﬁned as the smallest model depth τ such that for any nodes’ column-normalized featured matrix input X in the network G the following inequality holds:
[Equation]
For representation convenience, we can also denote the suspended animation limit of GCN deﬁned on network G as ζ ( G ) (or ζ for simplicity if there is no ambiguity problems).
Based on the above deﬁnition, for GCN with identity FC mappings, there exists a tight bound of the suspended animation limit for the input network.
T HEOREM 2.
Let 1 ≥ λ 1 ≥ λ 2 ≥ · · · ≥ λ n be the eigen-values of matrix ˆ A deﬁned based on network G , then the corresponding suspended animation limit of the GCN model on G is bounded
[Equation]
In the case that the network G is a d-regular, then the suspended animation limit of the GCN model on G can be simpliﬁed as
[Equation]
The suspended animation limit bound derived in the above theorem generally indicates that the network structure G determines the maximum allows depth of GCN.
Among all the eigen-values of ˆ A deﬁned on network G , λ 2 measures how far G is from being disconnected; and λ n measures how far G is from being bipartite.
In the case that G is reducible (i.e., λ 2 = 1 ) or bipartite (i.e., λ n = − 1 ), we have ζ → ∞ and the model will not suffer from the suspended animation problem .
In the appendix of Kipf & Welling (2016), the authors introduce a naive-residual based variant of
GCN, and the sepctral graph convolutional operator is changed as follows (the activation function is also changed to sigmoid function instead):
[Equation]
For the identity fully-connected layer mapping, the above equation can be reduced to the following lazy Markov chain based layers
[Equation]
Such a residual term will not help resolve the problem, and it will still suffer from the suspended animation problem with the following suspended animation limit bound:
C OROLLARY 1.
Let 1 ≥ λ 1 ≥ λ 2 ≥ · · · ≥ λ n be the eigen-values of matrix ˆ A deﬁned based on network G , then the corresponding suspended animation limit of the GCN model (with lazy Markov chain based layers) on G is bounded
[Equation]
Proofs to Theorem 2 and Corollary 1 will be provided in the appendix as well.
4. O THER P RACTICAL F ACTORS I MPACTING THE S USPENDED A NIMATION L IMIT
According to our preliminary experimental studies, GCN is quite a sensitive model.
Besides the impact of input network explicit structures (e.g., network size n , directed vs undirected links, and network eigenvalues) as indicated in the bound equation in Theorem 2, many other factors can also inﬂuence the performance of GCN model a lot, which are summarized as follows:
• Network Degree Distribution :
According to Theorem 1, if the input network G is directed and unweighted, the Markov chain layers at convergence will project the features to Π ∗ =
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
are correlated based on network structure, and are determined by the raw input features.
[ π ∗ , π ∗ , · · · , π ∗ ] , where π ∗ ( i ) is determined by the degree of node v i in the network.
For any two nodes v i , v j ∈ V , the differences of their learned representation can be denoted as
[Equation]
According to Newman (2003)
[Table]
the node degree distributions in most complex networks follow the power-law Faloutsos et al.
(1999), i.e., majority of the nodes are of very small degrees.
Therefore, for massive nodes in the input network with the same (or close) degrees, the differences between their learned representations will become not distinguishable.
• Raw Feature Coding :
Besides the input network, the raw feature coding can also affect the learning performance greatly.
Here, we can take the GCN
with one single Markov chain layer and one identity mapping layer.
For any two nodes v i , v j ∈ V with raw feature vectors
X ( i, :) and X ( j, :) , we can denote the differences between their learned representations as follows:
[Equation]
Different from Equation (11), analysis in the above equation is not based on the stationary representation of the nodes, and it is based on the MC-layer representation as indicated in
Equation (3) and Equation (4).
For the one-hot feature coding used in the source code of
GCN Kipf & Welling (2016) and other GNNs, matrix X can be also very sparse as well.
Meanwhile,
vector ˆ
A
( i, :
− ˆ
A
( j, :) is also a sparse vector, which renders the right-handside term to be a very small value.
• Training Set Size : Actually, the nodes have identical representation and the same labels will not degrade the learning performance of the model.
However, if such node instances actually belong to different classes, it will become a great challenge for both the training and test stages of the GCN model.
Gradient Vanishing/Exploding : Similar to the existing deep models, deep GNNs will also suffer from the gradient vanishing/exploding problems Pascanu et al.
(2012), which will also greatly affect the learning performance of the models.
Although these factors mentioned above are not involved in the suspended animation limit bound representation, but they do have great impacts on the GCN model in practical applications.
In the following section, we will introduce graph residual network GR ES N ET , which can be useful for resolving such a problem for GCN.
G RAPH R ESIDUAL N ETWORK
Different from the residual learning in other areas, e.g., computer vision He et al.
(2015), where the objective data instances are independent with each other, in the inter-connected network learning setting, the residual learning of the nodes in the network are extensively connected instead.
It renders the existing residual learning strategy less effective for improving the performance of GCN.
5. G RAPH R ESIDUAL L EARNING
Residual learning initially introduced in He et al.
(2015) divides the objective mapping into two parts: the inputs and the residual function.
For instance, let H ( x ) be the objective mapping which projects input x to the desired domain.
The ResNet introduced in He et al.
(2015) divides H ( x ) as
F ( x ) + R ( x ) (where R ( x ) = x is used in He et al.
This reformulation is motivated by the counterintuitive phenomena about the degradation problem observed on the deep CNN.
Different
[Table] [Table]
imGN4NMqCRFrtjyUJhKgjHJCyAjoTlDObOEMi1sVsImVFOGtqayLcFb/
[Table] [Table]
+AIjAkVY=</latexit>
SGC
SGC SGC
SGC SGC
SGC SGC
G <latexit sha1_base64="/Vt+0RdSS5A/sbViRuabh3HoiBU
[Table] [Table]
[Table]
̂
A  ̂
A
[Table]
A
[Table]
A
[Table]
[Table]
[Table]
[Table]
A
[Table]
A
[Table]
A
[Table] [Table]
/u9R6Y0j+Q9zGM2DMlE8oBTAkYa2dW6NyWQeiGBqR+kD1lWH9k1p+EsgNeJW5AaKtAe2V/eOKJJyCRQQbQeuE4Mw5Qo4FSwrOIlmsWEzsiEDQyVJGR6mC6yZ/jcKGMcRMo8CXih/t5ISaj1PPTNZJ5Rr3q5+J83SCC4GqZcxgkwSZeHgkRgiHBeBB5zxSiIuSGEKm6yYjolilAwdVVMCe7ql9dJt9lwnYZ716y1ros6yugUnaEL5KJL1EK3qI06iKIn9Ixe0ZuVWS/Wu/WxHC1Zxc4J+gPr8wfmo5RV</latexit> softmax
ReLU softmax ReLU softmax
ReLU from the learning settings of CNN, where the data instances are assumed to be independent, the nodes inside the input network studied in GCN are closely correlated.
Viewed in such a perspective, new residual learning mechanism should be introduced for GCN speciﬁcally.
Here, we need to add a remark : For the presentation simplicity in this paper, given the objective
H ( x ) =
F ( x ) + R ( x ) , we will misuse the terminologies here: we will name F ( x ) as the approximated mapping of H ( x ) , and call R ( x ) as the graph residual term .
Formally, by incorporating the residual learning mechanism into the GCN model, the node representation updating equations (i.e.,
Equation 2) can be rewritten as follows:
[Equation] [Equation]
The graph residual term R (cid:0) H ( k − 1) , X ; G (cid:1) , ∀ k ∈ { 1 , 2 , · · · , K } can be deﬁned in different ways.
We have also examined to put R (cid:0) H ( k − 1) , X ; G (cid:1) outside of the ReLU( · ) function for the hidden layers
(i.e., H ( k ) = ReLU (cid:16) ˆ AH ( k − 1) W ( k − 1) (cid:17) + R (cid:0) H ( k − 1) , X ; G (cid:1) ), whose performance is not as good as what we show above.
In the appendix of Kipf & Welling (2016), by following the ResNet (CNN)
He et al.
(2015), the graph residual term R (cid:0) H ( k − 1) , X ; G (cid:1) is simply deﬁned as H ( k − 1) , which is named as the naive residual term in this paper (Here, term “naive” has no disparaging meanings).
However, according to the studies, such a simple and independent residual term for the nodes fail to capture information in the inter-connected graph learning settings.
5. GR ES N ET A RCHITECTURE
In this paper, we introduce several other different representations of the graph residual term , which are summarized in Table 1.
If feature dimension adjustment is needed, e.g., for raw residual term, an extra variable matrix W adj can be added to redeﬁne the terms (which are not shown in this paper).
For the graph residual term representations in Table 1, naive residual and raw residual are based on the assumption that node residuals are independent and determined by either the current state or the raw features.
Meanwhile, the graph naive residual and graph raw residual assume the residual terms of different nodes are correlated instead, which can be computed with the current state or raw features.
We also illustrate the architectures of vanilla 2-layer GCN and the 7-layer
GR ES N ET s (taking GCN as the base model) with different graph residual terms in Figure 2.
We
[Figure]
need to add a remark here that similar solutions to the raw residual terms have been studied in the latest research papers Wang et al.
; Wang & Gupta (2018) as well.
However, in this paper, we aim to generalize such techniques as a uniﬁed framework, which can accept different types of residual terms not just the raw residual term.
Vanilla GCN : For the vanilla GCN network used in Kipf & Welling (2016), i.e., the left plot of
Figure 2, given the inputs G and X , it employs two SGC layers to project the input to the objective labels.
For the intermediate representations, the hidden layer length is 16 , and ReLU is used as the activation function, whereas softmax is used for output label normalization.
GR ES N ET Network : For the GR ES N ET network, i.e., the right four models in Figure 2, they accept the identical inputs as vanilla GCN but will create graph residual terms to be added to the intermediate representations.
Depending on the speciﬁc residual term representations adopted, the corresponding high-way connections can be different.
For the hidden layers involved in the models, their length is also 16 , and ReLU is used as the activation function for the intermediate layers.
By comparing the output ˆ Y learned by the models against the ground-truth Y of the training instances, all the variables involved in the model, i.e., Θ , can be effectively learned with the backpropagation algorithm to minimize the loss functions (cid:96) ( Y , ˆ Y ; Θ ) (or (cid:96) ( Θ ) for simplicity).
In the following part, we will demonstrate that for the GR ES N ET model added with graph residual terms.
It can effectively avoid dramatic changes to the nodes’ representations between sequential layers.
5. G RAPH R ESIDUAL L EARNING E FFECTIVENESS A NALYSIS
In this part, we will illustrate why the inclusion of the graph residual learning can be effective for learning deep graph neural networks.
Here, we assume the ultimate model that we want to learn as H : X → Y , where X and Y denote the feature and label spaces, respectively.
For analysis simplicity, we have some assumptions about the function H Zaeemzadeh et al.
A SSUMPTION 1.
Function H is differentiable, invertible and satisﬁes the following conditions:
• ∀ x , y , z ∈ X with bounded norm, ∃ α > 0 , (cid:107) ( H (cid:48) ( x ) − H (cid:48) ( y ))
z (cid:107) ≤ α · (cid:107) x − y (cid:107) · (cid:107) z (cid:107) , • ∀ x , y ∈ X with bounded norm, ∃ β > 0 , (cid:13)(cid:13)
H − 1 ( x ) − H − 1 ( y ) (cid:13)(cid:13) ≤ β · (cid:107) x − y (cid:107) , • ∃ x ∈ X with bounded norm such that Det ( H (cid:48) ( x )) > 0 .
In the above conditions, terms α and β are constants.
To model the function H , the GR ES N ET actually deﬁnes a sequence of K sequential mappings with these K layers:
[Equation]
where x ( k − 1) and x ( k ) denote the intermediate node representations serving as input and output of the k th layer.
F ( k − 1) ( · ) and R ( k − 1) ( · ) denote the function approximation and residual term
[Table]
learned by the k − 1 th layer of the model.
When training these K sequential mappings, we have the following theorem hold for the representation gradients in the learning process.
T HEOREM 3.
Let H denote the objective function that we want to model, which satisﬁes Assumption 1, in learning the K -layer GR ES N ET model, we have the following inequality hold:
[Equation]
where δ ≤ c · log (2 K ) K and c = c 1 · max { α · β · (1 + β ) , β · (2 + α ) + α } for some c 1 > 0 .
Proof of Theorem 3 will be provided in the appendix.
The above theorem indicates that in the learning process, the norm of loss function against the intermediate representations doesn’t change signiﬁcantly between sequential layers.
In other words, GR ES N ET can maintain effective representations for the inputs and overcome the suspended animation problem .
In addition, we observe that the bound of the gap term δ , i.e., c · log (2 K ) K , decreases as K increases (when K ≥ 2 ).
Therefore, for deeper GR ES N ET , the model will lead to much tighter gradient norm changes, which is a desired property.
In the following section, we will provide the experimental results of GR ES N ET compared against their vanilla models on several graph benchmark datasets.
E XPERIMENTS
To demonstrate the effectiveness of GR ES N ET in improving the learning performance for graph neural networks with deep architectures, extensive experiments will be done on several graph benchmark datasets.
Similar to the previous works on node classiﬁcation Kipf & Welling (2016);
Veliˇckovi´c et al.
(2018), the graph benchmark datasets used in the experiments include Cora, Citeseer and Pubmed from Sen et al.
For fair comparison, we follow exactly the same experimental settings as Kipf & Welling (2016) on these datasets.
In this paper, we aim at studying the suspended animation problem with the existing graph neural networks, e.g., GCN Kipf & Welling (2016), GAT Veliˇckovi´c et al.
and L OOPY N ET Zhang (2018), where L OOPY N ET is not based on the spectral graph convolutional operator.
We also aim to investigate the effectiveness of these proposed graph residual terms in improving their learning performance, especially for the models with deep architectures.
In addition, to make the experiments self-contained, we also provide the latest performance of the other baseline methods on the same datasets in this paper, which include state-of-the-art graph neural networks, e.g., APPNP Klicpera et al.
GOCN Jiang et al.
and GraphNAS Gao et al.
existing graph embedding models, like DeepWalk Perozzi et al.
Planetoid Yang et al.
and MoNet Monti et al.
(2016) and representation learning approaches, like ManiReg Belkin et al.
SemiEmb Weston et al.
LP Zhu et al.
and ICA Lu & Getoor (2003).
[Table]
Reproducibility : Both the datasets and source code used in this paper can be accessed via link 2 .
Detailed information about the server used to run the model can be found at the footnote 3 .
6. E FFECTIVENESS OF THE G RAPH R ESIDUAL T ERMS
In addition to Figure 1 for GCN (bias disabled) on the Cora dataset, as shown in Figures 9in the appendix, for the GCN (bias enabled) and GAT with deep architectures, we have observed similar suspended animation problems.
Meanwhile, the performance of L OOPY N ET is different.
Since L OOPY N ET is not based on the spectral graph convolution operator, which accepts nodes’ raw features in all the layer (it is quite similar to the raw residual term introduced in this paper).
As the model depth increase, performance of L OOPY N ET remains very close but converge much more slowly.
By taking GCN as the base model, we also show the performance of GR ES N ET with different residual terms in Figure 3.
By comparing these plots with Figure 1, both naive and graphnaive residual terms help stabilize the performance of deep GR ES N ET (GCN)s.
Meanwhile, for the raw and graph-raw residual terms, their contributions are exceptional.
With these two residual terms, deep GR ES N ET (GCN)s can achieve even better performance than the shallow vanilla GCNs.
Besides the results on the Cora dataset, in Table 2, we illustrate the best observed performance by GR ES N ET with different residual terms based on GCN, GAT and L OOPY N ET base models respectively on all the datasets.
Both the best accuracy score and the achieved model depth are provided.
According to the results, for the vanilla models, GCN, GAT and L OOPY N ET can all obtain the best performance with shallow architectures.
For instance on Cora, GCN(2-layer) obtains 0.815; GAT(2-layer) gets 0.830; and L OOPY N ET (2-layer) achieves 0.826, respectively.
Added with the residual terms, the performance of all these models will get improved.
In addition, deep
GR ES N ET (GCN), GR ES N ET (GAT) and GR ES N ET
(L OOPY N ET ) will be able to achieve much better results than the shallow vanilla models, especially the ones with the graph-raw residual terms.
The best scores and the model depth for each base model on these datasets are also highlighted.
The time costs of learning the GR ES N ET model is almost identical to the required time costs of learning the vanilla models with the same depth, which are not reported in this paper.
6. A C OMPLETE C OMPARISON WITH E XISTING N ODE C LASSIFICATION M ETHODS
Besides the comparison with GCN, GAT and L OOPY N ET shown in Table 2, to make the experimental studies more complete, we also compare GR ES N ET (GCN), GR ES N ET (GAT) and GR ESN ET (L OOPY N ET ) with both the classic and the state-of-the-art models, whose results are provided in Table 3.
In the table, we didn’t indicate the depth of the GR ES N ET models and results of GCN,
GAT and L OOPY N ET (shown in Table 2 already) are not included.
According to the results, compared against these baseline methods, GR ES N ET s can also outperform them with great advantages.
Without the complex model architecture extension or optimization techniques used by the latest methods APPNP Klicpera et al.
GOCN Jiang et al.
and GraphNAS Gao et al.
(2019), adding the simple graph residual terms into the base models along can already improve the learning performance greatly.
C ONCLUSION
In this paper, we focus on studying the suspended animation problem with the existing graph neural network models, especially the spectral graph convolutional operator.
We provide a theoretic analysis about the causes of the suspended animation problem and derive the bound for the maximum allowed graph neural network depth, i.e., the suspended animation limit.
To resolve such a problem, we introduce a novel framework GR ES N ET , which works well for learning deep representations from the graph data.
Assisted with these new graph residual terms, we demonstrate that GR ES N ET can effectively resolve the suspended animation problem with both theoretic analysis and empirical experiments on several benchmark node classiﬁcation datasets.
R EBUTTAL N EW M ATERIALS 8. E XPERIMENTAL R ESULTS ON PPI AND R EDDIT L ARGER D ATASETS
We added the experimental results on the PPI and Reddit datasets in the following table.
Detailed information about datasets is as follows: • PPI :
Node # : 14,755; Edge # : 225,270; Feature # : 50; Class # :• Reddit :
Node # : 232, 965; Edge # : 11,606,919; Feature # : 602; Class # :
We need to mention that GAT used in GResNet is very slow on large datasets (it is not our problem but the problem with the GAT base model), we cannot get the results of GResNet(GAT) out on
Reddit during the rebuttal period.
Since PPI and Reddit are not common used in other graph neural network papers, so many of the entries in Table 4 are not provided. We will add these results in the camera-ready version of this paper later [Table] [Table]
APPNP (Klicpera et al. [Table] (Jiang et al. - - Graph-S (Gao et al.
[Table]
GCN (Kipf & Welling (2016)) 0.
0.930 GAT ( [Table] [Table] [Table]
[Table]
[Table]
[Table]
- - GraphSAGE (Hamilton et al
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
8. E XPERIMENTAL R ESULTS ON D EEPER M ODELS 8.2. D EEPER M ODEL P ERFORMANCE S UMMARY
As suggested by the reviewers
[Table]
we also add the studies of GCN, GR ES N ET (GCN, naive), GR ESN ET (GCN, graph-naive), GR ES N ET (GCN, raw) and GR ES N ET (GCN, graph-raw) with deeper architectures (more than 7 layers) on the Cora dataset in the following table and plots
As illustrated in Table
[Table]
we show the experimental results on Cora obtained by the models with 2, 10, 20, 30, 40, 50 layers, respectively.
According to the numbers, GCN cannot really work well for all the layers other than
[Table]
and the performance of the naive and graph-naive residual terms is not good neither.
However, the performance of the raw and graph-raw residual terms is exceptionally great.
For instance, for the model with 50 layers, GR ES N ET (GCN, raw) and GR ES N ET (GCN, graph-raw) can still achieve 0.81 and 0.82 as the accuracy scores.
[Table]
8.2. D EEPER M ODEL L EARNING L OSS ON T RAINING AND T ESTING S ETS PER E POCH
In addition to Table 5, we also illustrate the performance of the models in the learning process in
Figures 4-8, where both the training accuracy and testing accuracy on each epoch are provided.
According to the results, GCN and GR ES N ET (GCN, graph-naive) will fail to work when the model architecture goes deeper.
GR ES N ET (GCN, naive) can slightly revive the models for 10 layers, but will also fail to work for deeper architectures, e.g., 20, 30, 40 and 50 layers.
However, GR ESN ET (GCN, raw) and GR ES N ET (GCN, graph-raw) work very great for all these different depth in the plot, which also illustrate the effectiveness of the raw and graph-raw residual terms proposed in this paper.
[Figure]
[Figure] [Figure] [Figure] [Figure]
We have also studied even deeper model architectures on GR ES N ET (GCN, raw) and GR ESN ET (GCN, graph-raw) to 100 layers, which can still perform well
S TUDIES ON O THER I MPACTING F ACTORS
Via changes to the original raw data, we will be able to get different samples of the data with different properties, e.g., degree distribution, raw features, and the training set size.
We will illustrate the learning results based on these different settings as follows for the reviewers’ information.
9. N ODE D EGREE D ISTRIBUTION 9. R AW F EATURE E NCODING 9. T RAINING S ET S IZE
[Figure]
[Figure]
[Figure]
10. P ROOFS OF T HEOREM AND L EMMA 10.2. P ROOF OF L EMMA
L EMMA 1.
Given an irreducible, ﬁnite and aperiodic graph G , starting from any initial distribution vector x ∈ R n × 1 ( x ≥ 0 and (cid:107) x (cid:107) 1 = 1 ), the Markov chain operating on the graph has one unique stationary distribution vector π ∗ such that lim t →∞ ˆ A t x = π ∗ , where π ∗ ( i ) =
d ( v i ) 2 |E| .
If matrix ˆ A is symmetric (i.e., G is undirected), π ∗ will be a uniform distribution over the nodes, i.e., π ∗ ( i ) = 1 n .
Proof.
The stationary distribution vector existence and uniqueness has been proved in Norris (1998).
Here, we need to identify on vector π at convergence such that ˆ A π = π , i.e.,
[Equation]
According to the deﬁnition of ˆ A , it is easy to have
[Equation]
17) where w i,j denotes the initial connection weight between v
i and v j .
For the unweighted network, w i,j will be in { 0 , 1 } indicating if v i and v j are connected or not.
Notation d w ( i ) denotes the rough degree of node v i in the network subject to the weight w : E ∈ R , which sums the weight of edges connected to the nodes in the network.
So, it is enough to have π ( j ) ∝ d w ( j ) .
More precisely, we can set
[Equation] [Equation]
In this case,
[Equation]
Meanwhile, for the symmetric and normalized adjacency matrix ˆ A , we can prove the stationary distribution π ∗ ( i ) = 1 n in a similar way, which concludes the proof.
10.2. P ROOF OF T HEOREM
T HEOREM 1.
Given a input network G =
( V , E ) , which is unweighted, irreducible, ﬁnite and aperiodic, if there exist enough nested Markov chain layers in the GCN model, it will reduce the nodes’ representations from the column-normalized feature matrix X ∈ R n × d x to the stationary representation Π ∗ =
[ π ∗ , π ∗ , · · · , π ∗ ] ∈ R n × d x .
Furthermore, if G is undirected, then the stationary representation will become Π ∗ = 1 n · 1 n × d x .
Proof.
This theorem can be proved based on Lemma 1.
For any initial state distribution vector x ∈ R |V|× 1 , for the Markov chain at convergence, we have
[Equation]
We misuse the notation ˆ A ∗ =
lim t →∞ ˆ A t .
In this case,
[Equation]
(21) which together with Lemma 1 conclude the proof.
10.2. P ROOF OF T HEOREM
Prior to introducing the proof of Theorem 2, we will introduce the following lemma ﬁrst.
L EMMA 2.
For any vector x ∈ R n , the following inequality holds:
[Equation]
Proof.
According to H¨older’s inequality H¨older (1889), for ∀ a , b ∈ R n × 1 and r > 1 , n (cid:88) i = |
[Equation]
i
[Equation]
| ≤
[Equation]
|
[Equation]
| r
[Equation]
= | b ( i ) |
rr −(cid:33) 1 − 1 r .
Let | a ( i ) | = | x i | p , | b ( i ) |
= 1 and r = qp , n (cid:88)
i =
| x i | p = n (cid:88)
i =
| x i | p ·≤ (cid:32) n (cid:88) i
=
( | x i | p ) qp (cid:33) pq (cid:32) n (cid:88) i
=
qq − p (cid:33) 1 − pq =
(cid:32) n (cid:88) i =
| x i | q (cid:33) pq ( n ) 1 − pq .
Therefore, (cid:107) x (cid:107) p = (cid:32) n (cid:88) i =
| x ( i ) | p (cid:33) 1 p ≤   (cid:32) n (cid:88)
i =
| x i | q (cid:33) pq ( n ) 1 − pq 

[Equation] [Equation]
T HEOREM 2.
Let 1 ≥ λ 1 ≥ λ 2 ≥ · · · ≥ λ n be the eigen-values of matrix ˆ A deﬁned based on network G , then the corresponding suspended animation limit of the GCN model on G is tightly bounded
[Equation]
In the case that the network G is a d-regular, then the suspended animation limit of the GCN model on G can be simpliﬁed as
[Equation]
Proof.
Instead of proving the above inequality directly, we propose to prove that ζ is suspended animation limit by the following inequality instead
[Equation]
which can derive the following inequality according to Lemma 2:
[Equation]
Let v 1 , v 2 , · · · , v n be the eigenvectors of ˆ A and ∀ x
[Equation]
where α i = (cid:104) x , v i (cid:105) .
Therefore, we have
[Equation]
Considering that λ ζ 1 = 1 and v 1 = [ 1 √ n , 1 √ n , · · · , 1 √ n ] (cid:62) , then
[Equation] [Equation]
32) and
[Equation]
33) where
Therefore, we have
[Equation]
34) where λ max = max {| λ 2 | , | λ 3 | , · · · , | λ n |} = max { λ 2 , | λ n |} .
Therefore, to ensure
[Equation]
we can have
[Equation]
10.2. P ROOF OF C OROLLARY
C OROLLARY 1.
Let 1 ≥ λ 1 ≥ λ 2 ≥ · · · ≥ λ n be the eigen-values of matrix ˆ A deﬁned based on network G , then the corresponding suspended animation limit of the GCN model (with lazy Markov chain based layers) on G is tightly bounded
[Equation]
Proof.
For the lazy Markov chain layer, we have its updating equation as follows
[Equation]
It is easy to show that ˜ A is positive deﬁnite and we have its eigen-values λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 .
Therefore,
[Equation]
which together with Theorem 2 conclude the proof.
10. P ROOF OF T HEOREM
Prior to introducing the proof of Theorem 3, we ﬁrst introduce the following lemma.
L EMMA 3.
For any non-singular matrix I + M , we have
[Equation]
where σ max ( · ) and σ min ( · ) denote the maximum and minimum singular values of the input matrix, respectively.
Proof.
Due to the triangle inequality, the upper bound is easy to prove:
[Equation]
In the case that σ max ( M ) ≥ 1 , the lower bound is trivial to prove since I + M is non-singular, we have
[Equation]
Meanwhile, in the case that σ max ( M ) < 1 , it is easy to know that | λ max ( M ) | < 1 , where λ max ( · ) denotes the latest eigenvalue of the input matrix.
[Equation]
which concludes the proof for the lower bound.
T HEOREM 3.
Let H denote the objective function that we want to model, which satisﬁes Assumption 1, in learning the K -layer GR ES N ET model, we have the following inequality hold:
[Equation] [Equation]
where δ ≤ c log (2 K ) K and c = c 1 max { αβ (1 + β ) , β (2 + α ) + α } for some c 1 > 0 .
Proof.
We can represent the Jacobian matrix J of x ( k ) with x ( k − 1) .
Therefore, we have
[Equation] [Equation]
Matrix J can be rewritten as J
= I + ∇ F ( k − 1) ( x ( k − 1) ) , where
[Equation]
t (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Meanwhile, it is easy to know that
[Equation]
Based on the above lemma, we have
[Equation]
where σ = σ max ( ∇ F ( k − 1) ( x ( k − 1) )) .
Furthermore
[Equation]
we know that
[Equation]
where (cid:107)·(cid:107)
L denotes the Lipschitz seminorm of the input function and it is deﬁned as
[Equation]
Meanwhile, according to the Theorem 1 in Bartlett et al.
(2018) (whose proof will not be introduced here), we know that
[Equation]
which concludes the proof.
In the above equation, K denotes the layer depth of the model.