G RAPH N EURAL N ETWORKS E XPONENTIALLY L OSE E XPRESSIVE P OWER FOR N ODE C LASSIFICATION
A BSTRACT
Graph Neural Networks (graph NNs) are a promising deep learning approach for analyzing graph-structured data.
However, it is known that they do not improve (or sometimes worsen) their predictive performance as we pile up many layers and add non-lineality.
To tackle this problem, we investigate the expressive power of graph NNs via their asymptotic behaviors as the layer size tends to inﬁnity.
Our strategy is to generalize the forward propagation of a Graph Convolutional Network (GCN), which is a popular graph NN variant, as a speciﬁc dynamical system.
In the case of a GCN, we show that when its weights satisfy the conditions determined by the spectra of the (augmented) normalized Laplacian, its output exponentially approaches the set of signals that carry information of the connected components and node degrees only for distinguishing nodes.
Our theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra.
To demonstrate this, we characterize the asymptotic behavior of GCNs on the Erd ˝ os – R ´ enyi graph.
We show that when the Erd˝os – R´enyi graph is sufﬁciently dense and large, a broad range of GCNs on it suffers from the “information loss” in the limit of inﬁnite layers with high probability.
Based on the theory, we provide a principled guideline for weight normalization of graph NNs.
We experimentally conﬁrm that the proposed weight scaling enhances the predictive performance of GCNs in real data 1 .
I NTRODUCTION
Motivated by the success of Deep Learning (DL), several attempts have been made to apply DL models to non-Euclidean data, particularly, graph-structured data such as chemical compounds, social networks, and polygons.
Recently, Graph Neural Networks (graph NNs)
(Duvenaud et al., 2015;
Li et al., 2016; Gilmer et al., 2017; Hamilton et al., 2017; Kipf & Welling, 2017; Nguyen et al., 2017;
Schlichtkrull et al., 2018; Battaglia et al., 2018; Xu et al., 2019; Wu et al., 2019a) have emerged as a promising approach.
However, despite their practical popularity, theoretical research of graph NNs has not been explored extensively.
The characterization of DL model expressive power , i.e., to identify what function classes DL models can (approximately) represent, is a fundamental question in theoretical research of DL.
Many studies have been conducted for Fully Connected Neural Networks (FNNs) (Cybenko, 1989; Hornik, 1991;
Hornik et al., 1989; Barron, 1993; Mhaskar, 1993; Sonoda & Murata, 2017; Yarotsky, 2017) and Convolutional Neural Networks (CNNs) (Petersen & Voigtlaender, 2018; Zhou, 2018; Oono &
Suzuki, 2019).
For such models, we have theoretical and empirical justiﬁcation that deep and nonlinear architectures can enhance representation power (Telgarsky, 2016; Chen et al., 2018b; Zhou & Feng, 2018).
However, for graph NNs, several papers have reported that node representations go indistinguishable (known as over-smoothing ) and prediction performances severely degrade when we stack many layers (Kipf & Welling, 2017;
Wu et al., 2019b; Li et al., 2018).
Besides, Wu et al.
(2019a) reported that graph NNs achieved comparable performance even if they removed intermediate non-linear functions.
These studies posed a question about the current architecture and made us aware of the need for the theoretical analysis of the graph NN expressive power.
In this paper, we investigate the expressive power of graph NNs by analyzing their asymptotic behaviors as the layer size goes to inﬁnity.
Our theory gives new theoretical conditions under which neither layer stacking nor non-linearity contributes to improving expressive power.
We consider a speciﬁc dynamics that includes a transition deﬁning a Markov process and the forward propagation of a Graph Convolutional Network (GCN) (Kipf & Welling, 2017), which is one of the most popular graph NN variants, as special cases.
We prove that under certain conditions, the dynamics exponentially approaches a subspace that is invariant under the dynamics.
In the case of GCN, the invariant space is a set of signals that correspond to the lowest frequency of graph spectra and that have “no information” other than connected components and node degrees for a node classiﬁcation task whose goal is to predict the nodes’ properties in a graph.
The rate of the distance between the output and the invariant space is O (( sλ ) L ) where s is the maximum singular values of weights, λ is typically a quantity determined by the spectra of the (augmented) normalized Laplacian, and L is the layer size.
See Sections 3.3 (general case) and 4 (GCN case) for precise statements.
We can interpret our theorem as the generalization of the well-known property that if a ﬁnite and discrete Markov process is irreducible and aperiodic, it exponentially converges to a unique equilibrium and the eigenvalues of its transition matrix determine the convergence rate (see, e.g., Chung & Graham (1997)).
Different from the Markov process case, which is linear, the existence of intermediate non-linear functions complicates the analysis.
We overcame this problem by leveraging the combination of the ReLU activation function (Krizhevsky et al., 2012) and the positivity of eigenvectors of the Laplacian associated with the smallest positive eigenvalues.
Our theory enables us to investigate asymptotic behaviors of graph NNs via the spectral distribution of the underlying graphs.
To demonstrate this, we take GCNs deﬁned on the Erd˝os – R´enyi graph
G N,p , which has N nodes and each edge appears independently with probability p , for an example.
We prove that if log N pN =
o (1) as a function of N , any GCN whose weights have maximum singular values at most C (cid:113)
Np log( N/ε ) approaches the “information-less” invariant space with probability at least 1 − ε , where C is a universal constant.
Intuitively, if the graph on which we deﬁne graph NNs is sufﬁciently dense, graph-convolution operations mix signals on nodes fast and hence the feature maps lose information for distinguishing nodes quickly.
Our contributions are as follows:
• We relate asymptotic behaviors of graph NNs with the topological information of underlying graphs via the spectral distribution of the (augmented) normalized Laplacian.
• We prove that if the weights of a GCN satisfy conditions determined by the graph spectra, the output of the GCN carries no information other than the node degrees and connected components for discriminating nodes when the layer size goes to inﬁnity (Theorems 1, 2).
• We apply our theory to Erd˝os – R´enyi graphs as an example and show that when the graph is sufﬁciently dense and large, many GCNs suffer from the information loss (Theorem 3).
• We propose a principled guideline for weight normalization of graph NNs and empirically conﬁrm it using real data.
R ELATED W ORK
MPNN-type Graph NNs.
Since many graph NN variants have been proposed, there are several uniﬁed formulations of graph NNs (Gilmer et al., 2017; Battaglia et al., 2018).
Our approach is the closest to the formulation of Message Passing Neural Network (MPNN) (Gilmer et al., 2017), which uniﬁed graph NNs in terms of the update and readout operations.
Many graph NNs fall into this formulation such as Duvenaud et al.
, Li et al.
(2016), and Veliˇckovi´c et al.
Among others, GCN (Kipf & Welling, 2017) is an important application of our theory because it is one of the most widely used graph NNs.
In addition, GCNs are interesting from a theoretical research perspective because, in addition to an MPNN-type graph NN, we can interpret GCNs as a simpliﬁcation of spectral-type graph NNs (Henaff et al., 2015; Defferrard et al., 2016), that make use of the graph Laplacian.
Our approach, which considers the asymptotic behaviors graph NNs as the layer size goes to inﬁnity, is similar to Scarselli et al.
(2009), one of the earliest works about graph NNs.
They obtained node representations by iterating message passing between nodes until convergence.
Their formulation is general in that we can use any local aggregation operation as long as it is a contraction map .
Our theory differs from theirs in that we proved that the output of a graph NN approaches a certain space even if the local aggregation function is not necessarily a contraction map.
Expressive Power of Graph NNs.
Several studies have focused on theoretical analysis and the improvement of graph NN expressive power.
For example, Xu et al.
(2019) proved that graph NNs are no more powerful than the Weisfeiler – Lehman (WL) isomorphism test (Weisfeiler & A.A., 1968) and proposed a Graph Isomorphism Network (GIN), that is approximately as powerful as the WL test.
Although they experimentally showed that GIN has improved accuracy in supervised learning tasks, their analysis was restricted to the graph isomorphism problem.
Xu et al.
(2018) analyzed the non-asymptotic properties of GCNs through the lens of random walk theory.
They proved the limitations of GCNs in expander-like graphs and proposed a Jumping Knowledge Network (JKNet) to address the issue.
To handle the non-linearity, they linearized networks by a randomization assumption (Choromanska et al., 2015).
We take a different strategy and make use of the interpretation of ReLU as a projection onto a cone.
Recently, NT & Maehara (2019) showed that a GCN approximately works as a low-pass ﬁlter plus an MLP in a certain setting.
Although they analyzed ﬁnite-depth GCNs, our theory has similar spirits with theirs because our “information-less” space corresponds to the lowest frequency of a graph Laplacian.
Another point is that they imposed assumptions that input signals consist of low-frequent true signals and high-frequent noise, whereas we need not such an assumption.
Role of Deep and Non-linear Structures.
For ordinal DL models such as FNNs and CNNs, we have both theoretical and empirical justiﬁcation of deep and non-linear architectures for enhancing of the expressive power (e.g., Telgarsky (2016); Petersen & Voigtlaender (2018); Oono & Suzuki (2019)).
In contrast, several studies have witnessed severe performance degradation when stacking many layers on graph NNs (Kipf & Welling, 2017; Wu et al., 2019b).
Li et al.
(2018) reported that feature vectors on nodes in a graph go indistinguishable as we increase layers in several tasks.
They named this phenomenon over-smoothing .
Regarding non-linearity, Wu et al. (2019a) empirically showed that graph NNs achieve comparable performance even if we omit intermediate non-linearity.
These observations gave us questions about the current models of deep graph NNs in terms of their expressive power.
Several studies gave theoretical explanations of the over-smoothing phenomena for linear GNNs (Li et al., 2018; Zhang, 2019; Zhao & Akoglu, 2020).
We can think of our theory as an extension of their results to non-linear GNNs.
P ROBLEM S ETTING AND M AIN R ESULT 3. N OTATION
Let N + be the set of positive integers.
For N ∈ N + , we denote [ N ] := { 1 , . . .
, N } .
For a vector v ∈ R N , we write v ≥
if and only if v n ≥ 0 for all n ∈ [ N ] .
Similarly, for a matrix X ∈ R N × C , we write X ≥
if and only if X nc ≥ 0 for all n ∈ [ N ] and c ∈ [ C ] .
We say such a vector and matrix is non-negative .
(cid:104)· , ·(cid:105) denotes the inner product of vectors or matrices, depending on the context: (cid:104) u, v (cid:105) :
= u (cid:62) v for u, v ∈ R N and (cid:104) X, Y (cid:105) :
= tr( X T Y ) for X, Y ∈ R N × C .
1 P equals toif the proposition P is true else 0 .
For vectors v ∈ R N and w ∈ R C , v ⊗ w ∈ R N × C denotes the
Kronecker product of v and w deﬁned by ( v ⊗ w ) nc := v n w c .
For X ∈ R N × C , (cid:107) X (cid:107) F := (cid:104) X, X (cid:105)denotes the Frobenius norm of X .
For a vector v ∈ R N , diag( v ) :
= ( v n δ nm ) n,m ∈
[ N ] ∈ R N × N denotes the diagonalization of v .
I N ∈ R N × N denotes the identity matrix of size N .
For a linear operator P : R N → R M and a subset V ⊂ R N , we denote the restriction of P to V by P | V .
D YNAMICAL S YSTEM
Although we are mainly interested in GCNs, we develop our theory more generally using dynamical systems.
We will specialize to the GCNs in Section 4.
For N, C, H l ∈ N + ( l ∈ N + ), let P ∈ R N × N be a symmetric matrix and W lh ∈ R C × C for l ∈ N + and h ∈
[ H l ] .
We deﬁne f l : R N × C → R N × C by f l ( X ) :
= MLP l ( PX ) .
Here,
MLP l : R N × C → R N × C is the l -th multi-layer perceptron common to all nodes (Xu et al., 2019) and is deﬁned by MLP l ( X ) :
= σ ( · · · σ ( σ ( X ) W l 1 ) W l 2 · · · W lH l ) , where σ : R N × C → R N × C is an element-wise ReLU function (Krizhevsky et al., 2012) deﬁned by σ ( X ) nc := max( X nc , 0) for n ∈ [ N ] , c ∈
[ C ] .
We consider the dynamics X ( l +1) :
= f l ( X ( l ) ) with some initial value
X (0) ∈ R N × C .
We are interested in the asymptotic behavior of X ( l ) as l → ∞ .
For M ≤ N , let U be a M -dimensional subspace of R N .
We assume that U and P satisfy the following properties that generalize the situation where U is the eigenspace associated with the smallest eigenvalue of a (normalized) graph Laplacian ∆ (that is, zero) and P is a polynomial of ∆ .
Assumption 1.
U has an orthonormal basis ( e m ) m ∈
[ M ] that consists of non-negative vectors.
Assumption 2.
U is invariant under P , i.e., if u ∈ U , then Pu ∈ U .
We endow R N with the ordinal inner product and denote the orthogonal complement of U by U ⊥ :
= { u ∈ R N | (cid:104) u, v (cid:105) = 0 , ∀ v ∈ U } .
By the symmetry of P , we can show that U ⊥ is invariant under
P , too (Appendix E.1, Proposition 2).
Therefore, we can regard P as a linear mapping P | U ⊥ :
U ⊥ → U ⊥ .
We denote the operator norm of P | U ⊥ by λ .
When U is the eigenspace associated with the smallest eigenvalue of ∆ and P is g (∆) where g is a polynomial, then, λ corresponds to λ =
sup µ | g ( µ ) | where sup ranges over all eigenvalues except the smallest one.
3. M AIN R ESULT
We deﬁne the subspace M of R N × C by M :=
U ⊗ R C
=
{ (cid:80) Mm =1 e m ⊗ w m | w m ∈ R C } where ( e m ) m ∈
[ M ] is the orthonormal basis of U appeared in Assumption 1.
For X ∈ R N × C , we denote the distance between X and M by d M ( X ) :
= inf {(cid:107) X − Y (cid:107) F | Y ∈ M} .
We denote the maximum singular value of W lh by s lh and set s l := (cid:81)
H l h =1 s lh .
With these preparations, we introduce the main theorem of the paper.
Theorem 1.
Under Assumptions 1 and 2, we have d M ( f l ( X )) ≤ s l λd M ( X ) for any X ∈ R N × C .
The proof key is that the non-linear operation σ decreases the distance d M , that is, d M ( σ ( X ))
≤ d M ( X ) .
We use the non-negativity of e m to prove this claim.
See Appendix A for the complete proof.
We also discuss the strictness of Theorem 1 in Appendix E.3.
By setting d M ( X ) = 0 , this theorem implies that M is invariant under f l .
In addition, if the maximum value of singular values are small, X ( l ) asymptotically approaches M in the sense of
Johnson (1973) for any initial value X (0) .
That is, the followings hold under Assumptions 1 and 2.
Corollary 1.
M is invariant under f l for any l ∈ N + , that is, if X ∈ M , then we have f l ( X ) ∈ M .
Corollary 2.
Let s :=
sup l ∈ N + s l .
We have d M ( X ( l ) )
=
O (( sλ ) l ) .
In particular, if sλ < 1 , then
X l exponentially approaches M as l → ∞ for any initial value X (0) .
Suppose the operator norm of P | U : U → U is no larger than λ , then, under the assumption of sλ < 1 , X ( l ) converges to 0 , the trivial ﬁxed point (see Appendix E.2, Proposition 3).
Therefore, we are mainly interested in the case where the operator norm of P | U is strictly larger than λ (see
Proposition 1).
Finally, we restate Theorem 1 specialized to the situation where U is the direct sum of eigenspaces associated with the largest M eigenvalues of P .
Note that the eigenvalues of P is real since P is symmetric.
Corollary 3.
Let λ 1 ≤ · · · ≤ λ N be the eigenvalue of P , sorted in ascending order.
Suppose the multiplicity of the largest eigenvalue
λ
N is M ( ≤ N ) , i.e., λ N − M < λ N − M +
= · · · = λ N .
We deﬁne λ := max n ∈ [ N − M ] | λ n | .
We denote U by the eigenspace associated with λ N and assume that
U satisﬁes Assumption 1.
Then, we have d M ( X ( l +1) )
≤ s l λd M ( X ( l ) ) .
Remark 1.
It is known that any Markov process on ﬁnite states converges to a unique distribution (equilibrium) if it is irreducible and aperiodic (see e.g., Norris (1998)).
Theorem 1 includes this proposition as a special case with M = 1 , C = 1 , and W l = 1 for all l ∈ N + .
This is essentially the direct consequence of Perron – Frobenius’ theorem (see e.g., Meyer (2000)).
See Appendix F.
A PPLICATION TO GCN
We formulate a GCN (Kipf & Welling, 2017) without readout operations (Gilmer et al., 2017) using the dynamical system in the previous section and derive a sufﬁcient condition in terms of the spectra of underlying graphs in which layer stacking nor non-linearity are not helpful for node classiﬁcation.
Let G =
( V, E ) be an undirected graph where V is a set of nodes and E is a set of edges.
We denote the number of nodes in G by N = | V | and identify V with [ N ] by ﬁxing an order of V .
We associate a C dimensional signal to each node.
X in the previous section corresponds to concatenation of the signals.
GCNs iteratively update signals on V using the connection information and weights.
Let A :=
( 1 { ( i,j ) ∈ E } ) i,j ∈ [ N ] ∈ R N × N be the adjacency matrix and D := diag(deg( i )
i ∈
[ N ] ) ∈
R N × N be the degree matrix of G where deg( i ) :
= |{ j ∈ V
| ( i, j ) ∈ E }| is the degree of node i .
Let ˜ A :
=
A + I N , ˜ D :
= D + I N be the adjacent and degree matrix of graph G augmented with self-loops.
We deﬁne the augmented normalized Laplacian (Wu et al., 2019a) of G by ˜∆ :=
I N − ˜ D − 12 ˜
A ˜ D − 12 and set P :
= I N − ˜∆ .
Let L, C ∈ N + be the layer and channel sizes, respectively.
For weights W l ∈ R C × C ( l ∈ [ L ] ), we deﬁne a GCN 2 associated with G by f = f L ◦ · · · ◦ f 1 where f l : R N × C → R N × C is deﬁned by f l ( X ) :
= σ ( PXW l ) .
We are interested in the asymptotic behavior of the output
X ( L ) of the GCN as L → ∞ .
Suppose G has M connected components and let V
= V 1 (cid:116) · · · (cid:116) V M be the decomposition of the node set V into connected components.
We denote an indicator vector of the m -th connected component by u m :=
( 1 { n ∈ V m } ) n ∈ [ N ] ∈ R N .
The following proposition shows that GCN satisﬁes the assumption of Corollay 3 (see Appendix B for proof).
Proposition 1.
Let λ 1 ≤ · · · ≤ λ N be the eigenvalue of P sorted in ascending order.
Then, we have − 1 < λ 1 , λ N − M < 1 , and λ N − M +1 = · · · = λ N = 1 .
In particular, we have λ := max n =1 ,...,N − M | λ n | < 1 .
Further, e m := ˜
D 12 u m for m ∈
[ M ] are the basis of the eigenspace associated with the eigenvalue 1 .
Theorem 2.
For any initial value X (0) , the output of l -th layer X ( l ) satisﬁes d M ( X ( l ) ) ≤ ( sλ ) l d M ( X (0) ) .
In particular, d M ( X ( l ) ) exponentially converges to 0 when sλ < 1 .
In the context of node classiﬁcation tasks, we can interpret this corollary as the “information loss” of
GCNs in the limit of inﬁnite layers.
For any X ∈ M , if two nodes i, j ∈ V are in a same connected component and their degrees are identical, then, the column vectors of X that correspond to nodes i and j are identical.
It means that we cannot distinguish these nodes using X .
In this sense, M only has information about connected components and node degrees and we can interpret this theorem as the exponential information loss of GCNs in terms of the layer size.
Similarly to the discussion in the previous section, X ( l ) converges to the trivial ﬁxed point 0 when s < 1 (remember λ N = 1 ).
An interesting point is that even if s ≥ 1 , X ( l ) can suffer from this information loss when s < λ − 1 .
We note that the rate sλ in Theorem 2 depends on the spectra of the augmented normalized Laplacian, which is determined by the topology of the underlying graph G .
Hence, our result explicitly relates the topological information of graphs and asymptotic behaviors of graph NNs.
Remark 2.
The old preprint (version 2) of Luan et al.
(2019) formulated a theorem that explains the over-smoothing of non-linear GNNs.
Speciﬁcally, it claimed that if a graph does not have a bipartite component and the input distribution is continuous, the rank of the output of a GCN converges to the number of connected components of the underlying graph as the layer size goes to inﬁnity almost surely.
However, it is not true in general as we give a counterexample in Appendix C.
A SYMPTOTIC B EHAVIOR OF GCN ON E RD ˝ OS – R ´ ENYI G RAPH
Theorem 2 gives us a way to characterize the asymptotic behaviors of GCNs via the spectral distributions of the underlying graphs.
To demonstrate this, we consider an Erd˝os – R´enyi graph G N,p (Erd¨os & R´enyi, 1959; Gilbert, 1959), which is a random graph that has N nodes and whose edges between two distinct nodes appear independently with probability p ∈
[0 , 1] , as an example.
First, consider a (non-random) graph G with M connected components.
Let 0 = ˜ µ 1 = · · · = ˜ µ M < ˜ µ M +1 ≤ · · · ≤ ˜ µ N < 2 be eigenvalues of the augmented normalized Laplacian of G (see, Proposition 1) and set λ :=
min m =
M +1 ,...,N | 1 − ˜ µ m | ( < 1) .
By Theorem 2, the output of GCN “loses information” as the layer size goes to inﬁnity when the largest singular values of weights are strictly smaller than λ − 1 .
Therefore, the closer the positive eigenvalues µ m are to 1 , the broader range of
GCNs satisﬁes the assumption of Theorem 2.
[Figure]
For an Erd ˝ os – R ´ enyi graph G N,p , Chung & Radcliffe (2011) showed that when log N Np =
o (1) ,
the eigenvalues of the (usual) normalized Laplacian except for the smallest one converge to 1 with high probability (see Theorem 2 therein) 3 .
We can interpret this theorem as the convergence of Erd ˝ osR ´ enyi graphs to the complete graph in terms of graph spectra.
We can prove that the augmented normalized Laplacian behaves similarly (Lemma 6).
By combining this fact with the discussion in the previous paragraph, we obtain the asymptotic behavior of GCNs on the Erd ˝ os – R ´ enyi graph.
See Appendix D for the complete proof.
Theorem 3.
Consider a GCN on the Erd˝os-R´enyi graph G N,p such that log N Np
= o (1) as a function of N .
For any ε > 0 , if the supremum s of the maximum singular values of weights in the GCN satisﬁes s < s 0 :
= 17 (cid:113)
Np − p +1 log(4 N/ε ) , then, for sufﬁciently large N , the GCN satisﬁes the condition of
Theorem 2 with probability at least 1 − ε .
Theorem 3 requires that an underlying graph is not extremely sparse.
For example, suppose the node size is N = 20 , 000 , which is the approximately the maximum node size of datasets we use in experiments, and the edge probability is p
= log N/N .
Then, each node has the order of Np ≈ 4 .adjacent nodes.
Under the condition of Theorem 3, the upper bound s 0 → ∞ as N → ∞ .
It means that if the graph is sufﬁciently large and not extremely sparse, most GCNs suffer from the information loss.
For the dependence on the edge probability p , s 0 is an increasing function of p , which means the denser a graph is, the more quickly graph convolution operations mix signals on nodes and move them close to each other.
Theorem 3 implies that graph NNs perform poorly on dense NNs.
More aggressively, we can hypothesize that the sparsity of practically available graphs is one of the reasons for the success of graph NNs in node classiﬁcation tasks.
To conﬁrm this hypothesis, we artiﬁcially add edges to citation networks to make them dense in the experiments and observe the failure of graph NNs as expected (see Section 6.3).
E XPERIMENT 6. S YNTHESIS D ATA : O NE - STEP T RANSITION
We numerically investigate how the transition f
( X ) :
= σ ( PXW ) changes inputs using the vector ﬁeld V ( X ) :
= f ( X ) − X 4 .
For this purpose, we set N = 2 , M = 1 , and C = 1 .
Let λ 1 ≤ λbe the eigenvalues of P .
We choose W as | λ 2 | − 1 ≤ W < | λ 1 | − 1 so that Theorem 1 is applicable but is not reduced to the trivial situation (see, Appendix E.2).
We choose the eigenvector e ∈ R
[Figure] [Figure]
associated with λ 2 in two ways as described below.
See Appendix H.1 for the concrete values of P , e
[Figure]
and W .
Figure 1 shows the visualization of V .
First, we choose the non-negative eigenvector e so that it satisﬁes Assumption 1 ( Case 1 ).
We see that the transition function f uniformly decreases the distance from M .
This is consistent with the consequence of Theorem 1.
Next, we choose the eigenvector e =
[ ee 2 ] (cid:62) such that the signs of e 1 and e 2 differ ( Case 2 ), which violates
Assumption 1.
We see that M is not invariant under f and f does not uniformly decrease the distance from M .
Therefore, we cannot remove the non-negativity assumption from Theorem 1.
6. S YNTHESIS D ATA : D ISTANCE TO I NVARIANT S PACE
We evaluate the distance to the invariant space M using synthesis data.
We randomly generate an
Erd˝os – R´enyi graph, a GCN on it, and an input signal X (0) .
We compute the distance between the lth intermediate output X ( l ) and the invariant space M for various edge probability p and maximum singular value s .
Figure 2 plots the logarithm of the relative distance y ( l ) =
log d M ( X ( l ) )
d M ( X (0) ) with respect to the layer index l .
From Theorem 1, we know that it is upper bounded by y ( l ) =
l log( sλ ) .
We see that this bound well approximates the actual value when sλ is small.
On the other hand, it is loose for large sλ .
We leave tighter bounds for d M in such a case for future research.
6. R EAL D ATA : E FFECT OF M AXIMUM S INGULAR V ALUES ON P ERFORMANCE
Theorem 2 implies that if s is smaller than the threshold λ − 1 , we cannot expect deep GCN to achieve good prediction accuracy.
Conversely, if we can successfully train the model, s should avoid the region s ≤ λ − 1 .
We empirically conﬁrm these hypotheses using real datasets.
We use Cora, CiteSeer, and PubMed (Sen et al., 2008), which are standard citation network datasets.
The task is to classify the genre of papers using word occurrences and citation relationships.
We regard each paper as a node and citation relationship as an edge.
Due to space constraints, we focus on Cora in the main article.
See Appendix H.3 and I.3 for the other datasets.
The discussion in
Section 5 implies that Theorem 2 can support a wide range of GCNs when the underlying graph is relatively dense.
However, the citation networks are too sparse to examine the aforementioned hypotheses — Theorem 2 gives a non-trivial result only when 1 ≤ s < λ − 1 ≈ 1 + 3 .
62 × 10 − 3 .
To circumvent this, we make noisy versions of citation networks by randomly adding edges to graphs.
Through this manipulation, we can increase the value of λ − 1 to 1 . 11 .
Figure 3 (left) shows the accuracy for the test dataset in terms of the maximum singular values and the number of graph convolution layers.
We can observe that when GCNs whose maximum singular value s is out of the region s < λ − 1 outperform those inside the region in almost all conﬁgurations.
Furthermore, the accuracy of GCNs with s = 10 are better than those without normalization ( unnormalized ).
Figure 3 (right) shows the transition of the maximum singular values of the weights during training when we use a three-layered GCN.
We can observe that the maximum singular value s does not shrink to the region s ≤ λ − 1 .
In addition, when the layer size is small and predictive accuracy is high, GCNs gradually increase s from the initial value and avoid the region.
In conclusion, the experiment results are consistent with the theorems.
6. R EAL D ATA : E FFECT OF S IGNAL C OMPONENT P ERPENDICULAR T O I NVARIANT S PACE
[Figure]
We can decompose the output X of a model as X =
X
+ X
( X 0 ∈ M , X 1 ∈ M ⊥ ).
According to the theory, X 0 has limited information for node classiﬁcation.
We hypothesize that the model emphasizes the perpendicular component Xto perform good predictions.
To quantitatively evaluate it, we deﬁne the relative magnitude of the perpendicular component of the output X by t ( X ) :
= X 1 /X 0 .
Figure 4 compares this quantity and the prediction accuracy on the noisy version of
Cora (see Appendix I.4 for other datasets).
We observe that these two quantities are correlated ( R = 0 . 545 ).
If we remove
GCNs have only one layer (corresponding to right points in the ﬁgure), the correlation coefﬁcient is 0 .
This result does not contradict to the hypothesis above 5 .
D ISCUSSION
Applicability to Graph NNs on Sparse Graphs.
We have theoretically and empirically shown that when the underlying graph is sufﬁciently dense and large, the threshold λ − 1 is large (Theoremand Section 6.3), which means many graph CNNs are eligible.
However, real-world graphs are not often dense, which means that Theorem 2 is applicable to very limited GCNs.
In addition, CojaOghlan (2007) theoretically proved that if the expected average degree of G N,p is bounded, the smallest positive eigenvalue of the normalized Laplacian of G N,p is o (1) with high probability.
The asymptotic behaviors of graph NNs on sparse graphs are left for future research.
Remedy for Over-smoothing.
Based on our theory, we can propose several techniques for mitigating the over-smoothing phenomena.
One idea is to (randomly) sample edges in an underlying graph.
The sparsity of practically available graphs could be a factor in the success of graph NNs.
Assuming this hypothesis is correct, there is a possibility that we can relive the effect of over-smoothing by sparsiﬁcation.
Since we can never restore the information in pruned edges if we remove them permanently, random edge sampling could work better as FastGCN (Chen et al., 2018a) and GraphSAGE (Hamilton et al., 2017) do.
Another idea is to scale node representations (i.e., intermediate or ﬁnal output of graph NNs) appropriately so that they keep away from the invariant space M .
Our proposed weight scaling mechanism takes this strategy.
Recently, Zhao & Akoglu (2020) has pro- posed PairNorm to alleviate the over-smoothing phenomena.
Although the scaling target is different – they rescaled signals whereas we normalized weights – theirs and ours have similar spirits.
Graph NNs with Large Weights.
Our theory suggests that the maximum singular values of weights in a GCN should not be smaller than a threshold λ − 1 because it suffers from information loss for node classiﬁcation.
On the other hand, if the scale of weights are very large, the model complexity of the function class represented by graph NNs increases, which may cause large generalization errors.
Therefore, from a statistical learning theory perspective, we conjecture that the graph NNs with too-large weights perform poorly, too.
A trade-off should exist between the expressive power and model complexity and there should be a “sweet spot” on the weight scale that balances the two.
Relation to Double Descent Phenomena.
Belkin et al.
(2019) pointed out that modern deep models often have double descent risk curves: when a model is under-parameterized, a classical biasvariance trade-off occurs.
However, once the model has a large capacity and perfectly ﬁts the training data, the test error decreases as we increase the number of parameters.
To the best of our knowledge, no literature reported the double descent phenomena for graph NNs (it is consistent with the picture of the classical U-shaped risk curve in the previous paragraph).
It is known that double descent phenomena do not occur in some situations, especially depending on regularization types.
For example, while Belkin et al.
(2019) employed the interpolating hypothesis with the minimum norm, Mei & Montanari (2019) found that the double descent was alleviated or disappeared when they used Ridge-type regularization techniques.
Therefore, one can hypothesize the over-smoothing is a cause or consequence of regularization that is more like a Ridge-type rather than minimum-norm inductive bias.
Limitations in Graph NN Architectures.
Our analysis is limited to graph NNs with the ReLU activation function because we implicitly use the property that ReLU is a projection onto the cone { X ≥ 0 } (Appendix A, Lemma 3).
This fact enables the ReLU function to get along with the nonnegativity of eigenvectors associated with the largest eigenvalues.
Therefore, it is far from trivial to extend our results to other activation functions such as the sigmoid function or Leaky ReLU (Maas et al., 2013).
Another point is that our formulation considers the update operation (Gilmer et al., 2017) of graph NNs only and does not take readout operations into account.
In particular, we cannot directly apply our theory to graph classiﬁcation tasks in which each sample is a graph.
Over-smoothing of Residual GNNs.
Considering the correspondence of graph NNs and Markov processes (see Appendix F), one can imagine that residual links do not contribute to alleviating the over-smoothing phenomena because adding residual connections to a graph NN corresponds to converting a Markov process to its lazy version.
When a Markov process converges to a stable distribution, the corresponding lazy process also converges eventually under certain conditions.
It implies that residual links might not be helpful.
However, Li et al.
(2019) reported that graph NNs with as many as 56 layers performed well if they added residual connections.
Considering that, the situation could be more complicated than our intuitions.
The analysis of the role of residual connections in graph NNs is a promising direction for future research.
C ONCLUSION
In this paper, to understand the empirically observed phenomena that deep non-linear graph NNs do not perform well, we analyzed their asymptotic behaviors by interpreting them as a dynamical system that includes GCN and Markov process as special cases.
We gave theoretical conditions under which GCNs suffer from the information loss in the limit of inﬁnite layers.
Our theory directly related the expressive power of graph NNs and topological information of the underlying graphs via spectra of the Laplacian.
It enabled us to leverage spectral and random graph theory to analyze the expressive power of graph NNs.
To demonstrate this, we considered GCN on the Erd˝os – R´enyi graph as an example and showed that when the underlying graph is sufﬁciently dense and large, a wide range of GCNs on the graph suffer from information loss.
Based on the theory, we gave a principled guideline for how to determine the scale of weights of graph NNs and empirically showed that the weight normalization implied by our theory performed well in real datasets.
One promising direction of research is to analyze the optimization and statistical properties such as the generalization power (Verma & Zhang, 2019) of graph NNs via spectral and random graph theories.
A CKNOWLEDGMENTS
We thank Katsuhiko Ishiguro for providing a part of code for the experiments, Kohei Hayashi and
Haru Negami Oono for giving us feedback and comments on the draft, Keyulu Xu and anonymous reviewers for fruitful discussions via OpenReview and Ryuta Osawa for pointing out errors and suggesting improvements of the paper.
TS was partially supported by JSPS KAKENHI (15H05707, 18K19793, and 18H03201), Japan Digital Design, and JST CREST.
A P ROOF OF T HEOREM
As we wrote in the main article it is enough to show the following lemmas (deﬁnition of miscellaneous variables are as in Section 3.2).
Remember that λ
= sup n ∈ [ N − M ] | λ n |
and s lh is the maximum singular value of W lh
Lemma 1.
For any X ∈ R N × C , we have d M ( PX ) ≤ λd M ( X ) .
Lemma 2.
For any X ∈ R N × C , we have d M ( XW lh ) ≤ s lh d M ( X ) .
Lemma 3.
For any X ∈ R N × C , we have d M ( σ ( X ))
≤ d M ( X ) .
Proof of Lemma 1.
Since P is a symmetric linear operator on U ⊥ we can choose the orthonormal basis ( e m ) m
=
M +1 ,...,N of U ⊥ consisting of the eigenvalue of P | U ⊥ .
Let λ m be the eigenvalue of P to which e m is associated ( m = M + 1 , . . .
, N ).
Note that since the operator norm of P | U ⊥ is λ , we have | λ m | ≤ λ for all m =
M + 1 , . .
N .
Since ( e m ) m ∈ [ N ] forms the orthonormal basis of
R N , we can uniquely write X ∈ R N × C as X = (cid:80) Nm =1 e m ⊗ w m for some w m ∈ R C .
Then, we have d 2 M ( X ) =
(cid:80) Nm
= M +1 (cid:107) w m (cid:107) 2 where (cid:107)·(cid:107) is the 2-norm of a vector.
On the other hand, we have
PX
[Equation]
(cid:88)
[Equation]
Since U is invariant under P , for any m ∈ [ M ] , we can write Pe m as a linear combination of e n ( n ∈ [ M ]) .
Therefore, we have d 2 M ( PX ) =
(cid:80) Nm = M +1 (cid:107) λ m
[Equation]
m (cid:107) 2 .
Then, we obtain the desired inequality as follows:
[Equation]
Proof of Lemma 2.
Using the same decomposition of X as the proof in Lemma 1, we have
[Equation]
e m ⊗ ( W
[Equation]
lh w m ) .
Therefore, we have
[Equation]
Proof of Lemma 3.
We choose ( e m ) m
= N − M +1 ,...,N as in the proof of Lemma 1.
We denote X =
( X nc ) n ∈ [ N ] ,c ∈ [ C ] and e n = ( e mn ) m ∈
[ N ] , respectively.
Let ( e (cid:48) c )
c ∈ [ C ] be the standard basis of R C .
Then, ( e n ⊗ e (cid:48) c ) n ∈ [ N ] ,c ∈ [ C ] is the orthonormal basis of R N × C , endowed with the standard inner product as a Euclid space.
Therefore, we can decompose X as X = (cid:80)
Nn =1 (cid:80)
Cc =1 a nc e n ⊗ e (cid:48) c where a nc = (cid:104) X, e n ⊗ e (cid:48)
c (cid:105) =
(cid:80) Nm
=
X mc e mn .
Then, we have d 2 M ( X ) =
(cid:80) Nn
= M +1 (cid:107) (cid:80)
Cc =1 a nc e (cid:48)
c (cid:107) 2 , which is further transformed as
[Equation]
where X · c is the c -th column vector of X .
Similarly, we have
[Equation]
where we denote σ ( X ) =
( X + nc ) n ∈
[ N ] ,c ∈ [ C ] in shorthand.
Therefore, the inequality follow from the following lemma.
Lemma 4.
Let x ∈ R N and v 1 , . .
. , v M ∈ R N be orthonormal vectors (i.e., (cid:104) v m , v n (cid:105) =
δ mn ) satisfying v m ≥ 0 for all m ∈
[ M ] .
Then, we have (cid:107) x (cid:107) 2 − (cid:80) Mm =1 (cid:104)
[Equation]
where x + := max( x, 0) for x ∈ R .
Proof.
The value (cid:107) y (cid:107) 2 − (cid:80) Mm =1 (cid:104) y, u m (cid:105) 2 is invariant under simultaneous coordinate permutation of y and u m ’s.
Therefore, we can assume without loss of generality that the coordinate of x are sorted: x 1 ≤ . . .
≤ x L < 0 ≤ x L +1 ≤ · · · ≤ x N for some L ≤ N .
Then, we have
[Equation]
When L = 0 , the sum in the right hand side is treated as 0 .
On the other hand, writing as v m = ( v nm ) n ∈ [ N ] , direct calculation shows
[Equation]
Let I m :=
{ n ∈ [ N ] | v nm > 0 } be the support of v m for m ∈
[ M ] .
We note that if m (cid:54) = m (cid:48) ∈
[ M ] , we have I m ∩
I m (cid:48)
= ∅ since if there existed n ∈ I m ∩ I m (cid:48) , we have
[Equation]
≥ v nm v nm (cid:48) > 0 , which is contradictory.
Therefore,
N
[Equation]
Schwarz inequality
[Equation]
We used the fact that I m ’s are disjoint and v nm = 0 if n (cid:54)∈ ∪ m I m in the ﬁrst equality above.
Further, we have x n x l v nm v lm ≤ 0 for 1 ≤ n ≤ L and L + 1 ≤ l ≤ N by the deﬁnition of L and non-negativity of v m .
By combining (1), (2), and (3), we have
[Equation]
Proof of Theorem 1.
By Lemma 1, 2, and 3, we have
[Equation]
B P ROOF OF P ROPOSITION
Proof.
Let ˜ µ 1 ≤ · · · ≤ ˜ µ N be the eigenvalue of the augmented normalized Laplacian ˜∆ , sorted in ascending order.
Since P = I N − ˜∆ , it is enough to show ˜ µ
1 = · · · = ˜ µ M = 0 , ˜ µ M +1 > 0 , and ˜ µ N < 2 .
For the ﬁrst two, the statements are equivalent to that ˜∆ is positive semi-deﬁnite and that the multiplicity of the eigenvalue 0 is same as the number of connected components 6 .
This is well-known for Laplacian or its normalized version (see, e.g., Chung & Graham (1997)) and the proof for ˜∆ is similar.
By direct calculation, we have
[Equation]
for any x = [ x· · · x N ] (cid:62) ∈ R N .
Therefore, ˜∆ is positive semi-deﬁnite and hence ˜ µ 1 ≥ 0 .
Suppose temporally that G is connected.
If x ∈ R N is an eigenvector associated to 0 , then, by the aformentioned calculation, x i
[Equation]
d i +1 and x j √ d j +1 must be same for all pairs ( i, j ) such that a ij > 0 .
However, since G is connected, x i √ d i +1 must be same value for all i ∈
[ N ] .
That means the multiplicity of the eigenvalue 0 is 1 and any eigenvector associated to 0 must be proportional to ˜
D 12 1 .
Now, suppose G has M connected components V 1 , . .
. , V M .
Let ˜∆ m be the augmented normalized Laplacians corresponding to each connected component V m for m ∈
[ M ] .
By the aformentioned discussion, ˜∆ m has the eigenvalue 0 with multiplicity 1 .
Since ˜∆ is the direct sum of ˜∆ (cid:48) m s , the eigenvalue of ˜∆ is the union of those for ˜∆ m
’s. Therefore, ˜∆ has the eigenvalue 0 with multiplicity M and e m = ˜
D 12 1 m ’s are the orthogonal basis of the eigenspace.
Finally, we prove ˜ µ N < 2 .
Let µ N be the largest eigenvalue of the normalized Laplacian ∆ =
D − 12 ( D − A ) D − 12 , where D − 12 ∈ R N × N is the diagonal matrix deﬁned by
[Equation]
if
[Equation]
Note that D − 12 D 12 nor D 12 D − 12 are not equal to the identity matrix I N in general.
However, we have
[Equation]
where L = D − A is the (unnormalized) Laplacian.
Therefore, we have
[Equation]
D 12 ˜ D − 12 x (cid:107) (cid:107) x (cid:107) ≤ max x (cid:54) =
D 12 ˜ D − 12 x (cid:54) =
( D 12 ˜ D − 12 x ) (cid:62) ∆( D 12 ˜ D − 12 x ) (cid:107) D 12 ˜ D − 12 x (cid:107) max x (cid:54) =
D 12 ˜ D − 12 x (cid:54)
=
(cid:107) D 12 ˜ D − 12 x (cid:107) (cid:107) x (cid:107)
[Equation]
[Equation]
≤ µ N . Therefore, we have ˜ µ N ≤ µ N 7 .
Since max i ∈ [ N ] (cid:16) deg( i ) deg( i )+1 (cid:17) 12 < 1 , the equality ˜ µ N = µ N holds if and only
if µ N = 0 , that is, G has N connected components.
On the other hand, it is known that µ N ≤ 2 and the equality holds if and only if G has non-trivial bipartite graph as a connected component (see, e.g., Chung & Graham (1997)).
Therefore, ˜ µ N = µ N and µ N = 2 does not hold simultaneously and we obtain µ N < 2 .
C C OUNTEREXAMPLE OF P REVIOUS S TUDY ON O VER - SMOOTHING FOR N ON - LINEAR GNN S
We restate Theorem 1 of the preprint (version2) of Luan et al.
Let G be a simple undirected graph with N nodes and k connected components such that it does not have a bipartite component.
Let L =
˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 ∈ R N × N be the augmented normalized Laplacian of G .
Let F ∈ N + and W n ∈ R F × F be the weight of the n -th layer for n ∈ N + .
For the input X ∈ R N × F , we deﬁne the output Y n ∈ R N × F of the n -th layer of a GCN by Y n = σ ( L · · · σ ( LXW 0 ) · · · W n ) where σ is the ReLU function.
We assume the input X is drawn from a continuous distribution on R N × F .
Then, the theorem claims that we have lim n →∞ rank( Y n ) =
k almost surely with respect to the distribution of X .
We construct a conterexample.
Consider a graph G consisting of N = 4 nodes whose adjacency matrix is
[Equation]
[-  1 1 11 1 11 1 11 
-]
Note that G is connected (i.e., k = 1 ) and is not bipartite.
We make a GCN with F = 3 channels and whose weight matrices are W n = I 3 (the identity matrix of size 3 ) for all n ∈ N .
For the distribution of the input X , we consider an absolutely continuous distribution with respect to the
Lebesgue measure on R 4 × 3 such that P ( X ≥ 0) >
(here, X ≥ 0 means the element-wise comparison).
For example, the standard Gaussian distribution satisﬁes the condition.
Since L ≥ 0 , we have Y n =
L n X if X ≥ 0 .
Let L = P (cid:62) Λ P be the diagonalization of L where
P ∈ O (4) is an orthogonal matrix of size 4 .
Since rank( L ) = 3 , we have rank(Λ n ) = 3 for any n (we can assume that Λ 44 = 0 without loss of generality).
Therefore, under the condition X ≥ 0 , we have
[Equation]
Note that the last condition is independent of n .
Since the set of invertible matrices is dense in the set of all matrices of the same size (with respect to the standard topology of the Euclidean space), we have P ( { rank( Y n ) =
3 for all n ∈ N } ) > 0 .
Therefore, we have lim n →∞ rank
( Y n ) = 3 with a non-zero probability.
D P ROOF OF T HEOREM
We follow the proof of Theorem 2 of Chung & Radcliffe (2011).
The idea is to relate the spectral distribution of the normalized Laplacian with that of its expected version.
Since we can compute the latter one explicitly for the Erd ˝ os-R ´ enyi graph, we can derive the convergence of spectra.
We employ this technique and derive similar conclusion for the augmented normalized Laplacian.
First, we consider genral random graphs not restricted to Erd ˝ os-R ´ enyi graphs.
Let N ∈ N + , and
P = ( p ij ) i,j ∈ [ N ] be a non-negative symmetric matrix (meaning that p ij ≥ 0 for any i, j ∈ [ N ] ).
Let G be an undirected random graph with N nodes such that an edge between i and j is independently present with probability p ij .
Let A and D be the adjacency and the degree matrices of
G , respectively (that is, A ij ∼ Ber
( p ij ) , i .
i .
d . ).
Deﬁne the expected node degree of node i by t i :=
(cid:80) Nj =1 p ij .
Let ˜ A :
=
A + I N , ˜ D :
= D + I N and deﬁne ¯ A := E
[ ˜ A ] = P + I N and ¯ D := E [ ˜ D ] = diag( t 1 , . . .
, t N )
+ I N correspondingly.
We deﬁne the augmented normalized Laplacian ˜∆ of G by ˜∆ := I N − ˜ D − 12 ˜
A ˜ D − 12 and its expected version by ¯∆ :
=
I N − ¯ D − 12 ¯
A ¯ D − 12.
For a symmetric matrix X ∈ R N , we deﬁne its eigenvalues, sorted in ascending order by λ 1 ( X ) ≤ · · · ≤ λ N ( X ) and its operator norm by (cid:107) X (cid:107) =
max n ∈ [ N ] | λ n ( X ) | .
Lemma 5 (Ref. Chung & Radcliffe (2011) Theorem 2) .
Let δ :=
min n ∈ [ N ] t n be the minimum expected degree of G .
Set k ( ε ) :
= 3(1 + log(4 /ε )) .
Then, for any ε > 0 , if δ + 1 > k ( ε ) log N , we have
[Equation]
with probability at least 1 − ε .
Proof.
By Weyl’s theorem, we have max n ∈ [ N ] (cid:12)(cid:12)(cid:12) λ n ( ˜∆) − λ n ( ¯∆) (cid:12)(cid:12)(cid:12) ≤ (cid:107) ˜∆ − ¯∆ (cid:107) .
Therefore, it is enough to bound (cid:107) ˜∆ − ¯∆ (cid:107) .
Let C :
=
I N − ¯ D − 12 ˜
A ¯ D .
By the triangular inequality, we have (cid:107) ˜∆ − ¯∆ (cid:107) ≤ (cid:107) ˜∆ − C (cid:107) + (cid:107) C − ¯∆ (cid:107) .
We will bound these terms respectively.
First, we bound (cid:107) C − ¯∆ (cid:107) .
Direct calculation shows C − ¯∆ = − ¯ D − 12 ( A − P ) ¯ D − 12 .
Let E ij ∈ R N × N be a matrix deﬁned by ( E ij
[Equation]
kl = (cid:26) [-
-]1 if ( i = k
and i = l ) or ( i = l and j = k ), 0 otherwise .
We deﬁne the random variable Y ij by
[Equation]
Then, Y ij ’s are independent and we have C − ¯∆
=
(cid:80) Ni,j =
Y ij .
To apply Theorem 5 of Chung &
Radcliffe (2011) to Y ij ’s, we bound (cid:107) Y ij − E
[ Y ij ] (cid:107) and (cid:107) (cid:80) N i,j =1 E [ Y 2 ij ] (cid:107) .
First, we have [Equation] (cid:107) ≤
[Equation]
Since
E [ Y 2 ij ] =
p ij − p 2 ij ( t i + 1)( t j + 1) (cid:26) E ii + E jj (if i (cid:54) =
j ),
E ii (if i = j ), we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i,j =
E [ Y 2 ij ] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i,j = p ij − p 2 ij ( t i + 1)( t j + 1) E ii (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = max i ∈ [ N ]  
N (cid:88) j = p ij − p 2 ij ( t i + 1)( t j + 1) 
 ≤ max i ∈ [ N ]   N (cid:88) j = p ij ( t i + 1)( t j + 1)   ≤ ( δ + 1) − 1 .
By letting a ← (cid:113) 3log(4 N/ε ) δ + , M ← ( δ + 1) − 1 , v 2 ← ( δ + 1) − 1 and applying Theorem 5 of Chung & Radcliffe (2011), we have
[Equation]
By the deﬁnition of k ( ε ) , we have a < 1 if δ + 1 > k ( ε ) log n .
For such δ , we have
[Equation]
Next, we bound (cid:107) ˜∆ − C (cid:107) .
First, since a < 1 , by Chernoff bound (see, e.g. Angluin & Valiant (1979); Hagerup & R¨ub (1990))), we have
[Equation]
Therefore, if | d i − t i | ≤ a ( t i + 1) , then we have
[Equation]
Therefore, by union bound, we have
[Equation]
with probability at least 1 − ε/ 2 .
Further, since the eigenvalue of the augmented normalized Laplacian is in [0 , 2] by the proof of Proposition 1, we have (cid:107) I N − ˜∆ (cid:107) ≤ 1 .
By combining them, we have
[Equation]
From (5) and (6), we have
[Equation]
with probability at least 1 − ε by union bound.
Let N ∈ N + and p > 0 .
In the case of the Erd˝os-R´enyi graph G N,p , we should set P = p ( J N − I N ) where J N ∈ R N × N are the all-one matrix.
Then, we have ¯ A = pJ N + (1 − p )
I N , ¯ D
=
( Np − p + 1) I N , and ¯∆ = p
Np − p
+
( NI N − J N ) .
Since the eigenvalue of J N is N (with multiplicity 1 ) and 0 (with multiplicity N − 1 ), the eigenvalue of ¯∆ is 0 (with multiplicity 1 ) and
Np Np − p
+1 (with multiplicity N − 1 ).
For G N,p , δ is the expected average degree ( N − 1) p .
Hence, we have the following lemma from Lemma 5:
Lemma 6.
Let ˜∆ be its augmented normalized Laplacian of the Erd˝os-R´enyi graph G N,p .
For any ε > 0 , if Np − p +1 log N > k ( ε ) :
= 3(1 + log(4 /ε )) , then, with probability at least 1 − ε , we have
[Equation]
Corollary 4.
Consider GCN on G N,p .
Let W l be the weight of the l -th layer of GCN and s l be the maximum singular value of W l for l ∈ N + .
Set s :=
sup l ∈ N + .
Let ε > 0 .
We deﬁne k ( ε ) := 3(1 + log(4 /ε )) and l ( N, p, ε ) =
1 − p Np − p +1 + 4 (cid:113) 3log(4 N/ε )
Np − p
+1 .
If Np − p +1 log N > k ( ε ) and s ≤ l ( N, ε ) − 1 , then, GCN on G N,p satisﬁes the assumption of Theorem 2 with probability at least 1 − ε .
Proof of Theorem 3.
Since log N Np = o (1) , for ﬁxed ε , we have
[Equation]
for sufﬁciently large N .
Further, Np → ∞ as N → ∞ when log N Np = o (1) .
Therefore, we have
[Equation]
for sufﬁciently large N .
Hence.
[Equation]
Therefore, we have l ( N, p, ε ) ≤ 7 (cid:113) log(4 N/ε )
Np − p
+1 .
Therefore, if s ≤ 17 (cid:113) Np − p +1 log(4 N/ε ) , then we have s ≤ l ( N, p, ε ) − 1 .
E M ISCELLANEOUS P ROPOSITIONS E. I NVARIANCE OF O RTHOGONAL C OMPLEMENT S PACE
Proposition 2.
Let P ∈ R N × N be a symmetric matrix, treated as a linear operator P : R N → R N .
If a subspace U ⊂ R N is invariant under P (i.e., if u ∈ U , then Pu ∈ U ), then, U ⊥ is invariant under P , too.
Proof.
For any u ∈ U ⊥ and v ∈ U , by symmetry of P , we have
[Equation]
Since U is an invariant space of P , we have Pv ∈ U .
Hence, we have (cid:104) u, Pv (cid:105) = 0 because u ∈ U ⊥ .
We obtain Pu ∈
U ⊥
by the deﬁnition of U ⊥ .
E.
C ONVERGENCE TO T RIVIAL F IXED P OINT
Let P ∈ R N × N be a symmetric matrix, W l ∈ R C × C , s l be the maximum singular value of W l for l ∈ N + .
We deﬁne f l : R N × C → R N × C by f l ( X ) :
= σ ( PXW l ) where σ is the element-wise
ReLU function.
Proposition 3.
Suppose further that the operator norm of P is no larger than λ , then we have (cid:107) f l ( X ) (cid:107) F ≤ s l λ (cid:107) X (cid:107) F for any l ∈ N + .
In particular, let s :=
sup l ∈ N + s l .
If sλ < 1 , then, X l exponentially approaches 0 as l → ∞ .
Proof.
Since λ is the operator norm of P | U ⊥ , the assumption implies that the operator norm of
P itself is no larger than λ .
Therefore, we have (cid:107) PXW l (cid:107) F ≤ λ (cid:107) XW l (cid:107) F ≤ s l λ (cid:107) X (cid:107) F .
On the other hand, since σ ( x ) 2 ≤ x 2 for any x ∈ R , we have (cid:107) σ ( X ) (cid:107) F ≤ (cid:107) X (cid:107) F for any X ∈ R N × C .
Combining the two, we have (cid:107) f l ( X ) (cid:107) F ≤ (cid:107) PXW l (cid:107) F ≤ s l λ (cid:107) X (cid:107) F .
E. S TRICTNESS OF T HEOREM
Theorem 1 implies that if sλ ≤ 1 , then, one-step transition f l does not increase the distance to M .
In this section, we ﬁrst prove that this theorem is strict in the sense that, there exists a situation in which s l λ > 1 holds and the distance d M increases by one-step transition f l at some point X .
Set N ← 2 , C ← 1 , and M ← 1 in Section 3.2.
For µ, λ > 0 , we set
P ← (cid:20) [Table]
Then, by deﬁnition, we can check that the 3-tuple ( P, e, U ) satisﬁes the Assumptions 1 and 2.
Set
M := U ⊗ R =
U
and choose W ∈ R so that W > λ − 1 .
Finally deﬁne f : R N × C → R N × C by f ( X ) :
= σ ( PXW ) where σ is the element-wise ReLU function.
Proposition 4.
We have d M ( f ( X )) >
d M ( X ) for any X
= [ xx 2 ] (cid:62) ∈ R 2 such that x 2 > 0 .
Proof.
By deﬁnition, we have d M ( X ) =
| x 2 | .
On the other hand, direct calculation shows that f l ( X ) = (cid:2) ( WµX 1 ) + ( WλX 2 ) + (cid:3) (cid:62) and d M ( f l ( X ))
= ( WλX 2 ) + where x + := max( x, 0) for x ∈ R .
Since W > λ − 1 and x 2 > 0 , we have d M ( f l ( X )) >
d M ( X ) .
Next, we prove the non-strictness of Theorem 1 in the sense that there exists a situation in which s l λ > 1 holds and the distance
d M uniformly decreases by f l .
Again, we set Set N ← 2 , C ← 1 , and M ← 1 .
Let λ ∈ (1 , 2) and set
[Equation]
Then, we can directly show that 3-tuple ( P, e, U ) satisﬁes the Assumptions 1 and 2.
Set W ← 1 .
Proposition 5.
We have Wλ > 1 and d M ( f l ( X )) <
d M ( X ) for all X ∈ R 2 .
Proof.
First, note that e (cid:48) :
= 1 √ 2 [1 − 1] (cid:62) is the eigenvector of P associated to λ :
Pe (cid:48)
=
λe (cid:48) .
For
X = ae
+ be (cid:48) ( a, b ¿ 0)
, the distance between X and M is d M ( X ) =
| b | .
On the other hand, by direct computation, we have
[Equation]
[Table]
[Equation]
Therefore, the distance between f ( X ) and M is d M ( f ( X ))
=
λ | b | / 2 .
Since λ < 2 , we have d M ( f ( X )) <
d M ( X ) for any X ∈ R 2 .
We have shown that the non-negativity of e (Assumption 1) is not a redundant condition in Section 6.1.
F R ELATION TO M ARKOV P ROCESS
It is known that any Markov process on ﬁnite states converges to a unique distribution ( equilibrium ) if it is irreducible and aperiodic (see, e.g., Norris (1998)).
As we see in this section, this theorem is the special case of Corollary 3.
Let S :
= { 1 , . . .
, N } be a ﬁnite discrete state space.
Consider a Markov process on S characterized by a symmetric transition matrix P =
( p ij )
i,j ∈ [ N ] ∈ R N × N such that P ≥ 0 and P 1 = 1 whereis the all-one vector.
We interpret p ij as the transition probability from a state i to j .
We associate
P with a graph G P =
( V P , E P ) by V P =
[ N ] and ( i, j ) ∈ E P
if and only if p ij > 0 .
Since P is symmetric, we can regard G P as an undirected graph.
We assume P is irreducible and aperiodic.
Perron – Frobenius’ theorem (see, e.g., Meyer (2000)) implies that P satisfy the assumption of
Corollary 3 with M = 1 .
Proposition 6 (Perron – Frobenius) .
Let the eigenvalues of P be λ 1 ≤ · · · ≤ λ N .
Then, we have − 1 < λ 1 , λ N − 1 < 1 , and λ N = 1 .
Further, there exists unique vector e ∈ R N such that e ≥ 0 , (cid:107) e (cid:107) = 1 , and e is the eigenvector for the eivenvalue 1 .
Corollary 5.
Let λ := max n =1 ,...,N − 1 | λ n | ( < 1) and M :=
{ e ⊗ w | w ∈ R C } .
If sλ < 1 , then, for any initial value X 1 , X l exponentially approaches M as l → ∞ .
If we set C = 1 and W l = 1 for all l ∈ N + , then, we can inductively show that X l ≥ 0 for any l ≥ 2 .
Therefore, we can interpret X l as a measure on S .
Suppose further that we take the initial value X 1 as X 1 ≥ 0 and X (cid:62) 1 1 = 1 so that we can interpret X 1 as a probability distribution on
S .
Then, we can inductively show that X l ≥ 0 , X (cid:62) l 1 = 1 (i.e., X l is a probability distribution on
S ), and X l +1 = σ ( PX l W l ) =
PX l for all l ∈ N + .
In conclusion, the corollary is reduced to the fact that if a ﬁnite and discrete Markov process is irreducible and aperiodic, any initial probability distribution converges exponentially to an equibrilium.
In addition, the the rate λ corresponds to the mixing time of the Markov process.
G GCN D EFINED B Y N ORMALIZED L APLACIAN
In Section 4, we deﬁned P using the augmented normalized Laplacian ˜∆ by P =
I N − ˜∆ .
We can alternatively use the usual normalized Laplacian ∆ instead of the augmented one to deﬁne P and want to apply the theory developed in Section 3.2.
We write the normalized Laplacian version as P ∆ := I N − ∆ .
The only obstacle is that the smallest eigenvalue λ 1 of P ∆ can be equal to − 1 , while that of P is strictly larger than − 1 (see, Proposition 1).
This corresponds to that fact the largest eigenvalue of ˜∆ is strictly smaller than 2 , while that for ∆ can be 2 .
It is known that the largest eigenvalue of ∆ is 2 if and only if the graph has a non-trivial bipartite connected component (see, e.g., Chung & Graham (1997)).
Therefore, we can develop a theory using the normalized
Laplacian instead of the augmented one in parallel for such a graph G .
In Section 5, we characterized the asymptotic behavior of GCN deﬁned by the augmented normalized Laplacian via its spectral distribution (Lemma 6 of Appendix D).
We can derive a similar claim for GCN deﬁned via the normalized Laplacian using the original theorem for the normalized Laplacian in Chung & Radcliffe (2011)
(Theorem 7 therein).
The normalized Laplacian version of GCN is advantegeous over the one made from the augmented one because we know its spectral distribution for broader range of random graphs.
For example, Chung & Radcliffe (2011) proved the convergence of the spectral distribution of the normalized Laplacian for Chung-Lu’s model (Chung & Lu, 2002), which includes power law graphs as a special case (see, Theorem 4 of Chung & Radcliffe (2011)).
H D ETAILS OF E XPERIMENT S ETTINGS H. E XPERIMENT OF S ECTION 6.
We set the eigenvalue of P to λ 1 = 0 .
5 and λ 2 = 1 . 0 and randomly generated P until the eigenvector e associated to λ 2 satisﬁes the condition of each case described in the main article.
We set W = 1 .and used the following values for each case as P and e .
H.1.
C ASE
P = (cid:20) 0 . 7469915 0 .0 .
2499819 0 .(cid:21) , e = (cid:20) 0 .− 0 .(cid:21) .
H.1.
C ASE
P = (cid:20) 0 .− 0 .− 0 .0 .(cid:21) , e = (cid:20) 0 .− 0 .(cid:21) .
H E XPERIMENT OF S ECTION 6.
We randomly generated an Erd˝os – R´enyi graph G N,p with N = 1000 and randomly generated a one-of- K hot vector for each node and embed it to a C -dimensional vector using a random matrix whose elements were randomly sampled from the standard Gaussian distribution.
Here, K =and C = 32 .
We treated the resulting single as the input signal X (0) ∈ R N × C .
We constructed a
GCN with L = 10 layers and C channels.
All parameters were i.i.d.
sampled from the Gaussian distribution whose standard deviation is same as the one used in LeCun et al.
(2012) 11 and multiplied a scalar to each weight matrix so that the largest singular value equals to a speciﬁed value s .
We used three conﬁgurations ( p, s ) =
λ of the generated GCNs are 0 .
194 , respectively.
See Appendix 6.2 for the results of other conﬁgurations of ( p, s ) .
[Table]
[Table]
H. E XPERIMENT OF S ECTION 6. H.3. D ATASET
We used the Cora (McCallum et al., 2000;
Sen et al., 2008), CiteSeer (Giles et al., 1998; Sen et al., 2008)
[Table]
and PubMed(Sen et al., 2008) datasets for experiments.
We obtained the preprocessed dataset from the code repository of Kipf & Welling (2017) 12 .
Table 1 summarizes speciﬁcations of datasets and their noisy version (explained in the next section)
[Table]
The Cora dataset is a citation network dataset consisting of 2708 papers and 5429 links.
Each paper is represented as the occurence of 1433 unique words and is associated to one of 7 genres (Case
Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule
Learning, Theory).
The graph made from the citation links has 78 connected components and the smallest positive eigenvalue of the augmented Normalized Laplacian is approximately ˜ µ = 3 .
62 × 10 − 3 .
Therefore, the upper bound of Theorem 2 is λ − 1 =
(1 − ˜ µ ) − 1 ≈ 1 + 3 .
62 × 10 − 3 .
818 out of 2708 samples are labelled as “Probabilistic Methods”, which is the largest proportion.
Therefore, the chance rate is 818 / 2708 = 30 .
The CiteSeer dataset is a citation network dataset consisting of 3312 papers and 4732 links.
Each paper is represented as the occurence of 3703 unique words and is associated to one of 6 genres (Agents, AI, DB, IR, ML, HCI).
The graph made from the citation links has 438 connected components and the smallest positive eigenvalue of the augmented Normalized Laplacian is approximately ˜ µ = 1 .
25 × 10 − 3 .
Therefore, the upper bound of Theorem 2 is λ − 1 =
(1 − ˜ µ ) − 1 ≈ 1+1 .
25 × 10 − 3 .
701 out of 2708 samples are labelled as “IR”, which is the largest proportion.
Therefore, the chance rate is 701 / 3312 = 21 .
The PubMed dataset is a citation network dataset consisting of 19717 papers and 44338 links.
Each paper is represented as the occurence of 500 unique words and is associated to one of 3 genres (“Diabetes Mellitus, Experimental”, “Diabetes Mellitus Type 1”, “Diabetes Mellitus Type 2”).
The graph made from the citation links has 438 connected components and the smallest positive eigenvalue of the augmented Normalized Laplacian is approximately ˜ µ = 9 .
48 × 10 − 3 .
Therefore, the upper bound of Theorem 2 is λ − 1 =
(1 − ˜ µ ) − 1 ≈ 1 + 9 .
57 × 10 − 3 .
7875 out of 19717 samples are labelled as “Diabetes Mellitus Type 2”, which is the largest proportion.
Therefore, the chance rate is 7875 / 19717 = 39 .
H.3. N OISY C ITATION N ETWORKS
We summarize the properties of noisy citation networks in Table 2.
[Figure]
We created two datasets from the Cora dataset: Noisy Cora 2500 and Noisy Cora 5000.
Noisy
Cora 2500 is made from the Cora dataset by uniformly randomly adding 2500 edges, respectively.
Since some random edges are overlapped with existing edges, the number of newly-added edges is 2495 in total.
We only changed the underlying graph from the Cora dataset and did not change word occurences (feature vectors) and genres (labels).
The underlying graph of the Noisy Cora dataset has two connected components and the smallest positive eigenvalue is ˜ µ ≈ 9 .
62 × 10 − 2 .
Therefore, the threshold of the maximum singular values of in Theorem 2 has been increased to λ − 1 =
(1 − ˜ µ ) − 1 ≈ 1 . 11 .
Similarly, Noisy Cora 5000 was made by adding 5000 edges uniformaly randomly.
The number of newly added edges is 4988 and the graph is connected (i
[Figure]
e., it has onlyconnected component).
˜ µ and λ are ˜ µ ≈ 1 .
32 × 10 − 1 and λ = (1 − ˜ µ ) − 1 ≈ 1 . 15 , respectively.
We made the noisy version of CiteSeer ( Noisy CiteSeer ) and PubMed ( Noisy PubMed ), in the similar way, by adding 5000 and 25000 edges uniformly randomly to the datasets.
Since some random edges were overlapped with existing edges, 4991 and 24993 edges are newly added, respectively.
This manipulation reduced the number of connected component of the graph to 3.
˜ µ is approximately 1 .
11 × 10 − 1 (Noisy CiteSeer) and 1 .
43 × 10 − 1 (Noisy PubMed) and λ − 1 = (1 − ˜ µ ) − 1 is approximately 1 .
(Noisy CiteSeer) and 1 .
(Noisy PubMed), respectively.
Figure 5 (right) shows the spectral distribution of the augmented normalized Laplacian
For comparison, we show in Figure 5 (left) the spectral distribution of the normalized Laplacian for these datasets 13 .
H.3. M ODEL A RCHITECTURE
We used a GCN consisting of a single node embedding layer, one to nine graph convolution layers, and a readout operation (Gilmer et al., 2017), which is a linear transformation common to all nodes in our case.
We applied softmax function to the output of GCN.
The output dimension of GCN is same as the number of classes (i.e., seven for Noisy Cora 2500/5000, six for Noisy CiteSeer, and three for Noisy PubMed).
We treated the number of units in each graph convolution layer as a hyperparameter.
Optionally, we speciﬁed the maximum singular values s of graph convolution
[Table]
layers.
The choice of s is either 0 .
5 (smaller than 1 ), s 1 (in the interval { 1 ≤ s < λ − 1 } ), 3 and(larger than λ − 1 ).
We used s 1 = 1 . 05 for Noisy Cora 2500, Noisy CiteSeer, and Noisy PubMed, and s 1 = 1 . 1 for Noisy Cora 5000 and Noisy CiteSeer so that s 1 is not close to the edges of the the interval { 1 ≤ s < λ − 1 } .
H.3. P ERFORMANCE E VALUATION P ROCEDURE
We split all nodes in a graph (either Noisy Cora 2500/5000 or Noisy CiteSeer) into training, validation, and test sets.
Data split is the same as the one done by Kipf & Welling (2017).
This is a transductive learning (Pan & Yang, 2010) setting because we can use node properties of the validation and test data during training.
We trained the model three times for each choice of hyperparemeters using the training set and deﬁned the objective function as the average accuracy on the validation set.
We chose the combination of hyperparameters that achieves the best value of objective function.
We evaluate the accuracy of the test dataset three times using the chosen combination of hyperparameters and computed their average and the standard deviation.
H.3. T RAINING
At initialization, we sampled parameters from the i.i.d.
Gaussian distribution.
If the scale of maximum singular values s was speciﬁed, we subsequently scaled weight matrices of graph convolution layers so that their maximum singular values were normalized to s .
The loss function was deﬁned as the sum of the cross entropy loss for all training nodes.
We train the model using the one of gradient-based optimization methods described in Table 3.
H.3. H YPERPRAMETERS
Table 3 shows the set of hyperparameters from which we chose.
Since we compute the representations of all nodes at once at each iteration, each epoch consists of 1 iteration.
We employ
Tree-structured Parzen Estimator (Bergstra et al., 2011) for hyperparameter optimization.
H.3. I MPLEMENTATION
We used Chainer Chemistry 14 , which is an extension library for the deep learning framework
Chainer (Tokui et al., 2015; 2019), to implement GCNs and Optuna (Akiba et al., 2019) for hyperparameter tuning.
We conducted experiments in a signel machine which has 2 Intel(R)
Xeon(R)
Gold 6136 CPU@3.00GHz (24 cores), 192 GB memory (DDR4), and 3 GPGPUs (NVIDIA Tesla
V100).
Our implementation achieved 68.1% with Dropout (Srivastava et al., 2014) (2 graph convolution layers) and 64.2% without Dropout (1 graph convolution layer) on the test dataset.
These are slightly worse than the accuracy reported in Kipf & Welling (2017), but are still comparable with it.
H. E XPERIMENT OF S ECTION 6.
The experiment settings are almost same as the experiment in Section 6.3.
The only difference is that we did not train the node embedding layer, which we put before convolution layers of a GCN, while we did in Section 6.3.
This is because we wanted to see the the effect of convolution operations on the perpendicular component of signals, while we interested in the prediction accuracy in real training settings in the previous experiment.
I A DDITIONAL E XPERIMENT R ESULTS I. E XPERIMENT OF S ECTION 6.
We show the vector ﬁeld V for various W in Figure 6 ( Case 1 ) and Figure 7 ( Case 2 ).
Parameters other than W are same as experiments in Section 6.1 (detail values are available in Appendix H.1).
I. E XPERIMENT OF S ECTION 6.
Figure 8 shows the relative log distance of signals and their upper bound for various edge probability p and the maximum singular value s .
Note that we generate a new graph for each conﬁguration of ( p, s ) .
Therefore, different conﬁgurations may have different graphs and hence different λ
even they have a same edge probability p in common
. I. E XPERIMENT OF S ECTION 6. I.3. P REDICTIVE A CCURACY
Figure 9 shows the comparison of predictive performance in terms the maximum singular value and layer size when the dataset is Noisy Cora 5000 (left) and Noisy Citeseer (right), respectively.
Concrete values are available in Table 4.
I.3. T RANSITION OF M AXIMUM S INGULAR V ALUES
Figure 10 – 13 show the transition of weight of graph convolution layers during training when the dataset is Noisy Cora 2500, Noisy Cora 5000, and Noisy CiteSeer, respectively.
We note that the result of 3-layered GCN from the Noisy Cora 2500 is identical to Figure 3 (right) of the main article.
I
E XPERIMENT OF S ECTION 6.
Figure 14 shows the logarithm of relative perpendicular component and prediction accuracy on
Noisy Cora, Noisy CiteSeer, and Noisy PubMed datasets.
We use Pearson R as a correlation coefﬁcient.
If GCNs have only one layer, it has more large relative perpendicular components (corresponding to right points in the ﬁgures) than GCNs which have other number of layers.
The correlation between the logarithm of relative perpendicular components and prediction accuracies are 0 .
( p = 6 . 890 × 10 − 6 ) for Noisy Cora, 0 .
( p = 1 .
× 10 − 2 ) for Noisy CiteSeer, and 0 .
679( p = 1 .
002 × 10 − 3 ) for Noisy PubMed, if we treat the one-layer case as outliers and remove them.
[Figure] [Figure]
[Table]
[Figure]
l
[Figure]
t i v
[Figure]
i
[Figure]
t
i
[Figure]
Layer Size 0.
A cc u r a c y s = 0.
s = 1.
s = 3.
s = 10.
Unnormalized
Layer Size 0.
A cc u r a c y s = 0.
s = 1.
s = 3.
s = 10.
Unnormalized
Layer Size
[Figure]
cc u r a c y s = 0.
s = 1.
s = 3.
s = 10.
Unnormalized
[Figure]
[Figure] [Figure]
[Figure] [Figure]