E FFECTS OF G RAPH C ONVOLUTIONS IN M ULTI - LAYER N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Convolutional Networks (GCNs) are one of the most popular architectures that are used to solve classiﬁcation problems accompanied by graphical information.
We present a rigorous theoretical understanding of the effects of graph convolutions in multi-layer networks.
We study these effects through the node classiﬁcation problem of a non-linearly separable Gaussian mixture model coupled with a stochastic block model.
First, we show that a single graph convolution expands the regime of the distance between the means where multi-layer networks can classify the data by a factor of at least 1 / 4 √ deg , where deg denotes the expected degree of a node.
Second, we show that with a slightly stronger graph density, two graph convolutions improve this factor to at least 1 / 4 √ n , where n is the number of nodes in the graph.
Finally, we provide both theoretical and empirical insights into the performance of graph convolutions placed in different combinations among the layers of a neural network, concluding that the performance is mutually similar for all combinations of the placement.
We present extensive experiments on both synthetic and real-world data that illustrate our results.
I NTRODUCTION
A large amount of interesting data and the practical challenges associated with them are deﬁned in the setting where entities have attributes as well as information about mutual relationships.
Traditional classiﬁcation models have been extended to capture such relational information through graphs (Hamilton, 2020) where each node has individual attributes and the edges of the graph capture the relationships among the nodes.
A variety of applications characterized by this type of graph-structured data include works in the areas of social analysis (Backstrom & Leskovec, 2011),
recommendation systems (Ying et al., 2018), computer vision (Monti et al., 2017), study of the properties of chemical compounds (Gilmer et al., 2017; Scarselli et al., 2009), statistical physics (Bapst et al., 2020; Battaglia et al., 2016), and ﬁnancial forensics (Zhang et al., 2017; Weber et al., 2019).
The most popular learning models for relational data use graph convolutions (Kipf & Welling, 2017), where the idea is to aggregate the attributes of the set of neighbours of a node instead of only utilizing its own attributes.
Despite several empirical studies of various GCN-type models (Chen et al., 2019;
Ma et al., 2022) that demonstrate that graph convolutions can improve the performance of traditional classiﬁcation methods, such as a multi-layer perceptron (MLP), there has been limited progress in the theoretical understanding of the beneﬁts of graph convolutions in multi-layer networks in terms of improving node classiﬁcation tasks.
Related work.
The capacity of a graph convolution for one-layer networks is studied in Baranwal et al.
(2021), along with its out-of-distribution (OoD) generalization potential.
A more recent work (Wu et al., 2022) formulates the node-level OoD problem, and develops a learning method that facilitates GNNs to leverage invariance principles for prediction.
In Gasteiger et al.
(2019), the authors utilize a propagation scheme based on personalized PageRank to construct a model that outperforms several GCN-like methods for semi-supervised classiﬁcation.
Through their algorithm,
APPNP, they show that placing power iterations at the last layer of an MLP achieves state of the art performance.
Our results align with this observation.
There exists a large amount of theoretical work on unsupervised learning for random graph models where node features are absent and only relational information is available (Decelle et al., 2011;
Massoulié, 2014; Mossel et al., 2018; 2015; Abbe & Sandon, 2015; Abbe et al., 2015; Bordenave et al., 2015; Deshpande et al., 2015; Montanari & Sen, 2016; Banks et al., 2016; Abbe & Sandon, 2018; Li et al., 2019; Kloumann et al., 2017; Gaudio et al., 2022).
For a comprehensive survey, see Abbe (2018);
Moore (2017).
For data models which have node features coupled with relational information, several works have studied the semi-supervised node classiﬁcation problem, see, for example, Scarselli et al.
; Cheng et al.
; Gilbert et al.
; Dang & Viennet (2012);
Günnemann et al.
; Yang et al.
Hamilton et al.
; Jin et al.
; Mehta et al.
; Chien et al.
Yan et al.
These papers provide good empirical insights into the merits of graph structure in the data.
We complement these studies with theoretical results that explain the effects of graph convolutions in a multi-layer network.
In Deshpande et al.
; Lu & Sen (2020), the authors explore the fundamental thresholds for the classiﬁcation of a substantial fraction of the nodes with linear sample complexity and large but ﬁnite degree.
Another relatively recent work (Hou et al., 2020) proposes two graph smoothness metrics for measuring the beneﬁts of graphical information, along with a new attention-based framework.
In Fountoulakis et al.
(2022), the authors provide a theoretical study of the graph attention mechanism (GAT) and identify the regimes where the attention mechanism is (or is not) beneﬁcial to nodeclassiﬁcation tasks.
Our study focuses on convolutions instead of attention-based mechanisms.
Several works have studied the expressive power and extrapolation of GNNs, along with the oversmoothing phenomenon (see, for e.g., Balcilar et al.
; Xu et al.
; Oono & Suzuki (2020);
Li et al.
Some other works have also studied the homophily and heterophily problem in GNNs (Luan et al., 2021; Ma et al., 2022).
However, our focus is to draw a comparison of the beneﬁts and limitations of graph convolutions with those of a traditional MLP that does not utilize relational information.
Similar to Wei et al.
(2022), our setting is immune to the heterophily problem, and the focus of our study is on regimes where oversmoothing does not occur.
To the best of our knowledge, this area of research still lacks theoretical guarantees that explain when and why graphical data, and in particular, graph convolutions, can boost traditional multilayer networks to perform better on node-classiﬁcation tasks.
To this end, we study the effects of graph convolutions in deeper layers of a multi-layer network.
For node classiﬁcation tasks, we also study whether one can avoid using additional layers in the network design for the sole purpose of gathering information from neighbours that are farther away, by comparing the beneﬁts of placing all convolutions in a single layer versus placing them in different layers.
Our contributions.
We study the performance of multi-layer networks for the task of binary node classiﬁcation on a data model where node features are sampled from a Gaussian mixture, and relational information is sampled from a symmetric two-block stochastic block model 1 (see Section 2.1 for details).
The node features are modelled after XOR data with two classes, and therefore, has four distinct components, two for each class.
Our choice of the data model is inspired from the fact that it is non-linearly separable.
Hence, a single layer network fails to classify the data from this model.
Similar data models based on the contextual stochastic block model (CSBM) have been used extensively in the literature, see, for example, Deshpande et al.
; Binkiewicz et al.
Chien et al.
; Baranwal et al.
We now summarize our contributions below.
We show that when node features are accompanied by a graph, a single graph convolution enables a multi-layer network to classify the nodes in a wider regime as compared to methods that do not utilize the graph, improving the threshold for the distance between the means of the features by a factor of at least 1 / 4 √ E deg .
Furthermore, assuming a slightly denser graph, we show that with two graph convolutions, a multi-layer network can classify the data in an even wider regime, improving the threshold by a factor of at least 1 / 4 √ n , where n is the number of nodes in the graph.
We show that for multi-layer networks equipped with graph convolutions, the classiﬁcation capacity is determined by the number of graph convolutions rather than the number of layers in the network.
In particular, we study the gains obtained by placing graph convolutions in a layer, and compare the beneﬁts of placing all convolutions in a single layer versus placing them in different combinations across different layers.
We ﬁnd that the performance is mutually similar for all combinations with the same number of graph convolutions.
We verify our theoretical results through extensive experiments on both synthetic and real-world data, showing trends about the performance of graph convolutions in various combinations across multiple layers of a network, and in different regimes of interest.
The rest of our paper is organized as follows:
In Section 2, we provide a detailed description of the data model and the network architecture that is central to our study, followed by our analytical results in Section 3.
Finally, Section 4 presents extensive experiments that illustrate our theoretical ﬁndings.
P RELIMINARIES 2. D ESCRIPTION OF THE DATA MODEL
Let n, d be positive integers, where n denotes the number of data points (sample size) and d denotes the dimension of the features.
Deﬁne the Bernoulli random variables ε 1 , . .
ε n ∼ Ber( 1 / 2 ) and η 1 , . . . ,
η n ∼ Ber( 1 / 2 ) .
Further, deﬁne two classes C b = { i ∈ [ n ] | ε i =
b } for b ∈ { 0 , 1 } .
Let µ and ν be ﬁxed vectors in R d , such that (cid:107) µ (cid:107)
= (cid:107) ν (cid:107) 2 and (cid:104) µ , ν (cid:105) = 0 .
2 Denote by X ∈ R n × d the data matrix where each row-vector X i ∈ R
d is an independent Gaussian random vector distributed as X i ∼ N ((2 η i − 1)((1 − ε i ) µ + ε i ν ) , σ 2 ) .
We use the notation X ∼ XOR-GMM ( n, d, µ , ν , σ 2 ) to refer to data sampled from this model.
Let us now deﬁne the model with graphical information.
In this case, in addition to the features X described above, we have a graph with the adjacency matrix,
A = ( a ij ) i,j ∈ [ n ] , that corresponds to an undirected graph including self-loops, and is sampled from a standard symmetric two-block stochastic block model with parameters p and q , where p is the intra-block and q is the inter-block edge probability.
The SBM ( n, p, q ) is then coupled with the XOR-GMM ( n, d, µ , ν , σ 2 ) in the way that a ij ∼ Ber( p ) if ε i = ε j and a ij ∼ Ber( q ) if ε i (cid:54) =
ε j .
For data ( A , X )
=
( { a ij } i,j ∈ [ n ] , { X i } i ∈ [ n ] ) sampled from this model, we say ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) .
We will denote by D the diagonal degree matrix of the graph with adjacency matrix A , and thus, deg ( i ) =
D ii =
(cid:80) nj =1 a ij denotes the degree of node i .
We will use N i = { j ∈ [ n ] | a ij = 1 } to denote the set of neighbours of a node i .
We will also use the notation i ∼ j or i (cid:28) j throughout the paper to signify, respectively, that i and j are in the same class, or in different classes.
2. N ETWORK ARCHITECTURE
Our analysis focuses on MLP architectures with ReLU activations.
In particular, for a network with
L layers, we deﬁne the following:
[Equation]
Here, X ∈ R n × d is the given data, which is an input for the ﬁrst layer and ϕ ( x )
= sigmoid( x ) =1+ e − x , applied element-wise.
The ﬁnal output of the network is represented by ˆy = { ˆ y i } i ∈
[ n ] .
Note that D − 1 A is the normalized adjacency matrix 3 and k l denotes the number of graph convolutions placed in layer l .
In particular, for a simple MLP with no graphical information, we have A = I n .
We will denote by θ , the set of all weights and biases, ( W ( l ) , b ( l ) )
l ∈ [ L ] , which are the learnable parameters of the network.
For a dataset ( X , y ) , we denote the binary cross-entropy loss obtained by a multi-layer network with parameters θ by (cid:96) θ ( A , X )
= − 1 n (cid:80) i ∈ [ n ] y i
log(ˆ y i )
+(1 − y i )
log(1 − ˆ y i ) , and the optimization problem is formulated as
[Equation]
where C denotes a suitable constraint set for θ .
For our analyses, we take the constraint set C to impose the condition (cid:13)(cid:13) W (1) (cid:13)(cid:13) 2 ≤ R and (cid:13)(cid:13) W ( l ) (cid:13)(cid:13) 2 ≤ 1 for all 1 < l ≤ L , i.e., the weight parameters of all layers l > 1 are normalized, while for l = 1 , the norm is bounded by some ﬁxed value R .
This is necessary because without the constraint, the value of the loss function can go arbitrarily close to 0 .
Furthermore, the parameter R helps us concisely provide bounds for the loss in our theorems for various regimes by bounding the Lipschitz constant of the learned function.
In the rest of our paper, we use (cid:96) θ ( X ) to denote (cid:96) θ
( I n , X ) , which is the loss in the absence of graphical information.
R ESULTS
We now describe our theoretical contributions, followed by a discussion and a proof sketch.
3. S ETTING UP THE BASELINE
Before stating our main result about the beneﬁts and performance of graph convolutions, we set up a comparative baseline in the setting where graphical information is absent.
In the following theorem, we completely characterize the classiﬁcation threshold for the XOR-GMM data model in terms of the distance between the means of the mixture model and the number of data points n .
Let Φ( · ) denote the cumulative distribution function of a standard Gaussian, and Φ c ( · ) = 1 − Φ( · ) .
Theorem 1.
Let X ∈ R n × d ∼ XOR-GMM ( n, d, µ , ν , σ 2 ) and deﬁne γ = (cid:107) µ − ν (cid:107) 2 to be the distance between the means.
Then we have the following:
1. Assume that γ ≤ Kσ and let h ( x ) : R d → { 0 , 1 } be any binary classiﬁer.
Then for any K > 0 and any (cid:15) ∈ (0 , 1) , at least a fraction 2Φ c ( K / 2 ) 2 − O ( n − (cid:15)/ 2 ) of all data points are misclassiﬁed by h with probability at least 1 − exp( − 2 n 1 − (cid:15) ) .
For any (cid:15) > 0 , if the distance between the means is γ = Ω( σ (log n ) 12 + (cid:15) ) , then for any c > 0 , with probability at least 1 − O ( n − c ) , there exist a two-layer and a three-layer network that perfectly classify the data, and obtain a cross-entropy loss given by
[Equation]
where C ∈ [ 1 / 2 , 1] is an absolute constant and R is the optimality constraint from Eq.
Part one of Theorem 1 shows that if the means of the features of the two classes are at most O ( σ ) apart then with overwhelming probability, there is a constant fraction of points that are misclassiﬁed.
Note that the fraction of misclassiﬁed points is 2Φ c ( K / 2 ) 2 , which approaches 0 as K → ∞ and approaches/ 2 as K → 0 , signifying that if the means are very far apart then we successfully classify all data points, while if they coincide then we always misclassify roughly half of all data points.
Furthermore, note that if K = c √ log n for some constant c ∈ [0 , 1) , then the total number of points misclassiﬁed is 2 n Φ c ( K ) 2 (cid:16) nK 2 e − K 2 (cid:16) n 1 − c 2 log n = Ω(1) .
Thus, intuitively, K (cid:16) √ log n is the threshold beyond which learning methods are expected to perfectly classify the data.
This is formalized in part two of the theorem, which supplements the misclassiﬁcation result by showing that if the means are roughly ω ( σ √ log n ) apart then the data is classiﬁable with overwhelming probability.
3. I MPROVEMENT THROUGH GRAPH CONVOLUTIONS
We now state the results that explain the effects of graph convolutions in multi-layer networks with the architecture described in Section 2.2.
We characterize the improvement in the classiﬁcation threshold in terms of the distance between the means of the node features.
Let erf( t ) =
2Φ( t √ 2) − 1 be the
Gauss error function and ζ ( t ) =
t erf( t ) − (1 − exp( − t 2 )) / √ π .
Theorem 2.
Let ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) , γ = (cid:107) µ − ν (cid:107) 2 , and Γ( p, q ) =
| p − q | / ( p + q ) .
There exist a two-layer network and a three-layer network with the following properties:
• If the intra-class and inter-class edge probabilities are p, q = Ω( log 2 n n ) , and it holds that Γ( p, q ) ζ ( γ / 2 σ )
= ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) , then for any c > 0 , with probability at least 1 − O ( n − c ) , the networks equipped with a graph convolution in the second or the third layer perfectly classify the data, and obtain the following loss:
[Equation]
where C > 0 and C (cid:48) ∈ [ 1 / 2 , 1] are constants and R is the constraint from Eq.
• If p, q = Ω( log n √ n ) and Γ( p, q ) 2 ζ ( γ / 2 σ ) = ω (cid:18)(cid:113) log nn (cid:19) , then for any c > 0 , with probability at least 1 − O ( n − c ) , the networks with any combination of two graph convolutions in the second and/or the third layers perfectly classify the data, and obtain the following loss:
[Equation]
where C > 0 and C (cid:48) ∈ [ 1 / 2 , 1] are constants and R is the constraint from Eq.
To understand Theorem 2, it helps to consider the regime where Γ( p, q ) = Ω(1) .
Part one of the theorem shows that under the assumption that p, q = Ω( log 2 n / n ) , a single graph convolution improves the classiﬁcation threshold for γ , the distance between the means by a factor of at least 1 / 4 √ n ( p + q ) as compared to the case without the graph.
Part two then shows that with a slightly stronger assumption on the graph density, we observe further improvement in the threshold up to a factor of at least 1 / 4 √ n .
We refer to Appendix A.8 for a comprehensive explanation of this simpler case.
Note that although the regime of graph density is different for part two of the theorem, the result itself is an improvement.
In particular, if p, q = Ω( log n / √ n ) then part one of the theorem states that one graph convolution achieves an improvement of at least 1 / 8 √ n , while part two states that two convolutions improve it to at least 1 / 4 √ n .
However, we also emphasize that in the regime where the graph is dense, i.e., when p, q = Ω n (1) , two graph convolutions do not have a signiﬁcant advantage over one graph convolution.
Our experiments in Section 4.1 demonstrate this effect.
The XOR-CSBM data model also demonstrates why graph convolutions in the ﬁrst layer can severely hurt the classiﬁcation accuracy.
Hence, for Theorem 2, our analysis only considers networks with no graph convolution in the ﬁrst layer, i.e., k 1 = 0 .
This effect is visualized in Fig.
1, and is attributed to the averaging of data points in the same class but different components of the mixture that have means with opposite signs.
We defer the reader to Appendix A.5 for a more formal argument, and to Appendix B.1 for experiments that demonstrate this phenomenon.
As n (the sample size) grows, the difference between the averages of node features over the two classes diminishes (see Figs.
1a and 1b).
In other words, the means of the two classes collapse to the same point for large n .
However, in the last layer, since the input consists of transformed features that are linearly separable, a graph convolution helps with the classiﬁcation task (see Figs. 1c and 1d).
3. P LACEMENT OF GRAPH CONVOLUTIONS
We observe that the improvements in the classiﬁcation capability of a multi-layer network depends on the number of convolutions, and does not depend on where the convolutions are placed.
In particular, for the XOR-CSBM data model, putting the same number of convolutions among the second and/or the third layer in any combination achieves mutually similar improvements in the classiﬁcation task.
Corollary 2.1.
Consider the data model XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) and the network architecture from Section 2.2.
• Assume that p, q = Ω( log 2 n / n ) , and consider the three-layer network characterized by part one of Theorem 2, with one graph convolution.
For this network, placing the graph convolution in the second layer ( k 2 = 1 , k 3 = 0 ) obtains the same results as placing it in the third layer ( k 2 = 0 , k 3 = 1 ).
• Assume that p, q = Ω( log n / √ n ) , and consider the three-layer network characterized by part two of Theorem 2, with two graph convolutions.
For this network, placing both convolutions
[Figure]
in the second layer ( k 2 = 2 , k 3 = 0 ) or both of them in the third layer ( k 2 = 0 , k 3 = 2 ) obtains the same results as placing one convolution in the second layer and one in the third layer ( k 2 = 1 , k 3 = 1 ).
Corollary 2.1 is immediate from the proof of Theorem 2 (see Appendices A.6 and A.7).
In Section 4, we also show extensive experiments on both synthetic and real-world data that demonstrate this result.
3. P ROOF SKETCH
In this section, we provide an overview of the key ideas and intuition behind our proof technique for the results.
For comprehensive proofs, see Appendix A.
For part one of Theorem 1, we utilize the assumption on the distribution of the data.
Since the underlying distribution of the mixture model is known, we can ﬁnd the (Bayes) optimal classiﬁer 4 , h ∗ ( x ) , for the XOR-GMM, which takes the form h ∗ ( x ) = 1 ( | (cid:104) x , ν (cid:105) | − | (cid:104) x , µ (cid:105) | ) , where 1 ( · ) is the indicator function.
We then compute a lower bound on the probability that h ∗ fails to classify one data point from this model, followed by a concentration argument that computes a lower bound on the fraction of points that h ∗ fails to classify with overwhelming probability.
Consequently, a negative result for the Bayes optimal classiﬁer implies a negative result for all classiﬁers.
For part two of Theorem 1, we design a two-layer and a three-layer network that realize the (Bayes) optimal classiﬁer.
We then use a concentration argument to show that in the regime where the distance between the means is large enough, the function representing our two-layer or three-layer network roughly evaluates to a quantity that has a positive sign for one class and a negative sign for the other class.
Furthermore, the output of the function scales with the distance between the means.
Thus, with a suitable assumption on the magnitude of the distance between the means, the output of the networks has the correct signs with overwhelming probability.
Following this argument, we show that the cross-entropy loss obtained by the networks can be made arbitrarily small by controlling the optimization constraint R (see Eq.
(1)), implying perfect classiﬁcation.
For Theorem 2, we observe that for the (Bayes) optimal networks designed for Theorem 1, placing graph convolutions in the second or the third layer reduces the effective variance of the functions representing the network.
This stems from the fact that for the data model we consider, multi-layer networks with ReLU activations are Lipschitz functions of Gaussian random variables.
First, we compute the precise reduction in the variance of the data characterized by K > 0 graph convolutions (see Lemma A.3).
Then for part one of the theorem where we analyze one graph convolution, we use the assumption on the graph density to conclude that the degrees of each node concentrate around the expected degree.
This helps us characterize the variance reduction, which further allows the distance between the means to be smaller than in the case of a standard MLP, hence, obtaining an improvement in the threshold for perfect classiﬁcation 5 .
Part two of the theorem studies the placement of two graph convolutions using a very similar argument.
In this case, the variance reduction is characterized by the number of common neighbours of a pair of nodes rather than the degree of a node, and is stronger than the variance reduction offered by a single graph convolution.
E XPERIMENTS
In this section we provide empirical evidence that supports our claims in Section 3.
We begin by analyzing the synthetic data models XOR-GMM and XOR-CSBM that are crucial to our theoretical results, followed by a similar analysis on multiple real-world datasets tailored for node classiﬁcation tasks.
We show a comparison of the test accuracy obtained by various learning methods in different regimes, along with a display of how the performance changes with the properties of the underlying graph, i.e., with the intra-class and inter-class edge probabilities p and q .
For both synthetic and real-world data, the performance of the networks does not change signiﬁcantly with the choice of the placement of graph convolutions.
In particular, placing all convolutions in the last layer achieves a similar performance as any other placement for the same number of convolutions.
This observation aligns with the results in Gasteiger et al.
4. S YNTHETIC DATA
In this section, we empirically show the landscape of the accuracy achieved for various multi-layer networks with up to three layers and up to two graph convolutions 6 .
In Fig.
2, we show that as claimed in Theorem 2, a single graph convolution reduces the classiﬁcation threshold by a factor of 1 / 4 √ E deg and two graph convolutions reduce the threshold by a factor of 1 / 4 √ n , where E deg = n 2 ( p + q ) .
We observe that the placement of graph convolutions does not matter as long as it is not in the ﬁrst layer.
Figs.
2a and 2b show that the performance is mutually similar for all networks that have one graph convolution placed in the second or the third layer, and for all networks that have two graph convolutions placed in any combination among the second and the third layers.
In Figs.
2c and 2d, we observe that two graph convolutions do not obtain a signiﬁcant advantage over one graph convolution in the setting where p and q are large, i.e., when the graph is dense.
We observed similar results for various other values of p and q (see Appendix B.1 for some more plots).
Furthermore, in Appendix B.1 we verify that if a graph convolution is placed in the ﬁrst layer of a network, then it is difﬁcult to learn a classiﬁer for the XOR-CSBM data model.
In this case, test accuracy is low even for the regime where the distance between the means is quite large.
4. R EAL - WORLD DATA
For real-world data, we test our results on three graph benchmarks: CORA , CiteSeer , and Pubmed citation network datasets (Sen et al., 2008).
Results for larger datasets are presented in Appendix B.2.
We observe the following trends: First, as claimed in Theorem 2, networks that utilize the graph perform remarkably better than a traditional MLP that does not use relational information.
Second,
[Figure]
all networks with one graph convolution in any layer achieve a mutually similar performance, and all networks with two graph convolutions in any combination of placement achieve a mutually similar performance.
This demonstrates a result similar to Corollary 2.1 for real-world data.
Finally, networks with two graph convolutions perform better than networks with one graph convolution.
In Fig.
3, we present for all networks, the maximum accuracy over 50 trials, where each trial corresponds to a random initialization of the networks.
For 2 -layer networks, the hidden layer has width 16 , and for 3 -layer networks, both hidden layers have width 16 .
We use a dropout probability of 0 .
5 and a weight decay of 10 − 5 while training.
For this study, we attribute minor changes in the accuracy to hyperparameters involving dropout and weight decay.
This helps us clearly observe the important difference in the accuracy of networks with one graph convolution versus two graph convolutions.
For example, in Fig.
3a, we note that there are differences among the accuracy of the networks with one graph convolution (red and blue).
However, these differences are minor compared to the networks with one convolution (red and blue) and networks with two convolutions (green and yellow).
We also show the averaged accuracy in
Appendix B.2.
Note that the accuracy slightly differs from well-known results in the literature due to implementation differences.
In particular, the GCN implementation in Kipf & Welling (2017) uses ˜
A = D − 12 AD − 12 as the normalized adjacency matrix, however, we use ˜
A
= D − 1 A .
In
Appendix B.2, we also show empirical results for the normalization ˜
A = D − 12 AD − 12 .
C ONCLUSION AND FUTURE WORK
We study the fundamental limits of the capacity of graph convolutions when placed beyond the ﬁrst layer of a multi-layer network for the XOR-CSBM data model, and provide theoretical guarantees for their performance in different regimes of the signal in the data.
Through our experiments on both synthetic and real-world data, we show that the number of convolutions is a more signiﬁcant factor for determining the performance of a network, rather than the number of layers in the network.
[Figure]
Furthermore, we show that placing graph convolutions in any combination achieves mutually similar performance enhancements for the same number of them.
We observe that multiple graph convolutions are advantageous when the underlying graph is relatively sparse.
Intuitively, this is because in a dense graph, a single convolution can gather information from a large number of nodes, while in a sparser graph, more convolutions are needed to gather information from a larger number of nodes.
Our analysis is limited to a positive result and we only provide a minimum guarantee for improvement in the classiﬁcation threshold.
To fully understand the limitations of graph convolutions, a complementary negative result (similar to part one of Theorem 1) for data models with relational information is required, showing the maximum improvement that graph convolutions can realize in a multi-layer network.
This problem is hard due to two reasons: First, there does not exist a concrete notion of an optimal classiﬁer for data models which have node features coupled with relational information.
Second, a graph convolution transforms an iid set of features into a highly correlated set of features, making it difﬁcult to apply classical high-dimensional concentration arguments.
Another potential line of work is to generalize our results for arbitrary data models.
However, since our arguments rely heavily on concentration of Gaussian variables, it is hard to extend the analysis to arbitrary distributions.
Therefore, we require mathematical tools that are distribution-agnostic.
A P ROOFS A. A SSUMPTIONS AND NOTATION
Assumption 1.
For the XOR-GMM data model the means of the Gaussian mixture are such that (cid:104) µ , ν (cid:105) =
and (cid:107) µ (cid:107)
= (cid:107) ν (cid:107) 2 .
We denote [ x ] + = ReLU( x ) and ϕ ( x ) = sigmoid( x ) = 1 / 1+ e − x , applied element-wise on the inputs.
For any vector v , ˆ v = v (cid:107) v (cid:107) 2 denotes the normalized v .
We use γ = (cid:107) µ − ν (cid:107) 2 to denote the distance between the means of the inter-class components of the mixture model, and γ (cid:48) to denote the norm of the means, γ (cid:48)
= γ / √ 2 = (cid:107) µ (cid:107) [Equation]
Given intra-class and inter-class edge probabilities p and q , we deﬁne Γ( p, q ) =
| p − q | p + q .
We denote the probability density function of a standard Gaussian by φ ( x ) , and the cumulative distribution function by Φ( x ) .
The complementary distribution function is denoted by Φ c ( x ) = 1 − Φ( x ) .
A. E LEMENTARY RESULTS
In this section, we state preliminary results about the concentration of the degrees of all nodes and the number of common neighbours for all pairs of nodes, along with the effects of a graph convolution on the mean and the variance of some data.
Our results regarding the merits of graph convolutions rely heavily on these arguments.
Proposition A.1 (Concentration of degrees) .
Assume that the graph density is p, q = Ω( log 2 n n ) .
Then for any constant c > 0 , with probability at least 1 − 2 n − c , we have for all i ∈ [ n ] that
[Equation]
where the error term o n (1) =
O (cid:16)(cid:113)
c log n (cid:17) .
Proof.
Note that deg ( i ) is a sum of n Bernoulli random variables, hence, we have by the Chernoff bound (Vershynin, 2018, Section 2) that
[Equation]
for some C > 0 .
We now choose δ = (cid:113) ( c +1)log n Cn ( p + q ) for a large constant c > 0 .
Note that since p, q = Ω( log 2 n / n ) , we have that δ
= O ( (cid:113) c log n )
=
o n (1) .
Then following a union bound over i ∈ [ n ] , we obtain that with probability at least 1 − 2 n − c ,
[Equation]
Note that
[Equation]
a ij for any b ∈ { 0 , 1 } is a sum of independent Bernoulli random variables.
Hence, by a similar argument, we have that with probability at least 1 − 2 n − c ,
[Equation]
Proposition A.2 (Concentration of the number of common neighbours) .
Assume that the graph density is p, q = Ω( log n √ n ) .
Then for any constant c > 0 , with probability at least 1 − 2 n − c , | N i ∩ N
[Equation]
for all i ∼ j, | N
[Equation]
| =
[Equation]
± o n (1)) for all i (cid:28) j, where the error term o n (1) =
O (cid:16)(cid:113) c log n (cid:17) .
Proof.
For any two distinct nodes i, j ∈ [ n ] we have that the number of common neighbours of i and j is | N i ∩ N j | = (cid:80) k ∈ [ n ] a ik a jk .
This is a sum of independent Bernoulli random variables, with mean
E | N i ∩ N j | = n 2 ( p 2 + q 2 ) for i ∼ j and E | N i ∩ N j | = npq for i (cid:28) j .
Denote µ ij = E |
N i ∩ N j | .
Therefore, by the Chernoff bound (Vershynin, 2018, Section 2), we have for a ﬁxed pair of nodes ( i, j ) that
[Equation]
for some constant C > 0 .
We now choose δ ij = (cid:113) ( c +2)log n Cµ ij for any large c > 0 .
Note that since p, q = Ω( log n / √ n ) , we have that δ ij =
O ( (cid:113) c log n )
=
o n (1) .
Then following a union bound over all pairs ( i, j ) ∈ [ n ] × [ n ] , we obtain that with probability at least 1 − 2 n − c , for all pairs of nodes ( i, j ) we have
[Equation]
for all i ∼ j,
[Equation]
for all i (cid:28) j.
Lemma A.3 (Variance reduction) .
Denote the event from Proposition A.1 to be B .
Let { X i } i ∈ [ n ] ∈
R n × d be an iid sample of data.
For a graph with adjacency matrix A (including self-loops) and a ﬁxed integer K > 0 , deﬁne a K -convolution to be ˜ X
= ( D − 1 A ) K X .
Then we have
[Equation]
˜
[Equation]
| [Equation] [Equation] ρ
[Equation]
K ) [Equation] ( X i ) , where ρ ( K ) =
[Equation]
j ) 2 . Here, A K ( i, j ) is the entry in the i th row and j th column of the exponentiated matrix A K and ∆ =
E deg = n 2 ( p + q ) .
Proof.
For a matrix M , the i th convolved data point is ˜ X i =
M (cid:62) i X , where M (cid:62) i denotes the i th row of M .
Since X i are iid, we have
[Equation]
It remains to compute the entries of the matrix M =
( D − 1 A ) K .
Note that we have D − 1 A ( i, j )
= a ij / deg ( i ) , so we obtain that
[Equation]
Recall that on the event B , the degrees of all nodes are ∆(1 ± o n (1)) , and hence, we have that
[Equation]
where the error o n (1) =
O ( 1 √ log n ) .
The sum of these products of the entries of A is simply the number of length- K paths from node i to j , i.e., A K ( i, j ) .
Thus, we have
[Equation]
Since X j are iid, we obtain that ρ ( K ) = (cid:16)
[Equation]
+ o
[Equation]
Let us brieﬂy discuss the implications of Lemma A.3.
Consider a sample ( A , X ) drawn from
XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) for the symmetric case where exactly n/ 2 nodes are in each of the two classes.
We have that
[Equation]
This gives us E ρ ( K ) ≈ 1 n (1 + Γ( p, q ) 2 K ) for any K ≥ 2 .
Recall that a single graph convolution reduces the distance between the means by a factor of Γ( p, q ) .
Hence, to comment on the performance of an arbitrary number of convolutions, K , we might hope to compare the reduction in this distance, Γ( p, q ) K with the reduction in the variance ( ρ ( K ) ) to obtain a condition on K in terms of n , p , and q .
The challenge, however, lies in the fact that in deeper layers, computing ρ ( K ) is non-trivial due to node features being highly correlated.
Moreover, an argument to claim that ρ ( K ) ≈ E ρ ( K ) is needed for this approach, which seems to require strong density assumptions on the graph.
We now state a result about the output of the (Bayes) optimal classiﬁer for the XOR-GMM data model that is used in several of our proofs.
Lemma A.4.
Let h ( x ) = | (cid:104) x , ˆ ν (cid:105) | − | (cid:104) x , ˆ µ (cid:105) | for all x ∈ R d and deﬁne
[Equation]
Then we have
The expectation E h ( X i ) = (cid:26) −√ 2 σζ ( γ / 2 σ ) i ∈ C 0 √ 2 σζ ( γ / 2 σ )
i ∈ C 1 .
For any γ, σ > 0 such that γ = Ω n ( σ ) , we have that ζ ( γσ ) =
Ω( γσ ) .
For any γ, σ > 0 such that γ = o n ( σ ) , we have that ζ ( γσ ) =
Ω( γ 2 σ 2 ) .
Proof.
For part one, observe that (cid:104) X
i , ˆ µ (cid:105) and (cid:104) X
i , ˆ ν (cid:105) are Gaussian random variables with variance σ 2 and means γ / √ 2 ,
if ε i = 0 and 0 , γ / √ 2 if ε i = 1 , respectively.
Thus, | (cid:104) X i , ˆ µ (cid:105) | and |
(cid:104) X i , ˆ ν (cid:105) | are folded-Gaussian random variables
and we have E h
( X i )
= −√ 2 ζ ( γ / √ 2 σ ) if i ∈ C 0 and E h
( X i ) = √ 2 ζ ( γ / √ 2 σ ) otherwise.
We now write ζ ( t ) = t
[Equation]
, where H ( t ) = erf( t ) − 1 / t √ π (1 − e − t 2 ) .
For part two, note that H ( t ) is an increasing function in the range [ − 1 , 1] and H ( t ) > 0 for t > 0 .
Hence, for t ≥ C for some positive constant C , H ( t ) ≥ H ( C )
=
C (cid:48) , therefore, ζ ( t ) =
tH ( t ) ≥ C (cid:48) t .
For part three when t = o n (1) , we use the series expansion of h ( t ) about t = 0 to obtain that
[Equation]
Hence, ζ ( t ) = th ( t ) =
Ω( t 2 ) .
Putting t = γ/σ completes the proof.
Fact A.5.
For any x ∈ [0 , 1] , x 2 ≤ log(1 + x ) ≤ x .
A. P ROOF OF T HEOREM 1 PART ONE
In this section we prove our ﬁrst result about the fraction of misclassiﬁed points in the absence of graphical information.
We begin by computing the Bayes optimal classiﬁer for the data model XOR-GMM (see Section 2.1).
A Bayes classiﬁer, denoted by h ∗ ( x ) , maximizes the posterior probability of observing a label given the input data x .
More precisely, h ∗ ( x ) = argmax b ∈{ 0 , 1 } Pr [ y = b
| x = x ] , where x ∈ R d represents a single data point.
Lemma A.6.
For some ﬁxed µ , ν ∈ R d and σ 2 > 0 , the Bayes optimal classiﬁer, h ∗ ( x ) : R d → { 0 , 1 } for the data model XOR-GMM ( n, d, µ , ν , σ 2 ) is given by
[Equation]
where 1 is the indicator function.
Proof.
Note that Pr [ y = 0] =
Pr [ y = 1] = 12 .
Let f x ( x ) denote the density function of a continuous random vector x .
Therefore, for any b ∈ { 0 , 1 } ,
[Equation]
Let’s compute this for b = 0 .
We have
[Equation]
where in the last equation we used the assumption that (cid:107) µ (cid:107)
= (cid:107) ν
(cid:107) .
The decision regions are then identiﬁed by: Pr [ y =
| x ] ≥ 1 / 2 for label
and
Pr [ y = 0 | x ] < 1 / 2 for label 1 .
Thus, for label 0 , we need f x | y ( x | y =1) f x | y ( x | y =0) < 1 , which implies that cosh( [Equation] x [Equation]
ν [Equation] /σ 2 ) cosh( (cid:104) x , µ (cid:105) /σ 2 ) ≤ 1 . Now we note that cosh( x ) ≤ cosh( y ) =
⇒ | x | ≤ | y | for all x, y ∈ R , hence, we have |(cid:104) x , µ (cid:105)| ≥ |(cid:104) x , ν (cid:105)| .
Similarly, we have the complementary condition for label 1 .
Next, we design a two-layer and a three-layer network and show that for a particular choice of parameters θ = ( W ( l ) , b ( l ) ) for l ∈ { 1 , 2 } for the two-layer case and l ∈ { 1 , 2 , 3 } for the three-layer case, the networks realize the optimal classiﬁer described in Lemma A.6.
Proposition A.7.
Consider two-layer and three-layer networks of the form described in Section 2.2, without biases (i.e., b ( l ) = 0 for all layers l ), for parameters W ( l ) and some R ∈ R + as follows.
For the two- layer network
[Equation]
For the three-layer network,
[Equation]
Then for any σ > 0 , the deﬁned networks realize the Bayes optimal classiﬁer for the data model
XOR-GMM ( n, d, µ , ν , σ 2 ) .
Proof.
Note that the output of the two-layer network is ϕ ([ XW (1) ]
+ W (2) ) , which is interpreted as the probability with which the network believes that the input is in the class with label 1 .
The ﬁnal prediction for the class label is thus assigned to be 1 if the output is ≥ 0 .
5 , and 0 otherwise.
For each i ∈ [ n ] , we have that the output of the network on data point i is
[Equation]
where we used the fact that [ t ] + + [ − t ] + = | t | for all t ∈ R .
Similarly, for the three-layer network [Equation] the output is ϕ ([[ XW (1) ]
+ W (2) ] + W (3) ) .
So we have for each i ∈ [ n ] that
[Equation]
where in the last equation we used the fact that [ t ] + − [ − t ] + = t for all t ∈ R .
The ﬁnal prediction is then obtained by considering the maximum posterior probability among the class labels 0 and 1 , and thus,
[Equation]
which matches the Bayes classiﬁer in Lemma A.6.
We now restate the relevant theorem below for convenience.
Theorem (Restatement of part one of Theorem 1) .
Let X ∈ R n × d ∼ XOR-GMM ( n, d, µ , ν , σ 2 ) .
Assume that (cid:107) µ − ν (cid:107) 2 ≤ Kσ and let h ( x ) : R d → { 0 , 1 } be any binary classiﬁer.
Then for any K > 0 and any (cid:15) ∈ (0 , 1) , at least a fraction 2Φ c ( K / 2 ) 2 − O ( n − (cid:15)/ 2 ) of all data points are misclassiﬁed by h with probability at least 1 − exp( − 2 n 1 − (cid:15) ) .
Proof.
Recall from Lemma A.6 that for successful classiﬁcation, we require for every i ∈ [ n ] ,
[Equation]
| i ∈ C 1 .
Let’s try to upper bound the probability of the above event, i.e., the probability that the data is classiﬁable.
We consider only class C 0 , since the analysis for C 1 is symmetric and similar.
For i ∈ C 0 , we can write X i = µ + σ g
i , where g i ∼ N ( 0 , I ) .
Recall that γ = (cid:107) µ − ν (cid:107) 2 and γ (cid:48)
= γ / √ 2 = (cid:107) µ (cid:107) 2 = (cid:107) ν (cid:107) 2 .
Then we have for any ﬁxed i ∈ C 0 that
[Equation]
(by triangle inequality)
[Equation]
(using γ ≤ Kσ ) .
We now deﬁne random variables Z 1 = (cid:104) g
i , ˆ ν (cid:105) and Z 2 = (cid:104) g
i , ˆ µ (cid:105) and note that Z 1 , Z 2 ∼ N (0 , 1) and E [ Z 1 Z 2 ]
= 0 .
Let K (cid:48)
=
K/ √ 2 .
We now have
Pr
[Equation]
[Equation]
To evaluate the integral above, we used (Owen, 1980, Table 1:10,010.6 and Table 2:2.3).
Thus, the probability that a point i ∈ C 0 is misclassiﬁed is lower bounded as follows
[Equation]
Note that this is a decreasing function of K , implying that the probability of misclassiﬁcation decreases as we increase the distance between the means, and is maximum for K = 0 .
Deﬁne M ( n ) for a ﬁxed K to be the fraction of misclassiﬁed nodes in C 0 .
Deﬁne x i to be the indicator random variable 1 ( X i is misclassiﬁed ) .
Then x i are Bernoulli random variables with mean at least τ K , and E M ( n ) = 2 n (cid:80) i ∈
C
E x i ≥ τ K .
Using Hoeffding’s inequality (Vershynin, 2018,
Theorem 2.2.6), we have that for any t > 0 ,
[Equation]
Choosing t = n − (cid:15)/ 2 for any (cid:15) ∈ (0 , 1) yields
[Equation]
A. P ROOF OF T HEOREM 1 PART TWO
In this section, we show that in the positive regime (sufﬁciently large distance between the means), there exists a two-layer MLP that obtains an arbitrarily small loss, and hence, successfully classiﬁes a sample drawn from the XOR-GMM model with overwhelming probability.
Theorem (Restatement of part two of Theorem 1) .
Let X ∈ R n × d ∼ XOR-GMM ( n, d, µ , ν , σ 2 ) .
For any (cid:15) > 0 , if the distance between the means is (cid:107) µ − ν (cid:107) 2 =
Ω( σ (log n ) 12 + (cid:15) ) , then for any c > 0 , with probability at least 1 − O ( n − c ) , the two-layer and three-layer networks described in
Proposition A.7 classify all data points, and obtain a cross-entropy loss given by
[Equation]
where C ∈ [ 1 / 2 , 1] is an absolute constant.
Proof.
Consider the two-layer and three-layer MLPs described in Proposition A.7, for which we have ˆ y i =
[Equation] | [Equation]
i , [Equation] | [Equation]
|
[Equation]
X i , ˆ µ (cid:105)| )) . We now look at the loss for a single data point X i ,
[Equation]
Note that (cid:104)
X
i − E X
i , ˆ µ (cid:105) and (cid:104) X
i − E X
i , ˆ ν (cid:105) are mean
Gaussian random variables with variance σ 2 .
So for any ﬁxed i ∈ [ n ] and m c ∈ { µ , ν } , we use (Vershynin, 2018, Proposition 2.1.2) to obtain
[Equation]
Then by a union bound over all i ∈ [ n ] and m c ∈ { µ , ν } , we have that
[Equation]
We now set t = σ (cid:112) 2( c + 1) log n for any large constant c > 0 .
We now have with probability at least 1 − n − c √ π ( c +1)log n that
[Equation]
Thus
[Equation]
we can write
[Equation]
Using Eqs.
(2) and (3) in the expression for the loss
[Equation]
we obtain for all i ∈ [ n ] , (cid:96) i ( X , [Equation] = [Equation]
[Equation]
n
[Equation]
[Equation]
where the error term o n (1) =
(cid:112) c / log n .
The total loss is then given by
[Equation]
Next, Fact A.5 implies that for t < 0 , e t / 2 ≤ log(1 + e t ) ≤ e t , hence, we have that there exists a constant C ∈ [ 1 / 2 , 1] such that
[Equation]
Note that by scaling the optimality constraint R , the loss can go arbitrarily close to 0 .
A. G RAPH CONVOLUTION IN THE FIRST LAYER
In this section, we show precisely why a graph convolution operation in the ﬁrst layer is detrimental to the classiﬁcation task.
Proposition A.8.
Fix a positive integer d > 0 , σ ∈ R + and µ , ν ∈ R d .
Let ( A , X ) ∼
XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) .
Deﬁne ˜
X to be the transformed data after applying a graph convolution on X , i.e., ˜ X
= D − 1 AX .
Then in the regime where p, q = Ω( log 2 n n ) , with probability at least 1 − 1 / poly( n ) we have that
[Equation]
Hence, the distance between the means of the convolved data, given by p − q 2( p + q ) (cid:107) µ − ν (cid:107) 2 · o n (1) diminishes to 0 for n → ∞ .
Proof.
Fix µ , ν ∈ R d and deﬁne the following sets:
[Equation]
Denote ˜ X
= D − 1 AX and note that for any i ∈ [ n ] , the row vector ˜ X
i
[Equation]
i
[Equation]
(cid:88)
[Equation]
where we used the fact that X j =
(2 η j − 1)((1 − ε j ) µ + ε j ν + σ g j ) for a set of iid Gaussian random vectors g j ∼ N ( 0 , I d ) .
Note that since (cid:15) i , η i are Bernoulli random variables, using the Chernoff bound (Vershynin, 2018,
Section 2), we have that with probability at least 1 − 1 / poly( n ) ,
[Equation]
We now use an argument similar to Proposition A.1 to obtain that for any c > 0 , with probability at least 1 − O ( n − c ) , the following holds for all i ∈ [ n ] :
[Equation]
− µ a ij 
 =
O (cid:18) (1 − ε i ) p + ε i q 2( p + q ) (cid:114) c log n (cid:19) , deg ( i )   (cid:88)
j ∈ C ν a ij − (cid:88)
j ∈ C − ν a ij 
 =
O (cid:18) ε i p + (1 − ε i )
q 2( p + q ) (
cid:114) c log n (cid:19) .
Hence, we have that for all i ∈ [ n ] ,
[Equation]
Using the above result, we obtain the distance between the means, which is of the order o n ( γ ) and thus, diminishes to 0 as n → ∞ .
A. P ROOF OF T HEOREM 2 PART ONE
We begin by computing the output of the network when one graph convolution is applied at any layer other than the ﬁrst.
Lemma A.9.
Let h ( x ) = | (cid:104) x , ˆ ν (cid:105) | − | (cid:104) x , ˆ µ (cid:105) | for any x ∈ R d .
Consider the two-layer and threelayer networks in Proposition A.7 where the weight parameter of the last layer, W ( L ) , is scaled by a factor of ξ = sgn( p − q ) .
If a graph convolution is added to these networks in either the second or the third layer then for a sample ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) , the output of the networks for a point i ∈ [ n ] is
[Equation]
Proof.
The networks with scaled parameters are given as follows.
For the two-layer network,
[Equation]
For the three-layer network,
[Equation]
When a graph convolution is applied at the second layer of this two-layer MLP, the output of the last layer for data ( A , X ) is f (2) i ( X )
=
D − 1 A [ XW (1) ]
+ W (2) .
Then we have
[Equation]
Similarly, when the graph convolution is applied at the second layer of the three-layer MLP, the output is f (3) i ( X ) =
[ D − 1 A [ XW (1) ] + W (2) ]
+ W (3) , and we have f ( [Equation]
[Equation]
∈ [Equation] n [Equation]
−
[Equation] n [Equation] 
 [Equation]
i [Equation]
j ∈ [ n ] a ij h ( X j ) .
Finally, when the graph convolution is applied at the third layer of the three-layer MLP, the output is f (3) i ( X ) =
D − 1 A
[[ XW (1) ]
+ W (2) ]
+ W (3) , and we have
[Equation]
Therefore, in all cases where we have a single graph convolution, the output of the last layer is
[Equation]
where L ∈ { 2 , 3 } is the number of layers.
Theorem (Restatement of part one of Theorem 2) .
Let ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) .
Assume that p, q = Ω( log 2 n n ) , and it holds that Γ( p, q ) ζ ( γ / 2 σ )
= ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) , then for any c > 0 , with probability at least 1 − O ( n − c ) , the networks equipped with a graph convolution in the second or the third layer perfectly classify the data, and obtain the following loss:
[Equation]
where C > 0 and C (cid:48) ∈ [ 1 / 2 , 1] are constants and R is the constraint from Eq.
Proof.
First, we analyze the output conditioned on the adjacency matrix A .
Note that 1 R f ( L ) i ( X ) in
Lemma A.9 is Lipschitz with constant (cid:113) [Equation] i ) , and h ( X j ) are mutually independent for j ∈ [ n ] .
Therefore, by Gaussian concentration (Vershynin, 2018, Theorem 5.2.2) we have that for a ﬁxed i ∈ [ n ] , [Equation] − [Equation]
i
[Equation]
We refer to the event from Proposition A.1 as B and deﬁne Q ( t ) to be the event that
[Equation]
L
[Equation]
for all [Equation] ∈ [ n ] .
Then we can write
[Equation]
+ [Equation] n − c
[Equation]
Let ξ = sgn( p − q ) and note that ξ ( p − q ) p + q = | p − q | p + q = Γ( p, q ) .
We now choose t = 2 σ (cid:113) 2( c +1)log n n ( p + q ) to obtain that with probability at least 1 − 4 n − c , the following holds for all i ∈ [ n ] : σ [Equation] a [Equation]
j
[Equation]
Ca ij 
 [Equation] O (cid:32) R (cid:115) c log n n ( p + q ) (cid:33)
[Equation]
where in the last equation we used the assumption that Γ( p, q ) ζ ( γ / 2 σ )
= ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) .
Overall, we obtain that with probability at least 1 − 4 n − c ,
[Equation]
for all i ∈ [ n ] .
Recall that the loss for node i is given by
[Equation]
[Equation]
The total loss is given by 1 n (cid:80) i ∈
[ n ] (cid:96) ( i ) θ
( A , X ) .
Next, Fact A.5 implies that for any t < 0 , e t / 2 ≤ log(1 + e t ) ≤ e t , hence, we have for some C (cid:48) ∈
[ 1 / 2 , 1] that
[Equation]
A. P ROOF OF T HEOREM 2 PART TWO
We begin by computing the output of the networks constructed in Proposition A.7 when two graph convolutions are placed among any layer in the networks other than the ﬁrst.
Lemma A.10.
Let h ( x ) : R d → R = |
(cid:104) x , ˆ ν (cid:105) | − | (cid:104) x , ˆ µ (cid:105) | .
Consider the networks constructed in
Proposition A.7 equipped with two graph convolutions in the following combinations:
Both convolutions in the second layer of the two-layer network.
Both convolutions in the second layer of the three-layer network.
One convolution in the second layer and one in the third layer of the three-layer network.
Both convolutions in the third layer of the three-layer network.
Then for a sample ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) , the output of the networks in all the above described combinations for a point i ∈ [ n ] is
[Equation]
X )) = ϕ   R deg ( i ) (cid:88) j ∈ [ n ] τ ij h ( X j ) 
 , where τ ij = (cid:88) k ∈ [ n ] a ik a jk deg ( k ) .
Proof.
For the two-layer network, the output of the last layer when both convolutions are at the second layer is given by f (2) i ( X ) =
( D − 1 A ) 2 [ XW (1) ]
+ W (2) .
Then we have f (2) i ( X ) =
R deg ( i ) (cid:88) j ∈ [ n ] (cid:88) k ∈ [ n ] a ij a jk deg ( j ) h ( X k ) =
R deg ( i ) (cid:88) j ∈ [ n ] τ ij h ( X j ) .
Next, for the three-layer network, the output of the last layer when both convolutions are at the second layer is given by f (3) i ( X ) =
[( D − 1 A ) 2 [ XW (1) ]
+ W (2) ] + W (3) , hence, we have f (3) i ( X ) =
R deg ( i )     (cid:88)
j ∈ [ n ] a ij deg ( j ) (cid:88)
k ∈ [ n ] a jk h ( X k ) 
 + −   − (cid:88)
j ∈ [ n ] a ij deg ( j ) (cid:88)
k ∈ [ n ] a jk h ( X k ) 
 + 
 =
R deg ( i ) (cid:88) j ∈ [ n ] a ij deg ( j ) (cid:88) k ∈ [ n ] a jk h ( X k ) (using [ t ] + − [ − t ] + = t for any t ∈ R ) =
R deg ( i ) (cid:88) j ∈ [ n ] τ ij h ( X j ) .
Similarly, the output of the last layer when one convolution is at the second layer and the other one is at the third layer is given by f (3) i ( X )
=
D − 1 A
[ D − 1 A [ XW (1) ]
+ W (2) ] + W (3) , hence, we have f (3) i ( X ) =
R deg ( i ) (cid:88) j ∈ [ n ] a ij deg ( j )     (cid:88) k [Equation] [Equation] 

[Equation] 
 =
R deg ( i ) (cid:88) j ∈ [ n ] a ij deg ( j ) (cid:88) k ∈ [ n ] a jk h ( X k ) (using [ t ] + − [ − t ] + = t for any t ∈ R ) =
R deg ( i ) (cid:88) j ∈ [ n ] τ ij h ( X j ) .
Finally, the output of the last layer when both convolutions are at the third layer is given by f (3) i ( X ) =
[Equation]
D − 1 A ) 2 [[ XW (1) ] + W (2) ]
+ W (3) , hence, we have
[Equation]
Hence, the output for two graph convolutions is the same for any combination of the placement of convolutions, as long as no convolution is placed at the ﬁrst layer.
We are now ready to prove the positive result for two convolutions.
Theorem (Restatement of part two of Theorem 2) .
Let ( A , X ) ∼ XOR-CSBM ( n, d, µ , ν , σ 2 , p, q ) .
Assume that p, q = Ω( log n √ n ) and Γ( p, q ) 2 ζ ( γ / 2 σ ) = ω (cid:18)(cid:113) log nn (cid:19) .
Then for any c > 0 , with probability at least 1 − O ( n − c ) , the networks with any combination of two graph convolutions in the second and/or the third layers perfectly classify the data, and obtain the following loss:
[Equation]
c / log n ) (cid:17) , where C > 0 and C (cid:48) ∈
[ 1 / 2 , 1] are constants and R is the constraint from Eq.
Proof.
The proof strategy is similar to that of part one of the theorem.
Note that 1 R f ( L ) i ( X ) in
Lemma A.10 is Lipschitz with constant
[Equation]
Since h ( X j ) are mutually independent for j ∈ [ n ] , by Gaussian concentration (Vershynin, 2018,
Theorem 5.2.2)
we have that for a ﬁxed i ∈ [ n ] ,
[Equation]
We refer to the event from Proposition A.2 as B .
Note that since the graph density assumption is stronger than Ω( log 2 n n ) , Proposition A.1 trivially holds in this regime, hence, the degrees also concentrate strongly around ∆ =
n 2 ( p + q ) .
On event B , we have that (cid:88) j ∈
[Equation]
n
[Equation]
(1 ± o n (1))
=∆ (cid:88)
j ∼ i | N i ∩ N j |
[Equation]
+ (cid:88) j
(cid:28) i | N i ∩ N j | 
(1 ± o n (1))
=∆ (cid:88)
j ∼ i (cid:16) n 2 ( p 2 + q 2 ) (cid:17) 2 + (cid:88) j (cid:28) i ( npq )
 (1 ± o n (1)) (using Proposition A.2)
= n 2∆(cid:18) n
[Equation]
p
[Equation]
(cid:1)
[Equation] [Equation]
Therefore, under this event we have that
[Equation]
( X
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
Lip ≤ (cid:118)(cid:117)(cid:117)(cid:116) deg ( i )(cid:88)
j ∈ [ n ] τ 2 ij = (cid:115) 4( p 4 + q 4 + 6 p 2 q 2 ) n ( p + q )(1 ± o n (1)) .
Note that K = K ( p, q ) = 4( p 4 + q 4 +6 p 2 q 2 ) ( p + q )≤ 4 .
We now deﬁne Q ( t ) to be the event that | f
( L ) i
( X ) −
E [ f ( L ) i ( X )] | ≤ t for all i ∈ [ n ] .
Then we have
[Equation]
We now choose t = σ (cid:113) 2 K ( c +1)log n n to obtain that with probability at least 1 − 4 n − c , the following holds for all i ∈ [ n ] :
[Equation]
Note that we have
[Equation]
using Lemma A
[Equation]
In the last two equations above, we used Proposition A.1 to replace, respectively,
[Equation]
Therefore, we obtain that
[Equation]
where in the last equation we used Γ( p, q ) 2 ζ ( γ / 2 σ )
=
ω
[Equation]
Recall that the loss for node i is given by
[Equation]
=
log(1 + exp((1 − 2 ε i )
f ( L ) i ( X )))
= log (cid:0) 1 + exp (cid:0)
− CσRζ ( γ / 2 σ )Γ( p, q ) 2 (1 ± o n (1))
(cid:1)(cid:1) .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
A. A NALYSIS FOR A SIMPLER CASE
Although Theorem 2 encapsulates the general condition for networks with up to two graph convolutions to achieve perfect classiﬁcation let us discuss the meaning of the theorem in a simpliﬁed setting where Γ( p, q ) = Ω(1) .
In this regime one can analyze two cases for both parts of the theorem:
Case γ = Ω( σ ) :
Using part two of Lemma A.4 implies that ζ ( γ/σ ) =
Ω( γσ ) .
Hence, for one graph convolution, the condition Γ( p, q ) ζ ( γ / 2 σ ) = ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) is satisﬁed when γσ = ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) , implying that γ = ω (cid:16) σ (cid:113) log n n ( p + q ) (cid:17) .
Similarly, for two graph convolutions, the condition Γ( p, q ) 2 ζ ( γ / 2 σ ) = ω (cid:18)(cid:113) log nn (cid:19) is satisﬁed when γ = ω (cid:18) σ (cid:113) log nn (cid:19) .
Case γ =
o ( σ ) : Using part three of Lemma A.4 implies that ζ ( γ/σ ) =
Ω( γ 2 σ 2 ) .
Hence, for one graph convolution, the condition Γ( p, q ) ζ ( γ / 2 σ ) = ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) is satisﬁed when (cid:0) γ σ (cid:1) 2 = ω (cid:16)(cid:113) log n n ( p + q ) (cid:17) , implying that γ = ω (cid:16) σ 4 (cid:113) log n n ( p + q ) (cid:17) .
Similarly, for two graph convolutions, the condition Γ(
[Equation]
=
[Equation]
(cid:18)(cid:113) log nn (cid:19) is satisﬁed when γ = ω (cid:18) σ 4 (cid:113) log nn (cid:19) .
Combining both cases, we ﬁnd that the theorems imply perfect classiﬁcation if the following holds:
[Equation]
for networks wth one graph convolution ,
[Equation]
for networks with two graph convolutions.
B A DDITIONAL EXPERIMENTS
For all synthetic and real-data experiments, we used PyTorch Geometric (Fey & Lenssen, 2019), using public splits for the real datasets.
The models were trained on an Nvidia Titan Xp GPU, using the Adam optimizer with learning rate 10 − 3 , weight decay 10 − 5 , and 50 to 500 epochs varying among the datasets.
B. S YNTHETIC DATA
In this section we show additional results on the synthetic data.
First, we show that placing a graph convolution in the ﬁrst layer makes the classiﬁcation task difﬁcult since the means of the convolved data collapse towards 0 .
This is shown in Fig.
[Figure]
[Equation]
[Figure] [Figure]
T e s t
A cc u r a c y
[Figure]
Next
[Figure]
we show experiments for two sets of values of p <
q to demonstrate that graph convolutions also work in this setting.
In Figs.
5a and 5b we have Γ( p, q ) ≈ 0 .
82 , while in Figs.
5c and 5d we have a lower signal in the graph, Γ( p, q ) ≈ 0 . 66 .
We observe that in the latter case that there is less gap in the performance of networks with one graph convolution and those with two graph convolutions.
In comparison to Fig. 2, we observe similar trends about the performance of all the networks in different regimes of interest.
In particular, networks with one graph convolution perform mutually similarly, and networks with two graph convolutions perform mutually similarly, as claimed in Theorem
[Figure]
[Figure] [Figure]
[Figure] [Figure]
T e s t
A cc u r a c y
[Figure]
Finally, in Fig.
6, we show the trends for the accuracy of various networks with and without graph convolutions, for different values of Γ( p, q ) .
For cases where Γ( p, q ) is relatively larger, networks with graph convolutions perform much better than a standard MLP (see Figs.
6a to 6d), while for the cases where Γ( p, q ) is much smaller, the networks with graph convolutions degrade in performance (see Figs.
6e to 6h).
The intuition behind this behaviour is that a smaller value of Γ( p, q ) represents more noise in the data, thus, networks with graph convolutions gather roughly an equivalent amount of information from nodes in both the classes, making the feature representations noisy.
B. R EAL - WORLD DATA
This section contains additional experiments on real-world data.
In Fig.
7, we plot the accuracy of the networks measured on the three benchmark datasets, averaged across 50 different trials (random initialization of the network parameters).
This corresponds to the plots in Fig.
3 that show the maximum accuracy across all trials.
Next, we evaluate the performance of the original GCN normalization (Kipf & Welling, 2017), D − 12 AD − 12 instead of D − 1 A , and show that we observe the same trends about the number of convolutions and their placement.
These results are shown in Figs.and 9.
Note the two general trends that are consistent: ﬁrst, networks with two graph convolutions perform better than those with one graph convolution, and second, placing all graph convolutions in the ﬁrst layer yields worse accuracy as compared to networks where the convolutions are placed in deeper layers.
[Figure]
Similar to the results in the main paper, we observe that there are differences within the group of networks with the same number of convolutions, however, these differences are smaller in magnitude as compared to the difference between the two groups of networks, one with one graph convolution and the other two graph convolutions.
We also note that in some cases, three-layer networks obtain a worse accuracy, which we attribute to the fact that three layers have a lot more parameters, and thus may either be overﬁtting, or may not be converging for the number of epochs used.
Furthermore, we perform the same experiments on relatively larger datasets, OGBN-arXiv and
OGBN-products (Hu et al., 2020).
We observe similar trends in these experiments.
First, we observe
[Figure]
better and perform better than a single convolution.
Furthermore, three graph convolutions do not have a signiﬁcant advantage over two graph convolutions.
This observation agrees with Lemma A.3, where one can compute ρ (2) and ρ (3) and realize that they are of the same order in n , i.e., the variance reduction offered by two graph convolutions is of the same order as three graph convolutions for sufﬁciently dense graphs.
We present the results of these experiments in Fig.
[Figure]
[Figure] [Figure]