Understanding Non-linearity in Graph Neural Networks from the Bayesian-Inference Perspective
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Graph neural networks (GNNs) have shown superiority in many prediction tasksover graphs due to their impressive capability of capturing nonlinear relations ingraph-structured data.
However, for node classiﬁcation tasks, often, only marginalimprovement of GNNs over their linear counterparts has been observed.
Previousworks provide very few understandings of this phenomenon.
In this work, we resortto Bayesian learning to deeply investigate the functions of non-linearity in GNNsfor node classiﬁcation tasks.
Given a graph generated from the statistical model CSBM, we observe that the max-a-posterior estimation of a node label given itsown and neighbors’ attributes consists of two types of non-linearity, a possibly nonlinear transformation of node attributes and a ReLU-activated feature aggregationfrom neighbors.
The latter surprisingly matches the type of non-linearity used inmany GNN models.
By further imposing Gaussian assumption on node attributes,we prove that the superiority of those ReLU activations is only signiﬁcant whenthe node attributes are far more informative than the graph structure, which nicelymatches many previous empirical observations.
A similar argument can be achievedwhen there is a distribution shift of node attributes between the training and testingdatasets.
Finally, we verify our theory on both synthetic and real-world networks.
Introduction
Learning on graphs (LoG) has been widely used in the applications with graph-structured data [1,2].
Node classiﬁcation, as one of the most crucial tasks in LoG, asks to predict the labels of nodes ina graph, which has been used in many applications such as community detection [3–6], anomalydetection [7,8], biological pathway analysis [9,10] and so on.
Recently, graph neural networks (GNNs) have become the de-facto standard used in many LoG tasksdue to their super empirical performance [11,12].
Researchers often attribute such success to nonlinearity in GNNs which associates them with great expressive power [13,14].
GNNs can approximatea wide range of functions deﬁned over graphs [15–17] and thus excel in predicting, e.g., the freeenergies of molecules [18], which are by nature non-linear solutions of some quantum-mechanicalequations.
However, for node classiﬁcation tasks, many studies have shown that non-linearity tocontrol the exchange of features among neighbors seems not that crucial.
For example, many worksuse linear propagation of node attributes over graphs [19,20], and others recommend adding nonlinearity while only to the transformation of initial node attributes [21–23].
Both cases achievecomparable or even better performance than other models with complex nonlinear propagation, suchas using neighbor-attention mechanism [24].
Recently, even in the complicated heterophilic settingwhere nodes with same labels are not directly connected, linear propagation still achieves competitiveperformance [25], compared with the models with nonlinear and deep architectures [26,27].
Although empirical studies on GNNs are extensive till now and many practical observations asabove have been made, there have been very few works attempting to characterize GNNs in theory,especially to understand the effect of non-linearity by comparing with the linear counterparts fornode classiﬁcation tasks.
The only work on this topic to the best of our knowledge still focuseson comparing the expressive power of the two methods to distinguish nodes with different localstructures [28].
However, the achieved statement that non-linear propagation improves expressivenessmay not necessarily reveal the above phenomenon that non-linear and linear methods have closeempirical performance while with subtle difference.
Moreover, more expressiveness is often at thecost of model generalization and thus may not necessarily yield more accurate prediction [29,30].
In this work, we expect to give a more precise characterization of the values of non-linearity in
GNNs from a statistical perspective, based on Bayesian inference speciﬁcally.
We resort to contextualstochastic block models (CSBM) [31, 32].
We make a signiﬁcant observation that given a graphgenerated by CSBM, the max-a-posterior (MAP) estimation of a node label given its own andneighbors’ features surprisingly corresponds to a graph convolution layer with ReLU as the activationcombined with an initial node-attribute transformation.
Such a transformation of node attributes isgenerally nonlinear unless they are generated from the natural exponential family [33].
Since the
MAP estimator is known to be Bayesian optimal [34], the above observation means that ReLU-basedpropagation has the potential to outperform linear propagation.
To precisely characterize such beneﬁt,we further assume that the node attributes are generated from a label-conditioned Gaussian model,and analyze and compare the node mis-classiﬁcation errors of linear and nonlinear models.
We haveachieved the following conclusions (note that we only provide informal intuition here)
• When the node attributes are less informative compared to the structural information, non-linearpropagation and linear propagation have almost the same mis-classiﬁcation error (case I in Thm. 2).
• When the node attributes are more informative, non-linear propagation shows advantages.
Themis-classiﬁcation error of non-linear propagation can be signiﬁcantly smaller than that of linearpropagation with sufﬁciently informative node attributes (case II in Thm.
2).• When there is a distribution shift of the node attributes between the training and testing datasets,non-linearity provides better transferability in the regime of informative node attributes (Thm. 3).Given that practical node attributes are often not that informative, the advantages of non-linearpropagation over linear propagation for node classiﬁcation is limited albeit observable.
Our analysis
and conclusion apply to both homophilic and heterophilic settings, i e., when nodes with same labelstend to be connected (homophily) or disconnected (heterophily), respectively [25–27,35,36]
Extensive evaluation on both synthetic and real datasets demonstrates our theory.
Speciﬁcally, thenode mis-classiﬁcation errors of three citation networks with different levels of attributed information(Gaussian attributes) are shown in Fig.
1, which precisely matches the above conclusions.
1.1 More Related Works
GNNs have achieved great empirical success while theoretical understanding of GNNs, their nonlinearity in particular, is still limited.
There are many works studying the expressive power of
GNNs [16,37–47], while they often assumes arbitrarily complex non-linearity with limited quantitative results.
Only a few works provide characterizations on the needed width or depth of GNN layers [44–47].
More quantitative arguments on GNN analysis often depend on linear or Lipschitzcontinuous assumptions to enable graph spectral analysis, such as feature oversmoothing [48,49]and over-squashing [50,51], the failure to process heterophilic graphs [25,26,52] and the limitedspectral representation [53,54].
Some works also study the generalization bounds [55–57] and thestability of GNNs [58–61].
However, the obtained results may not reveal a direct comparison betweennon-linearity and linearity of the model, and their analytic techniques avoid tackling the speciﬁcforms of non-linear activations by using a Lipschitz continuous bound which is too loose in our case.
Stochastic block models (SBM) and its contextual counterparts have been widely used to study thenode classiﬁcation problems [31, 32, 62–66], while these studies focus on the fundamental limits.
Recently, (C)SBM and its large-graph limitation also have been used to study the transferrabilityand expressive power of GNN models [67–69] and GNNs on line graphs [5], while these works didnot compare non-linear and linear propagation.
CSBM has also been used to show the advantage oflinear convolution over no convolution for node classiﬁcation [70].
A very recent result shows thatattention-based propagation [24] may be much worse than linear propagation given low-quality nodeattributes under CSBM [71].
Our results imply that ReLU is the de facto optimal non-linearity insteadof attention and may at most marginally outperform the linear model when with low-quality nodeattributes.
Some previous works also use Bayesian inference to inspire GNN architectures [72–79],while these works focus on empirical evaluation instead of theoretical analysis.
Preliminaries
In this section, we introduce preliminaries and notations for our later discussion.
Maximum-a-posteriori (MAP) estimation.
Suppose there are a set of ﬁnite classes C .
A class label
Y 2 C is generated with probability ⇡ Y , where
P
Y 2 C ⇡ Y = 1 .
Given Y , the corresponding feature
X in the space X is generated from the distribution X ⇠ P Y .
A classiﬁer is a decision f : X !
C andthe Bayesian mis-classiﬁcation error can be characterized as ⇠ ( f ) =
P
Y 2 C ⇡ Y
R 1 f ( X ) 6 = Y P Y ( X ) ,where and later 1 S indicates 1 if S is true and 0 otherwise.
The MAP estimation of Y given X isthe classiﬁer
f ⇤ ( X ) , arg max Y 2 C ⇡ Y P Y
( X ) that can minimize ⇠ ( f ) [34].
Later, we denote theminimal Bayesian mis-classiﬁcation error ⇠ ( f ) as ⇠ ⇤ =
⇠ ( f ⇤ ) .
Signal-to-Noise Ratio (SNR).
Detection of a signal from the background essentially corresponds toa binary classiﬁcation problem.
SNR is widely used to measure the potential detection performancebefore specifying the classiﬁer [80].
In particular, if we have two equiprobable classes C = {  1 , 1 }and the features follows 1-d Gaussian distributions P Y =
N ( µ Y ,  2 ) , Y 2 C .
The SNR ⇢ deﬁned asfollows precisely characterizes the minimal Bayesian mis-classiﬁcation error.
[Equation]
In this case, the MAP estimation f ⇤ ( X ) =
⇤
| X  µ 1 |  | X  µ  1 |  1 and the minimal Bayesian mis- 108 classiﬁcation error is  ( p ⇢ / 2) where  denotes the accumulative standard Gaussian distribution.
For more general cases where the two classes are associated with sub-Guassian distributions P Y , s.t.
P Y ( | X | > t ) 2 [ c 1 exp(  c 2 t 2 ) , C 1 exp(  C 2 t 2 )] , for some non-negative constants c 1 , c 2 , C 1 , C 2 ,a similar connection between ⇠ ⇤ and ⇢ can be shown by leveraging sharp sub-Gaussian lowerbounds [81].
We will specify the connection to SNR in our case in Sec. 4 and the SNR ⇢ will be usedas the main bridge to compare the mis-classiﬁcation errors of non-linear v.s. linear models.
Contextual Stochastic Block Model (CSBM).
Random graph models have been widely used tostudy the performance of algorithms on graphs [82,83].
For node classiﬁcation problems, CSBM isoften used [67–69], as it well combines the models of network structure and node attributes.
We study the case that nodes are in two equi-probable classes C
= {  1 , 1 } , where ⇡ Y = 1 2 , Y 2 C .
Our analysis can be generalized.
An attributed network G = ( V , E , X ) is sampled from CSBM withparameters ( n, p, q, P 1 , P  1 ) as follows.
Suppose there are n nodes, V = [ n ] .
For each node v , thelabel Y v 2 C is sampled from Rademacher distribution.
Given Y v , the node attribute X v is sampledfrom P Y v .
For two nodes u, v , if Y u
= Y v , there is an edge e uv 2 E connecting them with probability p .
If Y u
= Y v , there is an edge e uv 2 E connecting them with probability q .
All node attributes Xand edges E are independent given the node labels Y =
{ Y v | v 2 V} .
Note that p > q indicates the nodes with the same labels tend to be directly connected, whichcorresponds to the homophilic case, while p < q corresponds to the heterophilic case.
The gap | p  q | reﬂects structural information and the gap between P 1 , P  1 reﬂects attributedinformation , e.g., Jensen-Shannon distance JS ( P 1 , P  1 ) that is well connected to Bayesian misclassiﬁcation error [84].
Graph learning allows combining these two types of information.
In Sec.
4,we give more speciﬁc deﬁnitions of these two types of information and their regime for our analysis.
Bayesian Inference and Nonlinearity in Graph Neural Networks
In the previous section, we discuss that given conditioned feature distributions X ⇠ P Y , Y 2 C , the
MAP estimation f ⇤ ( X ) can minimize mis-classiﬁcation error.
For node classiﬁcation in an attributednetwork, the estimation of a node label should depend on not only one’s own attributes but alsoits neighbors’.
For example, in a homophilic network, nodes with same labels tend to be directlyconnected.
Intuitively, using the averaged neighbor attributes may provide better estimation of thelabel, which gives us graph convolution.
In a heterophilic network, nodes with different labels tendto be directed connected.
So, intuitively, checking the difference between one’s attributes and theneighbors’ may provide better estimation.
However, what could be the optimal form to combineone’s own attribute with the neighbors’ attributes?
We resort to the MAP estimation.
That is, giventhe attributes of a node v 2 V and its neighbors N v , we consider the MAP estimation as follows.
[Equation]
where ⇡ Y v , { Y u } u 2 N v denotes their prior distributions of node labels.
Note that here we simplify theproblem and consider only 1-hop neighbors by following the setting [70].
In practice, most GNNmodels can only work on local networks due to the scalability constraints [11,85,86].
Even with theabove simpliﬁcation, the above MAP estimation is generally intractable.
Therefore, we consider the CSBM with parameters ( n, p, q, P 1 , P  1 ) .
In this case, the prior distribution follows ⇡
Y v , { Y u } u 2 N v = 2  |N v |  1 , which is a constant given N v .
The rest term follows
[Equation]
Therefore, the MAP estimation f ⇤ ( X v , { X u } u 2 N v ) is to solve
[Equation]
This can be solved via the max-product algorithm [87].
To establish the connection to GNNs, werewrite the RHS of Eq. 3 in the logarithmic form and use the fact that C = {  1 , 1 } .
And, we achievef ⇤ ( X
[Equation] [Equation]
=
[Equation]
where
[Equation]
We leave the derivation in Appendix B. Amazingly, activation ReLUs in the message M well connectto the activations commonly-used in GNN models, e.g., graph convolution networks [12].
Given theoptimality of the MAP estimation, we summarize this observation in Proposition 1.
Proposition 1 ( Optimal Nonlinear Propagation ) .
Consider a network G ⇠ CSBM ( n, p, q, P 1 , P  1 ) .
To classify a node v , the optimal nonlinear propagation (derived by the MAP estimation) given theattributes of v and its neighbors follows:
[Equation]
4) where ( a ; P 1 , P  1 ) =
log P 1 ( a ) P  1 ( a ) and  ( a ; log pq ) =
ReLU ( a +log pq )  ReLU ( a  log pq )  log pq .
|log |pq
X
Y
Homophilic
Heterophilic (p>q) (p<q) logpq logpq
The optimal nonlinear propagation in Eq.
(4) may contain two typesof non-linear functions: (1) is to measure the likelihood ratiobetween two classes given the node attributes; (2)  is to propagatethe likelihood ratios of the neighbors.
ReLUs in  avoid the overuseof the likelihood ratios from neighbors, as  essentially provides abounded function (See Fig. 2).
One observation of the direct beneﬁtof this non-linear propagation is as follows.
Remark 1.
When there is no structural information, i.e., p = q ,  ( x ; 0) = 0 , 8 x 2 R , thepropagation is deactivated, which avoids potential contamination from the attributes of the neighbors.
In the equiprobable case, the MAP estimation also gives the maximum likelihood estimation (MLE)of Y if we view the labels as the ﬁxed parameters.
When the classes are unbalanced ⇡ Y 6 = 12 , similarresults can be obtained while additional terms log ⇡ 1 ⇡  1 may appear as bias in Eq.
Later, ouranalysis focuses on the equiprobable case while empirical results in Sec.
5 show more general cases.
Moreover, if one is to infer the posterior distribution of Y , one may replace the max-product algorithmto solve Eq.
(3) with the sum-product algorithm [88].
Then, the obtained non-linearity in  will turninto Tanh functions.
As ReLUs are more used in practical GNNs, we focus on the case with ReLUs.
Discussion on the Non-linearity.
Next, we discuss more insights into the non-linearity of and  .
The function essentially corresponds to a node-attribute transformation , which depends on thedistributions P ± 1 .
As these distributions are unknown in practice, a NN model to learn is suggested,such as the one in the model APPNP [21].
Due to the expressivity of NNs [89,90], a wide range ofcan be modeled.
One insightful example is that when P ± 1 are Laplace distributions, is a boundedfunction (same as  ) to control the heavy-tailed attributes generated by Laplace distributions.
Example 1 (Laplace Assumption) .
When node attributes follow m -dimensional independent Laplacedistribution, i.e., P Y v = 1 (2 b ) m exp( k X v  Y v µ k 1 /b ) for µ 2 R m , b > 0 and Y v 2 {  1 , 1 } .
According to Eq.
(4) , the function ( · ; P 1 , P  1 ) can be speciﬁed aslap ( X v ; P 1 , P  1 )
=
T  ( X v ; 2 µ/b ) , where  as deﬁned in Eq.
(4) works in an entry-wise manner.
As node-attribute distributions may vary a lot, is better to be modeled via a general NN in practice.
More interesting ﬁndings may come from  in Eq.
(4) as it has a ﬁxed form and well matches themost commonly-used GNN architecture.
Speciﬁcally, besides the extreme case stated in Remark 1,we are to investigate how non-linearity induced by the ReLUs in  may beneﬁt the model.
We expectthe ﬁndings to provide the explanations to some previous empirical experiences on using GNNs.
To simplify our discussion, when analyzing  , we focus on the case with a linear node-attributetransformation =
Gau in Eq.
(6) by assuming label-dependent Gaussian node attributes.
Thisfollows the assumptions in previous studies [70,71].
In fact, there are a class of distributions namednatural exponential family (NEF)
[33] that hold linear .
We conjecture that our results are appliedto the general NEF since the only difference is the bias term by comparing Eq.
(5) and Eq.
Example 2 (Natural Exponential Family Assumption) .
When node features follow m -dimensionalnatural exponential family distributions P Y v ( X )
=
h ( X v ) · exp( ✓ TY v X v  M ( ✓ Y v )) for ✓ Y v 2 R mand Y v 2 {  1 , 1 } where M ( ✓ Y v ) is a parameter function.
The function ( · ; P 1 , P  1 ) is speciﬁed as:
[Equation]
In particular, when P 1 = N ( µ, I/m ) , P  1 = N ( ⌫ , I/m ) for µ, ⌫ 2 R m ,
[Equation]
[Equation]
We summarize our main theoretical ﬁndings regarding the nonlinearity of  in the next section.
Main Results on ReLU-based Nonlinear Propagation
In this section, we summarize our analytical results on  in the optimal nonlinear propagation (Eq.
Our study assumes an attributed network generated from CSBM ( n, p, q, N ( µ, I/m ) , N ( ⌫ , I/m ))
where µ, ⌫ 2 R m .
We use CSBM-G ( n, p, q, µ, ⌫ ) later to denote this model for simplicity.
Weare interested in the asymptotic behavior when n ! 1 .
Note that all parameters µ, ⌫ , p, q, m mayimplicitly depend on n .
We are to compare the non-linear propagation model P v suggested by Eq.
(4)where =
Gau with the following linear counterpart P lv .
Baseline linear
[Equation] [Equation]
where w 2 R is an extra parameter to be tuned.
Note that this linear model can be claimed as anoptimal linear model up-to a choice of w because the distributions of both the center node attribute
X v and the linear aggregation from the neighbors
P u 2 N v X u are Gaussian and symmetric w.r.t.
thehyperplane { Z 2 R m | ( µ  ⌫ ) T Z
=
( k µ k 22  k ⌫ k 22 ) / 2 } for the two classes.
We are to comparetheir classiﬁcation errors ⇠ r = ⇠ ( sgn ( P v )) and ⇠ l ( w ) =
⇠ ( sgn ( P lv ( w ))) .
By following [70], we alsodiscuss separability of all nodes in the network, i.e., P ( 8 v 2 V , P v · Y v > 0) in Theorem 1.
To begin with, we introduce several quantities for the convenience of further statements.
The SNRs
[Equation]
are important quantities to later characterize different types of propagation.
Also, we characterizestructural information by S ( p, q ) =
( p  q ) 2 / ( p + q ) and attributed information by p m k µ  ⌫ k 2 .
Assumption 1 (Moderate Structural Information) .
S ( p, q ) = !
n ( (log n ) 2 n ) and S ( p,q ) | p
q | 9 1 .
Assumption 2 (Bounded Attributed Information) .
p m k µ  ⌫ k 2 =
o n (log n ) .
Assumption 1 states that structural information should be neither too weak nor too strong.
S ( p, q ) =!
n ( (log n ) 2 n ) excludes the extremely weak case discussed in Remark 1.
Moreover, the graph structureshould not be too sparse, so the aggregated information from neighbors dominates the propagation.
S ( p,q ) | p  q | 9 1 means neither p = !
n ( q ) nor q = !
n ( p ) , which avoids extremely strong structuralinformation.
This assumption is more general than some concurrent works on CSBM-based GNNanalysis [70,71] as we includes the case with less structural information | p  q |
= o n ( p + q ) andheterophily p < q .
Assumption 2 is to avoid too strong attributed information: when p m k µ  ⌫ k 2 =⌦ n (log n ) , all nodes in CSBM can be accurately classiﬁed in the asymptotic sense without structuralinformation, i.e. P ( 8 v 2 V , Gau ( X v ; µ, ⌫ ) · Y v > 0) =
1  o n (1) .
Now, we present our ﬁrst lemmawhich links the mis-classiﬁcation errors ⇠ r , ⇠ l to with the SNRs ⇢ r , ⇢ l :
Lemma 1.
Suppose ( p, q ) satisﬁes Assumption 1, for any G ⇠ CSBM-G ( n, p, q, µ, ⌫ ) ,
[Equation]
where C 1 is asymptotically a constant, and the notation a ( n ) !
b ( n ) denotes a ( n ) /b
( n ) !
Lemma 1 claims that the classiﬁcation errors under both nonlinear and linear model can be controlledby their SNRs.
By leveraging Lemma 1, we can further illustrate the separability of all nodes in thenetwork, which is presented in the following theorem.
Theorem 1 (Separability) .
Suppose that ( p, q ) satisﬁes Assumption 1, for G ⇠ CSBM-G ( n, p, q, µ, ⌫ ) ,if p m k µ  ⌫ k 2 = !
n ( p log n/S ( p, q ) n ) , then
[Equation]
Here, assume | w | > c for some positive constant c and sgn ( w ) =
sgn ( p  q ) in the linear model.
Theorem 1 applies to both homophilic ( p > q ) and heterophilic ( p < q ) scenarios.
Even for justthe linear case, compared to [70] which needs p m k µ  ⌫ k 2 = !
n (log n/ p
S ( p, q ) n ) to achieveseparability, we have p log n improvement due to a tight analysis.
As shown in Lemma 1 and Theorem 1, the errors are mainly determined by SNRs.
Large SNRimplies a fast decay rate of the errors of a single node and the entire graph, which motivates us tofurther explore SNRs to illustrate a comparison between non-linear and linear models.
We considercomparing with the optimal linear model, i.e., ⇢ ⇤ l = ⇢ l ( w ⇤ ) , where w ⇤ = arg min w 2 R ⇠ l ( w ) .
Theorem 2.
Suppose that ( p, q ) satisﬁes Assumption 1, for G ⇠ CSBM-G ( n, p, q, µ, ⌫ ) , under theseparable condition in Theorem 1 p m k µ  ⌫ k 2 = !
n ( p log n/S ( p, q ) n ) , we further have
• I. Limited Attributed Information : When p m k µ  ⌫ k 2 =
O n
[Equation] [Equation]
Further, if p m k µ  ⌫ k 2 = o n ( | log( p/q ) | ) , ⇢ r / ⇢ ⇤ l ! 1 ;
• II.
Sufﬁcient Attributed Information : When p m k µ  ⌫ k 2 = !
n (1) and satisﬁes Assumption 2,
[Equation]
Theorem 2 also works both homophilic ( p > q ) and heterophilic ( p < q ) scenarios.
Theorem 2implies that when attributed information is limited, nonlinear propagation behaves similar to thelinear model as their SNRs are in the same order.
Particularly, when attributed information is verylimited, p m k µ  ⌫ k 2 =
o n ( | log( p/q ) | ) , the SNRs of two models are asymptotically the same.
Inthe regime of sufﬁcient attributed information, nonlinear propagation brings order-level superioritycompared with the linear model.
The intuition is that in this regime, when the attributes are veryinformative, the bounds of  in Eq.
(4) help with avoiding overconﬁdence given by the node attributes.
The coefﬁcient before ⇢ ⇤ l in Eq.
(12) shows the trade-off between structural information and attributedinformation on controlling the superiority of nonlinear propagation.
Next, we analyze whether nonlinearity makes model more transferable or not when there often existsa distribution shift between the training and testing datasets, which is also practically useful.
We consider the following setting.
We assume using a large enough network generated by CSBMG ( n, p, q, µ, ⌫ ) for training so that the optimal parameters as in P v and P lv ( w ⇤ ) for this CSBM-Ghave been learnt.
We consider their mis-classiﬁcation errors over another CSBM-G with parameters( n, p 0 , q 0 , µ 0 , ⌫ 0 ) .
We keep the amounts of attributed information and structural information unchangedby setting p = p 0 , q = q 0 , µ 0 = ( µ + ⌫ ) / 2 + R ( µ  ⌫ ) / 2 , ⌫ 0 = µ + ⌫ / 2 + R ( ⌫  µ ) / 2 for a rotationmatrix R close to I .
Let  ⇠ r and  ⇠ l ( w ⇤ ) denote the increase of mis-classiﬁcation errors of models
P v and P lv ( w ⇤ ) , respectively, due to such a distribution shift.
We may achieve the following results.
Theorem 3 (Transferability) .
Suppose that ( p, q ) satisﬁes Assumption 1, for G 0 ⇠ CSBMG ( n, p 0 , q 0 , µ 0 , ⌫ 0 ) , under the linear separable condition p m k µ 0  ⌫ 0 k 2 = !
n ( p log n/S ( p 0 , q 0 ) n ) .
Suppose P v and P lv ( w ⇤ ) have learnt parameters from G ⇠ CSBM-G ( n, p, q, µ, ⌫ ) where the parameters of two CSBM-Gs follow the relation described above.
Then, we have
• I. Limited Attributed Information : When p m k µ  ⌫ k 2 =
o n ( | log( p/q ) | ) ,  ⇠ r /  ⇠ l ( w ⇤ ) !
1 .• II.
Sufﬁcient Attributed Information : When p m k µ  ⌫ k 2 = !
n (1) and and satisﬁes Assumption 2,  ⇠ r /  ⇠ l ( w ⇤ ) !
Similar to Theorem 2, when attributed information is very limited, nonlinearity will not bring anybeneﬁt, while in the regime with informative attributes, nonlinearity increases model transferability.
We leave the intermediate regime p m k µ  ⌫ k 2
[ ⌦ n ( | log( p/q ) | ) , O n (1)] for future study.
Experiments
In this section, we verify our theoretical results based on synthetic and real datasets.
In all experiments,we ﬁx w in the linear model (Eq. (7)) as w = 1 for the homophilic case ( p > q ) and w =  1 for theheterophilic case ( p < q ) .
Experiments on other w ’s can be found in Appendix H.1.1, which doesnot change the achieved conclusion.
This is because when the node number n is large, for a constantw , the neighbor information will dominate the results.
Later, we use P lv
= P
lv ( w ) for simplicity.
5.1 Asymptotic Experiments - Model Accuracy & Transferability Study
Our ﬁrst experiments focus on evaluating the asymptotic ( n ! 1 ) classiﬁcation performance ofnonlinear and linear models.
Given a CSBM-G, we generate 5 graphs and compute the averageaccuracy results (#correctly classiﬁed nodes / #total nodes).
We compare the nonlinear v.s. linearmodels under three different CSBM-G settings.
Fig.
3 shows the results.
All three cases satisfy the separability condition in Theorem 1, so, as n increases, the accuracyprogressively increases to 1.
Our results also match well with the implications provided by Theorem 2.
In the regime with limited attribute information (Fig. 1 LEFT) where ⇢ r = ⇥ n ( ⇢ ⇤ l ) as proved, thenonlinear model and the linear model behave almost the same (performance gap < 0 .
15% forn  10 5 ).
In the regime with sufﬁcient attribute information (Fig. 1 MIDDLE) where ⇢ r = !
n ( ⇢ ⇤ l )as proved, we may observe that the nonlinear model can signiﬁcantly outperform the linear modelas n ! 1 .
Fig.
1 RIGHT is to show the heterophilic graph case ( p < q ).
If we switch the valuesof p, q (and also change the models correspondingly), we obtain the exactly same ﬁgure up to someexperimental randomness (see Appendix H.1.2).
Also, Fig.
1 RIGHT considers a boundary case ofsufﬁcient attributed information, i.e., p m k µ  ⌫ k 2 = ⇥ n (1) .
We observe that Theorem 2 still welldescribes the asymptotic performance when n ! 1 .
We further study the transferability for the non-linear model and the linear model.
We follow thesetting in Theorem 3 by rotating µ, ⌫ !
µ 0 , ⌫ 0 .
Fig.
4 shows the result and well matches Theorem 3.
In the regime of limited attributed information, the two models have the almost same transferability,i.e., the perturbation error ratio is close to 1.
In contrast, with sufﬁcient attributed information, thenon-linear model is more transferrable than the linear counterpart as the ratio is smaller than 1.
5.2 Transition Curve
Our second experiment studies the tradeoff between attributed information and structural information.
302 We ﬁx the graph size n =⇥ 10 4 and get the averaged classiﬁcation accuracy based on 5 generatedgraphs.
For the homophilic case, we test different levels of attributed information ( k µ  ⌫ k from10  4 to 10 with m = 10 ) and structural information (ﬁxing q = 5 ⇥ 10  3 and increasing p fromp = q to 1 ).
The intermediate testing points are sampled in log scales.
Fig.
5 LEFT shows the results.
When structural information is limited and attributed information is sufﬁcient, the non-linear modelshows signiﬁcant advantage over the linear model while for most other parameter settings, these twomodels share similar performance.
Fig.
5 RIGHT shows the heterophilic case, where we observe asimilar pattern.
In the heterophilic case, we ﬁxing p = 5 ⇥ 10  3 and increasing q from q = p to 1 .
5.3 Real-world Network Experiments
This experiments compare non-linear models and linear models under Gaussian and Laplacianattributes on three benchmark citation networks PubMed, Cora, and CiteSeer [91].
In these threenetworks, nodes denote papers and edges denote the citation relationships between the papers.
Thestatistics ( # nodes, # edges, # classes) of these three networks are: PubMed (19,717, 44,338, 3);
Cora (2,708, 5,428, 7); CiteSeer (3,327, 4,732, 6). k  ⌫
[Table]
[Figure]
Experimental Settings.
We carry out one-v.s.-all and several-v s.-several classiﬁcation tasks.
Afternodes are put into two classes, we generate two graphs independently with attributes according to
Gaussian (or Laplace) distributions.
One graph is used for training and the other one for testing.
Forthe Gaussian case, we use a nonlinear model by following Eq.
4 with =
Gau while the parameterssuch as log( p/q ) , µ  ⌫ and other biases need to be learned.
For the Laplacian case, we consider threenonlinear models by following the form of (1) full Eq.
4 with = lap ; (b) only nonlinear attributetransformation = lap
(c) only nonlinear propagation  with linear attribute transformation.
Later, we call them nonlinear models (a), (b), (c), respectively.
Similar to the Gaussian case, all theparameters in these functions are obtained by training.
The model is trained with Adam optimizer(learning rate = 1 e  2 , weight decay = 5 e  4 ).
We give other details to Appendix H.2.1.
Result Analysis.
We report the averaged results over 5 trials in Fig.
1 (Gaussian) and Fig.
(Laplacian).
Due to the space limit, we leave the results for the several-v.s.-several case in Appendix H.2.1.
The Gaussian case well matches our theory.
Only when the node features are very informative, thegaps between the nonlinear model and the linear model become signiﬁcant.
This is true for all threenetworks.
The Laplacian case is more complicated.
Non-linear model (a) marginally outperforms
Non-linear model (b).
Non-linear model (c) marginally outperforms the linear model.
This indicatesthat the non-linear attribute transformation lap is very crucial while the nonlinear propagation  isonly slightly useful, which resonates the spirits of APPNP [21] and GPRGNN [25] who only applynonlinearity to node attributes.
Moreover, our theory on the function of nonlinear  seems applicableeven to the Laplacian case.
The difference between nonlinear model (a) and nonlinear model (b) ismore obvious though still marginal when the node attributes become informative.
Conclusion
This work uses Bayesian methods to investigate the function of non-linearity in GNNs.
Given agraph generated from CSBM, we observe the optimal non-linearity to estimate a node label givenits own and neighbors’ attributes is in twofold: attribute non-linear transformation and non-linearpropagation.
We further investigate the non-linear propagation by imposing Gaussian assumptions onnode attributes.
We prove that non-linear propagation shares a similar performance (with or withoutdistribution shift) with linear propagation in most cases except when node attributes become veryinformative.
These ﬁndings explain many previous empirical observations in this domain.