Not too little, not too much: a theoretical analysis of graph (over)smoothing
Abstract
We analyze graph smoothing with mean aggregation , where each node successively receives the average of the features of its neighbors.
Indeed, it has been observed that Graph Neural Networks (GNNs), which generally follow some variant of
Message-Passing (MP) with repeated aggregation, may be subject to the oversmoothing phenomenon: by performing too many rounds of MP, the node features tend to converge to a non-informative limit.
At the other end of the spectrum, it is intuitively obvious that some MP rounds are necessary, but existing analyses do not exhibit both phenomena at once.
In this paper, we consider simpliﬁed linear GNNs, and rigorously analyze two examples of random graphs for which a ﬁnite number of mean aggregation steps provably improves the learning performance, before oversmoothing kicks in.
We identify two key phenomena: graph smoothing shrinks non-principal directions in the data faster than principal ones, which is useful for regression, and shrinks nodes within communities faster than they collapse together, which improves classiﬁcation.
Introduction
In recent years, deep architectures such as Graph Neural Networks (GNNs), along with the availability of large sets of graph data, have signiﬁcantly broadened the ﬁeld of machine learning on graphs and structured data, see [2, 3, 8, 19] for reviews.
Most GNNs rely on the Message-Passing (MP) framework [7, 11].
At each layer k , for each node i , a representation z ( k ) i is computed using the representations of the neighbors N i of i in the graph at the previous layer:
z ( k ) i
[Equation]
where AGG is an aggregation function that treats { z ( k − 1) j } j ∈N i as an unordered set , to respect the absence of node ordering in the graph.
Here we consider one of the most classical, mean aggregation :
[Equation]
[Table]
[Equation]
where the a ij ∈ R + are the entries of the adjacency matrix of the graph, and Ψ is some function (usually a Multi-Layer Perceptron).While MP is a natural and rather general framework, its limitations were quickly observed by researchers and practitioners.
Foremost among them is the so-called oversmoothing phenomenon [14]: as the GNN gets deeper and many rounds of MP are performed, the node features z ( k ) i tend to become too similar across the graph.
To relieve it, researchers have explored residual mechanisms [6, 13], dropping connections [9], clever normalizations [21] or regularizations [5], among others.
Some works have acknowledged the important role of the aggregation function, and proposed new exotic diffusion strategies [1] or to optimize it [12].
On the theoretical side, oversmoothing has mostly been analyzed in the inﬁnite-layer limit k → ∞ .
In this case, classical spectral analysis of graph operators such as the Laplacian can be leveraged to indeed show that node features will always converge to some limit that carries a limited amount of information [17].
This is particularly true for mean aggregation (1).
However, there has been little research at the other end of the spectrum.
Generally, researchers show the power of GNNs for a sufﬁcient (unbounded) number of layers, such as the now-famous ability to distinguish graph isomorphism as well as the Weisfeiler-Lehman test and all its variants [16, 20].
Since these results are valid for an unbounded number of layers, the settings adopted in these works are, by deﬁnition, incompatible with non-informative oversmoothing.
In a recent preprint [10] 1 , we showcase two representative exemples, of regression and classiﬁcation, on which linear GNNs (sometimes called SGC [18]) are provably subject to this double phenomenon: some smoothing is useful for learning, while too much smoothing inevitably leads to oversmoothing.
We adopt on a model of latent space random graphs, with node features.
We identify two key phenomena for this: smoothing shrinks non-principal directions in the data faster than principal ones (Sec. 4), and shrinks communities faster than they collapse together (Sec. 5).
Although our theoretical settings are obviously simpliﬁed, we believe it is a step towards a better comprehension of graph aggregation, of the relationship between node features and graph structure.
All proofs are given in the full paper [10], of which the present document is an extended abstract.
Preliminaries
Notations.
The norm (cid:107)·(cid:107) is the Euclidean norm for vectors and spectral norm for (rectangular) matrices.
For a psd matrix Σ , the Mahalanobis norm is (cid:107) x (cid:107) 2Σ def.
= x (cid:62) Σ x .
The determinant of a matrix is | S | , and its smallest eigenvalue is λ min ( S ) .
The multivariate Gaussian distribution with mean µ and covariance Σ is denoted by N µ, Σ ( x ) = det (2 π Σ) − 12 e − 12 (cid:107) x − µ (cid:107) 2Σ − 1 .
SSL..
In this paper, we consider Semi-Supervised Learning (SSL) [4, 11] on an undirected graph of size n .
We observe a weighted adjacency matrix A
= [ a ij ] ni,j =1 ∈ R n × n + as well as node features z 1 , . . .
z n ∈ R p at each node of the graph.
We also observe some labels y 1 , . . .
, y n tr ∈ R at training time and aim to predict the remaining labels y n tr +1 , . . .
, y n .
For simplicity, we assume that n tr and n te are both in O ( n ) .
We denote by Z ∈ R n × p the matrix whose rows contain the node features,
Z tr , Z te respectively its ﬁrst n tr and last n te rows, and similarly Y tr , Y te the vectors containing the observed and non-observed labels.
Graph smoothing with mean aggregation.
Here we consider a simpliﬁed situation of linear GNN with mean aggregation, often used as a theoretical baseline [18].
A linear GNN with k layers just corresponds to performing k rounds of mean aggregation on the node features, then learning on the smoothed features.
We denote by d A =
[ (cid:80) i a ij ] j ∈ R n + the vector containing the degrees of the graph and D = diag( d A ) .
Assuming that all degrees are non-zero, performing one round of mean aggregation corresponds to multiplying Z by L = D − 1 A .
The smoothed node features after k rounds of mean aggregation are: Z ( k ) =
L k Z .
Each row, denoted by z ( k ) i ∈
R p , contains the smoothed features of an individual node.
Its ﬁrst n tr and last n te rows are denoted Z ( k ) tr , Z ( k ) te .
Learning.
In this paper, we consider learning with a Mean Square Error (MSE) loss and Ridge regularization.
For λ > 0 , the regression coefﬁcients vector on the smoothed features is ˆ
[Equation]
Then, the test risk is deﬁned as
[Equation]
where ˆ Y te ( k ) =
Z ( k
[Equation]
te
[Equation]
( k
[Equation] [Equation]
Our goal is to illustrate some situations where a ﬁnite amount of smoothing provably improves the test risk, that is, there is an optimal k (cid:63) > 0 such that R ( k (cid:63) ) <
min( R (0) , R ( ∞ ) ) .
Random graph model.
We adopt popular latent space random graph models akin to graphons [15].
In these models, to each node i is associated an unobserved latent variable x i ∈ R d with d (cid:62) p , and edge weights are assumed to be equal to a ij =
W ( x i , x j ) where W : R d × R d
→ R + is a connectivity kernel .
Note that edges may also be taken as random Bernoulli variables , but we do not consider this here for simplicity.
Moreover, we consider that the ( x i , y i ) are drawn iid from some joint distribution, and the node features are a linear projection of the latent variables to a lower dimension: z i = M (cid:62) x
i for some unknown M ∈ R d × p that satisﬁes M (cid:62) M = Id p .
To summarize:
[Equation]
For this model, note that Z ( k ) =
X ( k ) M where X ( k )
=
L k X .
In the rest of the paper, we use the
Gaussian kernel with a small additive term ε > 0 :
[Equation]
The coefﬁcient ε is added to lower-bound the degrees of the graph and avoid degenerate situations.
Oversmoothing
In this section, we brieﬂy examine the oversmoothing case, when k → ∞ while all other parameters are ﬁxed.
In this case, it is well-known that all node features converge even for general GNNs [17].
For completeness, we state below this result in our settings.
We denote by ¯ d = d
A /d (cid:62)
A 1 n the vector of normalized degrees, which is also the limit distribution of a random walk on the graph.
Theorem 1.
Deﬁne v =
Z (cid:62) ¯ d and ¯ y tr = n [Equation] n [Equation]
i
[Equation]
y i .
We have ˆ Y
[Equation]
Hence, in the limit k → ∞ , the predicted labels become all equal.
Using simple concentration inequalities, it is generally easy to show that R ( ∞ )
≈ Var ( y )
+ O (1 / √ n ) when λ → 0 .
In most cases, this leads to situations where R (0) <
R ( ∞ ) , and oversmoothing occurs.
Finite smoothing: Linear Regression
In this section, we consider a problem of linear regression on Gaussian data.
We consider x ∼ N 0 , Σ for some positive deﬁnite covariance matrix Σ , and y =
x (cid:62) β (cid:63) , without noise for simplicity.
For a symmetric positive semi-deﬁnite matrix S ∈ R d × d , we deﬁne the following function
[Equation]
where we recall that M is the projection matrix to obtain the node features z = M (cid:62) x .
Note that it satisﬁes 0 (cid:54) R ( S ) (cid:54) (cid:107) β
(cid:63) (cid:107) 2Σ .
We additionally deﬁne Σ ( k )
=
(Id + Σ − 1 ) − 2 k Σ .
Theorem 2 (Regression risk without smoothing.) .
With probability at least 1 − ρ ,
[Equation]
where C = poly ( (cid:107) Σ (cid:107) , e d , | Id + Σ | ) , C (cid:48)
= poly ( ε − 1 , (cid:107) Σ (cid:107) , (cid:107) β (cid:63) (cid:107) ) and λ ( k )min =
λ min ( M (cid:62) Σ ( k ) M ) .
This theorem gives a limiting expression of R (0) and R (1) with additional error terms.
Since it is easy to show that for n large enough R (0) (cid:54) R ( ∞ )
≈ Var ( y ) , we obtain the following result.
Corollary 1.
Take any ρ > 0 , and suppose R reg.
(Σ (1) )
< R reg.
(Σ) .
If ε is sufﬁciently small and n is sufﬁciently large, then with probability 1 − ρ , there is k (cid:63) > 0 such that R ( k (cid:63) ) <
min( R (0) , R ( ∞ ) ) .
In other words, under some hypothesis on R reg. , there is indeed coexistence of beneﬁcial ﬁnite smoothing and oversmoothing.
Below we exhibit a simple example where this hypothesis is satisﬁed.
As expected in linear regression, the covariance of the x ( k ) k is key in the expression of the risk.
It can be seen in the proof of Theorem 2 (available at [10]) that x (1) behaves like (Id + Σ − 1 ) − 1 x , whose covariance is Σ (1) , hence the consequence that R (1)
≈ R reg.
(Σ (1) ) .
Similarly, by applying repeated smoothing we can extrapolate that x ( k ) behaves like (Id + Σ − 1 ) − k x , such that R ( k )
≈ R reg.
(Σ ( k ) ) .
The matrix Σ ( k ) has the same eigendecomposition as Σ , but where every λ i is replaced by λ ( k ) i =
2 31 02    2 31 02    2 31 02      3 z2 0 y      3 z2 0 y      3 z2 0 y 1010 1 Order of smoothing 0.
M S E
[Figure] Theory
(1 + 1 /λ i ) − [-
-]2 k λ i .
This can be interpreted as follows: when λ i (cid:29) 1 is large, λ ( k ) i ∼ λ
i , while if λ i (cid:28) 1 is small, λ ( k ) i ∼ λ
2 k +1 i .
Hence smoothing shrinks the directions of the small eigenvalues faster than that of the large ones .
Thus, if β (cid:63) is mostly aligned with the eigenvectors of large eigenvalues, smoothing may reduce unwanted noise in the node features z
= M (cid:62) x .
We illustrate this on a toy situation (Fig. 1).
Consider the following settings: d = 2 , p = 1 , Σ has two eigenvalues λ 1 = 2 and λ 2 = 1 / 2 , with respective eigenvectors u
=
[1 , 1] / √ 2 and u 2 =
[ − 1 , 1] / √ 2 , and β (cid:63) = bu 1 .
Finally, M (cid:62) =
[1 , 0] is the projection on the ﬁrst coordinate.
In this case, we can compute explicitely: R ( k ) ≈ R reg.
(Σ ( k ) )
=
λ 1 b 2 (2 λ + λ ( k ) 2 ) 2 + λ ( k ) 2 λ ( k )
(2 λ + λ ( k ) 1 + λ ( k ) 2 ) 2 .
So, if λ ( k )decreases faster than λ ( k ) 1 , this function will ﬁrst decrease to a minimum of approximately λ 1 b 2 ( 2 λ 2 λ + λ ( k(cid:63) ))
2 (when λ ( k ) 2 ≈ 0 ), before increasing again to λ 1 b 2 = (cid:107) β (cid:63) (cid:107) 2Σ =
lim n →∞ R ( ∞ ) .
Finite smoothing: classiﬁcation
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
D e n s i t y      3 0.0 0. D e n s i t y      3 0.
D e n s i t y
[Figure]
In this last section, we examine a simple classiﬁcation problem for two balanced classes with Gaussian distribution: ( x, y ) ∼ (1 / 2)( N µ, Id ⊗ { 1 } + N − µ, Id ⊗ {− 1 } ) .
We note that this is not a difﬁcult problem per se , and that linear regression is certainly not the method of choice to solve it.
Our main goal is to illustrate the smoothing phenomenon.
Our main result is the following.
Theorem 3.
Take any ρ > 0 .
If ε is sufﬁciently small, and (cid:107) µ (cid:107) , n are sufﬁciently large, and (cid:13)(cid:13) M (cid:62) µ (cid:13)(cid:13) > 0 , then with probability 1 − ρ , there is k (cid:63) >
such that R ( k (cid:63) )
< min( R (0) , R ( ∞ ) ) .
Note that we have assumed (cid:107) µ (cid:107) to be sufﬁciently large here.
However, we do not assume that (cid:13)(cid:13) M (cid:62) µ (cid:13)(cid:13) is large (just non-zero), and the classiﬁcation problem on the z i alone may be very difﬁcult.
As seen in the proof [10] and Fig.
2 on a d = 2 example, the interpretation here is the following: in the proper regime, communities will initially concentrate in the latent space faster than they get closer from each other , which helps learning on the z ( k ) .
Then, they eventually collapse together.