Not too little, not too much: a theoretical analysis of graph (over)smoothing
Abstract
We analyze graph smoothing with mean aggregation , where each node successively receives the average of the features of its neighbors.
Indeed, it has quickly been observed that Graph Neural Networks (GNNs), which generally follow some variant of Message-Passing (MP) with repeated aggregation, may be subject to the oversmoothing phenomenon: by performing too many rounds of MP, the node features tend to converge to a non-informative limit.
In the case of mean aggregation, for connected graphs, the node features become constant across the whole graph.
At the other end of the spectrum, it is intuitively obvious that some
MP rounds are necessary, but existing analyses do not exhibit both phenomena at once: beneﬁcial “ﬁnite” smoothing and oversmoothing in the limit.
In this paper, we consider simpliﬁed linear GNNs, and rigorously analyze two examples for which a ﬁnite number of mean aggregation steps provably improves the learning performance, before oversmoothing kicks in.
We consider a latent space random graph model, where node features are partial observations of the latent variables and the graph contains pairwise relationships between them.
We show that graph smoothing restores some of the lost information, up to a certain point, by two phenomena: graph smoothing shrinks non-principal directions in the data faster than principal ones, which is useful for regression, and shrinks nodes within communities faster than they collapse together, which improves classiﬁcation.
Introduction
In recent years, deep architectures such as Graph Neural Networks (GNNs), along with the availability of large sets of graph data, have signiﬁcantly broadened the ﬁeld of machine learning on graphs and structured data, with a myriad of applications ranging from community detection [11] to molecule classiﬁcation [20], drug discovery [19], quantum chemistry [15], recommender systems [44], semisupervised learning and so on.
See [7, 16, 6, 46] for reviews.
Most GNNs rely on the MessagePassing (MP) framework [15, 23], with a plethora of variants.
At each layer k , for each node i , a representation z ( k ) i is computed using the representations of the neighbors N i of i in the graph at the previous layer:
[Equation]
where AGG is an aggregation function that, crucially, treats { z ( k − 1) j } j ∈N i as an unordered set , to respect the absence of node ordering in the graph.
There are many variants of aggregation functions, based on sum, mean, max, min, degree-normalized [23], attention-based [39], and so on.
In this work, we consider one of the most classical, mean aggregation :
[Equation]
[Figure]
where the a ij ∈ R + are the entries of the adjacency matrix of the graph: either positive edge weights or 0 , 1 for unweighted edges, and Ψ is some function (usually a Multi-Layer Perceptron).
In other words, the aggregation process is a weighted average over the neighbors.
As we will see, it corresponds to a multiplication by (identity minus)
the random walk Laplacian of the graph.
While MP is a natural and rather general framework, its limitations were quickly observed by researchers and practitioners.
Foremost among them is the so-called oversmoothing phenomenon [27]: as the GNN gets deeper and many rounds of MP are performed, the node features z ( k ) i tend to become too similar across the graph, especially for small-world graphs with small diameter.
Oversmoothing prevents GNN from being too deep unless one is particularly careful.
A non-negligible part of the literature is dedicated to ﬁghting oversmoothing with various strategies (see below).
On the theoretical side, oversmoothing has mostly been analyzed in the inﬁnite-layer limit k → ∞ .
In this case, classical spectral analysis of graph operators such as the Laplacian can be leveraged to indeed show that node features will always converge to some limit that carries a limited amount of information [34].
This is particularly true for mean aggregation (2), with a constant limit across all nodes for a connected graph, see Sec.
Unlike some other graph operators such as the symmetric normalized Laplacian, where the limit still carries a small amount of information such as the degrees, with the random walk Laplacian all information is lost in the limit (beyond a single constant).
However, there has been little research at the other end of the spectrum, showing that some smoothing is useful for learning , despite this fact being intuitively and empirically obvious.
Generally, researchers show the power of GNNs for a sufﬁcient (unbounded) number of layers, such as the now-famous ability to distinguish graph isomorphism as well as the Weisfeiler-Lehman test and all its variants [47, 30], the ability to compute some graph functions [28], and so on.
Since these results are valid for an unbounded number of layers, the settings adopted in these works are, by deﬁnition, incompatible with non-informative oversmoothing.
To our knowledge, there is no work that formally models both phenomena at once : some smoothing is provably useful for learning, while too much smoothing inevitably leads to oversmoothing.
This work aims to ﬁll this gap.
We showcase two representative exemples, of regression and classiﬁcation, on which linear GNNs (aka, here, simply Linear Ridge Regression (LRR) on smoothed features) are subject to this double phenomenon.
Note that restricting ourselves to mean aggregation makes this claim quite non-trivial: in the absence of any “informative” node features, no information can be recovered by mean aggregation alone.
For instance, it leaves constant node features unchanged, and the limit k → ∞ is always a constant.
So the challenge is the following: node features must carry some information, such that a ﬁnite number of steps of mean aggregation provably increases the amount of useful information , before it loses it in the limit.
See Fig.
1 for an illustration.
To show this we adopt on a model of latent space random graphs, with node features.
The latter contain partial information about the unobserved latent variables on which both the labels and the graph structure depend.
On our examples, we prove that with high probability, graph smoothing improves performance before oversmoothing occurs.
We identify two key phenomena for this: smoothing shrinks non-principal directions in the data faster than principal ones (Sec. 4), and shrinks communities faster than they collapse together (Sec. 5).
Although our theoretical settings are obviously simpliﬁed, we believe it is a step towards a better comprehension of graph aggregation and of the relationship between node features and graph structure, at the heart of many phenomena in graph machine learning.
Related Work Oversmoothing [27] is a very active area of research in geometric deep learning, and an exhaustive list of works would be out of scope here.
The research has been mainly focused on novel architectures to relieve it, such as residual mechanisms [26, 10], randomly dropping connections [18], introducing local jumps [48], clever normalizations [50, 17, 5, 37] or regularizations [9], among others.
Some works have acknowledged the important role of the aggregation function, and proposed new exotic diffusion strategies [5] or to optimize it [24].
On the theoretical side, it has been mainly shown that repeatedly applying graph smoothing operators indeed induces convergence of the node features [34].
In this work, we analyze a model that present both the beneﬁts of ﬁnite smoothing despite oversmoothing in the limit.
Our theoretical framework is based on simpliﬁed linear GNNs and random graphs that explicitely model the dependence between labels, node features, and graph structure.
Despite their simplicity, linear GNNs, sometimes called Simpliﬁed Graph Convolutional networks (SGC), have been observed to exhibits relatively good performance [45, 33] and are routinely used in theoretical analyses [51].
Random graphs have been used extensively to analyze graph machine learning algorithms [43, 35] and the theoretical properties of GNNs such as stability [21, 36], transferability [25] or universality [22].
Our model crucially includes observed node features, an essential part in analyzing the smoothing process.
They have been shown to be correlated to sought-for labels in real graphs [13], and that this fact is key in the success of GNNs.
Our proof is in fact more akin to analyzing a graph diffusion process [31]: given appropriate initial conditions (observed node features), at initial time the diffusion produces a better signal for learning, before it eventually collapses to a single point.
To the best of our knowledge, this is the ﬁrst proof of this kind in a machine learning context.
Outline We describe our framework in Sec. 2.
In Sec. 3, we brieﬂy prove the oversmoothing phenomenon when k → ∞ , which is just the Markov chains ergodic theorem in our settings.
In
Sec. 4, we study a regression problem.
We derive an expression that predicts with good accuracy the optimal smoothing order k (cid:63) in some cases.
In Sec. 5, we study a classiﬁcation problem between two Gaussians.
Although we formally prove the existence of k (cid:63) > 0 , deriving an explicit expression for the risk is still open in this case.
Code to reproduce the ﬁgures is available at https://github. com/nkeriven/graphsmoothing .
Preliminaries
Notations.
The norm (cid:107)·(cid:107) is the Euclidean norm for vectors and spectral norm for (rectangular) matrices.
For a psd matrix Σ , the Mahalanobis norm is (cid:107) x (cid:107) 2Σ def.
= x (cid:62) Σ x .
The determinant of a matrix is | S | , and its smallest eigenvalue is λ min ( S ) .
The multivariate Gaussian distribution with mean µ and covariance Σ is denoted by N µ, Σ ( x ) = det (2 π Σ) − 12 e − 12 (cid:107) x − µ (cid:107) 2Σ − 1 .
We will use the shortened notations N µ = N µ, Id and N = N 0 .
Our bounds will involve various multiplicative constants poly ( · ) which are polynomials in their input.
SSL.
In this paper, we consider Semi-Supervised Learning (SSL) [8, 23] on an undirected graph of size n .
We observe a weighted adjacency matrix A
= [ a ij ] ni,j =1 ∈ R n × n + as well as node features z 1 , . . .
z n ∈ R p at each node of the graph.
We also observe some labels y 1 , . . .
, y n tr ∈ R at training time and aim to predict the remaining labels y n tr +1 , . . .
, y n .
In a classiﬁcation framework, y ∈ {− 1 , 1 } .
For simplicity, we assume that n tr and n te = n − n tr are both in O ( n ) 1 .
We denote by
Z ∈ R n × p the matrix whose rows contain the node features, Z tr , Z te respectively its ﬁrst n tr and last n te rows, and similarly Y tr , Y te the vectors containing the observed and non-observed labels.
Graph smoothing with mean aggregation.
Here we consider a simpliﬁed situation of linear GNN with mean aggregation, that is, equation (2) with linear Ψ .
Since all linear weights collapses into a single matrix, a linear GNN with k layers just corresponds to performing k rounds of mean aggregation on the node features, then learning on the smoothed features.
We denote by d A =
[ (cid:80) i a ij ] j ∈ R n + the vector containing the degrees of the graph and D = diag( d A ) .
Assuming that all degrees are non-zero, performing one round of mean aggregation corresponds to multiplying Z by L = D − 1 A .
Note that Id − L is then the random walk Laplacian of the graph.
The smoothed node features after k rounds of mean aggregation are:
[Equation]
Each row, denoted by z ( k ) i ∈
R p , contains the smoothed features of an individual node.
Similar to the non-smoothed features, its ﬁrst n tr and last n te rows are denoted Z ( k ) tr , Z ( k ) te .
Learning.
In this paper, we consider learning with a Mean Square Error (MSE) loss and Ridge regularization.
For λ > 0 , the regression coefﬁcients vector on the smoothed features is
[Equation]
Then, the test risk is deﬁned as
[Equation]
where
[Equation]
It is well known that when k → ∞ , the matrix L k will converge to a matrix with constant rows, and
R ( ∞ ) def.
=
lim k →∞ R ( k ) will just be close to the variance of Y , see Sec.
3 for a precise statement.
Very often, this degrades the results with respect to doing a simple linear regression: R (0) <
R ( ∞ ) .
Our goal is to illustrate some situations where a ﬁnite amount of smoothing provably improves the test risk, that is, there is an optimal k (cid:63) > 0 such that R ( k (cid:63) ) <
min( R (0) , R ( ∞ ) ) .
Random graph model.
To perform a ﬁne-grained analysis of our problem, we need a statistical model linking the graph, the node features, and the labels.
We adopt popular latent space random graph models akin to graphons [29].
Although such models are obviously idealized, we believe that they faithfully convey the main insights.
In these models, to each node i is associated an unobserved latent variable x i ∈ R d with d (cid:62) p (often d (cid:29) p ), and edge weights are assumed to be equal to a ij =
W ( x i , x j ) where W : R d × R d
→ R + is a connectivity kernel .
Note that edges may also be taken as random Bernoulli variables , but we do not consider this here for simplicity.
Moreover, we consider that the ( x i , y i ) are drawn iid from some joint distribution, and the node features are a linear projection of the latent variables to a lower dimension: z i = M (cid:62) x
i for some unknown M ∈ R d × p that satisﬁes M (cid:62) M = Id p .
At the end of the day:
[Equation]
For this model, note that
[Equation]
In other words, the smoothed node features Z ( k ) also correspond to a linear projection of the (unknown) smoothed latent variables X ( k ) .
To summarize, compared to “classical” machine learning on the ( x i , y i ) , we do not observe directly the x i , but only a projection of them
z i =
M (cid:62) x i .
Although we assume that M is orthogonal, we do not assume that it is “information-preserving” (e.g. it does not satisfy the Johnson-Lindenstrauss lemma), but rather that information is lost between the x and the z .
However, we also observe the graph W ( x i , x j ) .
Our goal is illustrate how mean aggregation may restore some of the lost information.
In the rest of the paper, we use the Gaussian kernel with a small additive term ε > 0 :
[Equation]
) where [Equation]
[Equation]
The coefﬁcient ε is added to lower-bound the degrees of the graph and avoid degenerate situations.
While this seems to be needed for our current proof technique, we use ε = 0 in Fig.
2 and 3.
The
Gaussian kernel is a classical model in theoretical graph machine learning [38].
Oversmoothing
In this section, we brieﬂy examine the oversmoothing case, when k → ∞ while all other parameters are ﬁxed.
In this case, it is well-known that all node features converge even for general GNNs [34].
For completeness, we state below this result in our settings.
We have the following well-known ergodic theorem for stochastic matrices such as L .
Theorem 1 (Ergodic theorem for stochastic matrices, e.g. [2, Thm. 4.2].) .
Recall that d A is the vector of degrees, let ¯ d A
=
d A /d (cid:62)
A 1 n .
We have
[Equation]
This easily allows us to prove the next result.
Corollary 1.
We have the following
[Equation]
where v = Z (cid:62)
¯ d and ¯ y tr = n − 1 tr (cid:80) n tr i =1 y i .
Proof.
We use Thm.
1 to get L k XM → 1 n v (cid:62) , and ( λ Id + vv (cid:62) ) − 1 v = v λ + (cid:107) v (cid:107) 2 .
Hence, in the limit k → ∞ , the predicted labels become all equal.
When λ ≈ 0 , this value is, as expected, the average of the labels in the training set ¯ y tr .
Using simple concentration inequalities, it is generally easy to show that R ( ∞ )
≈ Var ( y )
+ O (1 / √ n ) .
In most cases, this leads to situations where R (0) <
R ( ∞ ) , that is, it is better to perform regression directly on the node features.
In the next sections, we analyze some examples where smoothing provably helps.
Finite smoothing: Linear Regression
In this section, we consider a problem of linear regression on Gaussian data.
We consider x ∼ N 0 , Σ for some positive deﬁnite covariance matrix Σ , and y =
x (cid:62) β (cid:63) , without noise for simplicity (noise would just add an additional variance terms to all our bounds).
We will ﬁrst describe our main result that holds under a certain condition that is not necessarily easy to interpret, then give a sketch of proof in Sec. 4.1, and an example in dimension d = 2 where this assumption is satisﬁed in Sec. 4.2.
For a symmetric positive semi-deﬁnite matrix S ∈ R d × d , we deﬁne the following function
[Equation]
where we recall that M is the projection matrix to obtain the node features z = M (cid:62) x .
Note that it satisﬁes 0 (cid:54) R ( S ) (cid:54) (cid:107) β
(cid:63) (cid:107) 2 Σ .
Our result will be valid under the following assumption:
Assumption 1.
We have R reg.
(Σ) > R reg.
((Id + Σ − 1 ) − 2 Σ) .
Note that (Id + Σ − 1 ) − 2 Σ is indeed symmetric since (Id + Σ − 1 ) − 1 and Σ permute.
Our main result can be stated informally as follows, it is detailed in the next section along with a sketch of proof.
Recall that the kernel is taken as (6).
Theorem 2 (Existence of optimal smoothing for regression.) .
Take any ρ > 0 , and suppose that
Assumption 1 holds.
If ε is sufﬁciently small and n is sufﬁciently large, then with probability 1 − ρ , there is k (cid:63) > 0 such that R ( k (cid:63) ) <
min( R (0) , R ( ∞ ) ) .
4.1 Sketch of proof
As we will see, it is easy to show that R (0) <
R ( ∞ ) with high probability.
Our main goal will therefore be to show that R (1) <
R (0) with high probability under Assumption 1, which is sufﬁcient to show the existence of an optimal k (cid:63) (cid:62) 1 .
Using concentration inequalities, we will prove a rigorous non-asymptotic bound for R (1) .
In the next section, we also derive an intuitive expression for R ( k ) (although without rigorous proof), which we observe to match the numerics quite well.
The ﬁrst step is to derive a closed form expression for R (0) , which is fairly easy using standard concentration techniques for subgaussian variables.
The next result is proved in App. A.1.
Theorem 3 (Regression risk without smoothing.) .
With probability at least 1 − ρ ,
[Equation]
where λ min = λ min ( M (cid:62) Σ M ) .
As expected, when p = d , M = Id and λ → 0 , we have R reg.
(Σ) →
and the risk is exactly 0 in the inﬁnite sample limit (recall that we have assumed zero noise on the labels).
When p < d however, the limit risk is generally non-zero.
The worst case is obtained when Σ β (cid:63) is orthogonal to M (cid:62) , where the risk reaches its maximum at (cid:107)
β (cid:63) (cid:107) 2Σ =
E | y
| 2 .
Since this is the variance of y , this is also lim n →∞ R ( ∞ ) , hence we always have R (0) (cid:54) R ( ∞ ) with high probability for n large enough.
Let us now turn to computing the risk after one step of smoothing k = 1 .
We deﬁne Σ ( k )
=
(Id + Σ − 1 ) − 2 k Σ .
The main result of this section is the following.
Theorem 4 (Regression risk with one step of smoothing.) .
With probability at least 1 − ρ ,
[Equation]
where C = poly ( (cid:107) Σ (cid:107) , e d , | Id + Σ | ) , C (cid:48)
= poly ( ε − 1 , (cid:107) Σ (cid:107) , (cid:107) β (cid:63) (cid:107) ) and λ min =
λ min ( M (cid:62) Σ (1) M ) .
This theorem gives a limiting expression or R (1) with two additional error terms.
The ﬁrst goes towith ε and is due to the deviation from the kernel (6) to the exact Gaussian kernel W g .
The second term goes to 0 when n → ∞ and is controlled via concentration inequalities.
The limit risk when ε → 0 , n → ∞ is R (1) ≈ R reg.
(Σ (1) ) , which is strictly lower than R reg.
(Σ) by Assumption 1 and proves Theorem 3.
Note that, to get R (1) <
R (0) , we generally need ε (cid:46) e − d
and therefore n (cid:38) e d , which seems to be an unavoidable artifact in our current proof technique.
Let us try to better understand Assumption 1 by sketching the proof of Thm. 4.
The proof relies on an approximate description of the distribution of the smoothed node features z (1)
i
[Equation]
where we recall that the x (1) i are the rows of X ( k )
=
L k X .
We deﬁne d ( x ) =
| Id + Σ | − 12 e − 12 (cid:107) x (cid:107) 2(Id+Σ) − 1 and
[Equation]
Then, using some chaining concentration inequalities for subgaussian variables (Lemma 7 in the appendix) and properties of Gaussian distributions (Lemma 5), we can prove the following.
Lemma 1.
With probability at least 1 − ρ , for all i = 1 , . . .
, n :
[Equation]
[Table]
[Equation]
[Table]
  
[Equation] [Equation]
ρ
[Equation]
√ n (13) where C = poly ( ε − 1 , (cid:107) Σ (cid:107) , | Id + Σ | ) .
Hence the smoothed latent variables behaves almost like (Id + Σ − 1 ) − 1 x , up to a deviation ε that is handled in Lemma 3 in the appendix.
The covariance of these data is Σ (1) =
(Id + Σ − 1 ) − 2 Σ , hence we can adapt the proof of Thm.
3 to obtain Thm. 4.
All details are given in App.
A.2.
4.2 Intuition and exact computation in dimension d =
We proved above that x (1) behaves almost like (Id + Σ − 1 ) − 1 x , whose covariance is Σ (1) .
Similarly, by applying repeated smoothing we can extrapolate that x ( k ) behaves like (Id + Σ − 1 ) − k x , such that R ( k )
≈ R reg.
(Σ ( k ) ) .
The rigorous proof of this fact becomes increasingly complicated and is skipped here.
The matrix Σ ( k ) has the same eigendecomposition as Σ , but where every eigenvalue λ
i
[Figure]
2 0 y      3 z2 0 y      3 z2 0 y (
[Figure]
[Figure]
3 z0 2 y
[Figure]
is replaced by λ ( k ) i
= (1 + 1 /λ i ) −
k λ i .
This can be interpreted as follows: when λ i (cid:29) 1 is large, λ (1) i ∼ λ i , while if λ i (cid:28) 1 is small, λ (1) i ∼ λ
2 k +1 i
(note that the constant “1” here is due to our kernel (6), it is not inherently signiﬁcant).
Hence smoothing shrinks the directions of the small eigenvalues faster than that of the large ones .
Thus, if β (cid:63) is mostly aligned with the eigenvectors of large eigenvalues, shrinking the small eigenvalues may reduce unwanted noise that emerges when projecting the node features z = M (cid:62) x .
On the other hand, if all eigenvalues of Σ are equal, then Σ ( k ) ∝ Σ , and smoothing does not help , since in the limit λ = 0 , the risk is invariant to scaling
R reg.
( aS ) =
R reg.
( S ) .
Worse, we will see on an example below that smoothing can actually degrade the performance when β (cid:63) is unpropery aligned.
We illustrate this in dimension d = 2 .
Consider the following settings: d = 2 , p = 1 , Σ has two eigenvalues λ 1 (cid:29) 1 and λ 2 (cid:28) 1 , with respective eigenvectors u
=
[1 , 1] / √ 2 and u 2 =
[ − 1 , 1] / √ 2 , and β (cid:63) is fully correlated with the ﬁrst eigenvector: β (cid:63) =
bu 1 .
Finally, M (cid:62) =
[1 , 0] is the projection on the ﬁrst coordinate.
This situation is represented in Fig.
In this case, we can compute explicitely:
[Equation]
So, if λ ( k ) 2 decreases faster than λ ( k ) 1 , this function will ﬁrst decrease to a minimum of approximately λ 1 b 2 (cid:18)
[Equation]
(cid:19)(when λ ( k )≈ 0 ), before increasing again to λ 1 b 2 = (cid:107) β (cid:63) (cid:107) 2Σ =
lim n →∞ R ( ∞ ) .
This is illustrated in Fig.
2, for λ 1 = 2 and λ 2 = 1 / 2 , where we empirically observes a minimum k (cid:63) that matches rather well the one predicted by (14).
Homophily vs. Heterophily and a failure case In graph theory, homophily refers to the concept that linked nodes tend to display similar properties: for instance, friends on social networks have similar preferences, and so on.
In graph machine learning, it generally means that linked nodes tend to have similar node features and labels.
This concept is at the core of many graph signal processing and graph machine learning methods: for instance, spectral clustering is akin to a low-pass ﬁlter on the graph structure.
However, it has been observed that real graphs may sometimes exhibit a low level of homophily [51, 5].
They are rather said to be heterophilic , a somewhat less “well-deﬁned” concept: in heterophilic graphs, linked nodes can be similar or dissimilar, some attributes can be homophilic and others heterophilic, and so on.
In our settings, at ﬁrst glance it seems that our very regular random graph model always results in homophilic graphs, as the Gaussian kernel decreases with the distance between latent variables, and the latter are strongly linked with the node features.
This is partly true, however is it also possible that nodes linked by a “strong” edge (with a high weight) have very different labels, which can be said to be a (toy) example of heterophily.
For instance, consider the 2D linear regression example above given by (14).
We have seen that when the regression vector is in the direction of the eigenvector corresponding to a high eigenvalue, then beneﬁcial smoothing appears, as it reduces the noise in the observed node features (Fig. 2a).
However, when the regression vector is instead in the low-eigenvalue direction, then close-by latent variables have very different labels, and the graph is more heterophilic.
In this case, beneﬁcial smoothing does not appear, and any smoothing strictly degrades the MSE!
(Fig. 2b)
This is due to the fact that in this case the “information” in node features vanishes faster than the noise.
Of course, this is an exceedingly simple model of heterophily, and a better understanding and modelization of this phenomenon remains an outstanding open question.
Discussion Recent literature on GNNs have adressed both oversmoothing and heterophily by clever normalization techniques [50, 17, 5, 37], combined with quantitative metrics of these phenomena [49, 51].
However, these tend to indiscriminately combat oversmoothing, without taking into account potential beneﬁcial smoothing.
In future work, our analysis could help designing more detailed normalization methods, e.g. after some estimation step that would identify which directions in the data are squeezed by smoothing, and which of them are relevant or not for learning.
Finite smoothing: classiﬁcation
In this section, we examine a simple classiﬁcation problem for two balanced classes with Gaussian distribution with identity covariance.
The distribution of the labels and latent variables is:
[Equation]
That is, with equal probability x is drawn from N µ and y = 1 , or x ∼ N − µ and y = − 1 .
As (cid:107) µ (cid:107) increases, the problem become simpler, there is an extensive literature on this problem [12, 40, 3].
Note that in this case z i are also Gaussian, with mean ν def.
=
M (cid:62) µ or − ν and identity covariance.
We note that this is not a difﬁcult problem per se , and that linear regression with the MSE is certainly not the method of choice to solve it: there are plethora of losses better adapted to binary classiﬁcation such as the binary cross-entropy (left for future investigations), or even other dedicated methods: a
Spectral Clustering algorithm on the graph alone would be able to perform the classiﬁcation task under some mild hypotheses [40, 1] (without using the node features!).
Nevertheless, let us recall that our main goal is to illustrate the smoothing phenomenon, and as we will see, the interpretation here will be quite different from the previous section.
Our main result is the following.
Theorem 5 (Existence of optimal smoothing for classiﬁcation.) .
Take any ρ > 0 .
If ε is sufﬁciently small, and (cid:107) µ (cid:107) , n are sufﬁciently large, and (cid:13)(cid:13) M (cid:62) µ (cid:13)(cid:13) > 0 , then with probability 1 − ρ , there is k (cid:63) >
such that R ( k (cid:63) )
< min( R (0) , R ( ∞ ) ) .
Note that we have assumed (cid:107) µ (cid:107) to be sufﬁciently large here.
However, we do not assume that (cid:13)(cid:13) M (cid:62) µ (cid:13)(cid:13) is large (just non-zero), and the classiﬁcation problem on the z i alone may be very difﬁcult.
The rest of this section presents a sketch of proof and intuitions behind this theorem.
5.1 Sketch of proof and intuition
As in the previous section, it will be easy to show that R (0) <
R ( ∞ ) with high probability, and we will prove that R (1) < R (0) with high probability.
Again, we start by providing an expression for
R (0) .
For s ∈ R + , we deﬁne the following function
[Equation]
The next result is proved in App.
B.1.
Recall that ν
= M (cid:62) µ .
Theorem 6 (Classiﬁcation risk without smoothing.) .
With probability at least 1 − ρ ,
[Equation]
When (cid:107) ν (cid:107) → ∞ , the risk goes to 0 , as expected, since the Gaussians get further and further away.
However, when (cid:107) ν (cid:107) → 0 , which can happen either when (cid:107) µ (cid:107) is small or when M becomes orthogonal to µ , the risk goes to 1 , its worst value, for random guesses.
Since it is also the variance of y , we have indeed R (0) (cid:54) 1 ≈ R ( ∞ ) with high probability for n large enough.
Let us now turn to computing the risk after one step of smoothing k = 1 .
The main result of this section is the following.
Theorem 7 (Classiﬁcation risk with one step of smoothing.) .
With probability at least 1 − ρ ,
[Equation]
where C = poly ( (cid:107) µ (cid:107) , e d ) and C (cid:48)
= poly ( ε − 1 , (cid:107) µ (cid:107) ) .
This theorem show that R (1) ≈ R cl.
(1 / 4) with two additional error terms.
First of all, a quick function study shows that R cl.
(1 / 4) < R cl.
(1) when (cid:107) ν (cid:107) > 0 , which shows Thm. 5 when the errors are small enough.
The last error term goes to 0 when n → ∞ and is controlled via concentration inequalities.
The ﬁrst one is small when ε is small and (cid:107) µ (cid:107) is large enough.
We remark that, unlike the previous section where the error terms vanished in the limit ε → 0 , n → ∞ , here there is a non-zero error term due to (cid:107) µ (cid:107) whose explicit expression is still open.
Hence, for instance, the discrepancy between the empirical observations and the theory in Fig.
3 compared to Fig.
Note that, as in the previous section, we need at least ε (cid:46) e − d and n (cid:38) e d .
However, here we also need (cid:107) µ (cid:107) (cid:38) √ d .
This rate is similar to early analyses of Gaussian Mixture learning [12], although they have been greatly improved since [3].
As previously, we deﬁne here d
[Equation]
, and
[Equation]
The following result is similar to Lemma 1 and is shown in App.
B.2.
Lemma 2.
With probability at least 1 − ρ ,
[Equation]
Let us now examine ϕ cl.
( x ) closer.
In the limit ε → 0 , ϕ cl.
( x ) is a convex combination of ( x + µ ) /and
( x − µ ) / 2 .
Hence, when x ∼ N µ , with high probability x is close to µ and d µ ( x ) (cid:29) d − µ ( x ) , and in this case, ϕ cl.
( x ) ≈ x + µ 2 , whose distribution is N µ, Id / 4 .
The same reasoning applies to the other community.
Hence, up to some error O (cid:16) e −(cid:107) µ (cid:107) 2 / 4 (cid:17) due to the communities getting closer to each other, the smoothed features in each community have the same mean but a reduced variance Id / 4 , thus the limit risk R cl.
(1 / 4) in our limit expression for R (1) .
In other words, the communities shrink faster than they collapses together , and this reﬂects on the projected node features.
An illustration of this phenomenon is given in Fig.
All proof details are in App.
B.2.
5.2 Numerical illustration
In light of the proof of the theorem above, when x i belongs to the ﬁrst community and x (1) i ≈ ϕ cl.
( x i )
≈ x
i + µ 2 , applying a second smoothing would transform it to ϕ cl.
( x i )
+
µ≈ x i +3 µ, that
D e n s i t y      3 0.0 0.
D e n s i t y      3 0.
D e n s i t y 101010 2 Order of smoothing 0.
M S E
Empirical Theory is, it keeps the same mean but now has variance Id / 16 .
If we look at the proof, the error term
O (cid:16) e −(cid:107) µ (cid:107) 2 / 4 (cid:17) would become in this case e −
[Equation]
(cid:107)
[Equation] [Equation]
While this expression is far from being exact and we do not a rigorous proof here (which seems far more complex than the case k = 1 ), we can infer some approximate expression:
[Equation] [Equation]
Unlike the expression (14), the term R cl.
(4 − k ) is strictly decreasing when k increases.
Oversmoothing is modelled by the error term, for which we do not have an exact expression, and for which we suspect that the quality of approximation degrades as k increases.
Nevertheless, we evaluate this expression on an example in Fig.
3 (with an adjusted multiplicative constant for the error term in (21)) and ﬁnd that it is a reasonably good approximation, at least for small k .
Conclusion and outlooks
While the oversmoothing phenomenon k → ∞ has been well characterized, until now there has been no theoretical studies that rigorously modelled both the beneﬁts of ﬁnite smoothing before oversmoothing kicks in.
In this paper, we adopted a simpliﬁed context of linear GNNs with mean aggregation and random graphs with partially observed latent variables, and proved on two representative examples the co-existence of both phenomena.
We identiﬁed two mechanisms for the beneﬁts of mean aggregation: it tends to shrink noisy principal components faster than meaningful ones, and it tends to gather nodes of the same community faster than they collapses together.
We obtained theoretical expressions up to some error terms that matched the numerics quite well on simple synthetic data.
There are many outlooks to this work.
First and foremost, deriving inspiration from our theoretical observations to design better methods of setting the order of smoothing in practical application is a major challenge.
As seen in Fig.
1 in the introduction, both mechanisms that we identiﬁed seem to come into play on real data.
However, many quantities appearing in the risks (14) and (21) need to be estimated.
Second, extending our theory to more complex loss functions (especially for classiﬁcation) and non-linear GNNs is crucial.
Finally, our work is a step towards a better understanding of the relationship between node features and graph structure , which is at the heart of (over)smoothing, heterophily, and all graph machine learning methods.
A more general theory, and more realistic models of random graphs to analyze it, is still an open question.
Acknowledgment
This work was supported by ANR JCJC GRandMa (ANR-21-CE23-0006).
NK thanks S. Vaiter for inspiring discussions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] In the Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We consider toy experiments, on which variance is insigniﬁcantly low.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
This appendix contains all proof of our results.
We start by Linear Regression in App.
A, then classiﬁcation in App.
B. App.
C contains technical Lemmas.
At their core, the proofs are a combination of chaining concentration inequalities for subgaussian variables and derivations on Gaussian distributions.
A Linear Regression A.1 Proof of Theorem
Proof.
Let us begin by the concentration of the optimal ˆ β
=
( λ Id + Z (cid:62) tr
Z tr /
[Equation]
) − 1 Z
[Equation]
tr Y tr /n .
[Equation]
By an application of [41, Corollary 5.50], which is a concentration inequality for covariance estimates of subgaussian vectors, we get that with probability at least 1 − ρ ,
[Equation]
n since n tr = O ( n ) .
In particular, for n large enough, we get λ min ( 1 n tr
[Equation]
i z i z
[Equation]
i ) (cid:62) λ min ( M (cid:62) Σ M ) / 2 .
Similarly,
[Equation]
and using the same concentration on the covariance of X we obtain
[Equation]
At the end of the day
[Equation]
Let us now compute the limit of the test risk:
[Equation]
By a reasoning identical to the one above on 1 n te X (cid:62) te X te ≈ Σ , 1 n te Z (cid:62) te Y te ≈ M
(cid:62) Σ β (cid:63) and 1 n te Z (cid:62) te Z te ≈
M (cid:62) Σ M , and using (cid:13)(cid:13) ( λ Id + M (cid:62) Σ M ) − 1 M (cid:62) Σ β (cid:63) (cid:13)(cid:13) (cid:54) (cid:107) β (cid:63) (cid:107) (which can be seen using an SVD decomposition of M Σ 12 ) and (cid:107) M (cid:107) (cid:54) (cid:107) 1 (cid:107) , after some computation we obtain
[Equation]
The ﬁrst term is R reg.
(Σ) , we use a union bound over all these inequalities to conclude the proof.
A.2 Proof of Theorem
We start with the proof of Lemma 1.
A small reminder on subgaussian variables is given in App.
C.
Proof of Lemma 1.
The proof relies on chaining concentration inequalities for subgaussian variables and properties of Gaussian distributions (Lemma 5 and 7 in App. C).
Note that (cid:107) ϕ reg.
( x ) (cid:107) Σ − 1 (cid:54) (cid:13)(cid:13)(cid:13) Σ 12 (Id+Σ) − 12 (cid:13)(cid:13)(cid:13)
ε d ( x ) (cid:107) x (cid:107) (Id+Σ)
− 1 (cid:46) 1 ε .
Moreover, by Lemma 5,
E W g ( x, X ) =
d ( x ) and E W g ( x, X ) X
= d ( x )(Σ −
+ Id) − 1 x .
We decompose
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For the ﬁrst term, applying Lemma 7 with W = 1 , with probability 1 − ρ we have
[Equation]
√ n
Similarly, since W g is C L Lipschitz in the ﬁrst variable with respect to (cid:107)·(cid:107) Σ − 1 with C L (cid:46) (cid:13)(cid:13)(cid:13) Σ 12 (cid:13)(cid:13)(cid:13) , applying Lemma 7 we get
[Equation]
√
[Equation]
and
[Equation]
n
[Equation]
which concludes the proof of the ﬁrst inequality.
The second is obtained by decomposing and using (cid:107) ϕ reg.
( x ) (cid:107) Σ − 1 (cid:46) 1 /ε .
We will also need the following Lemma, to bound the deviation brought by ε in the expression for ϕ reg. .
Lemma 3.
We have
[Equation]
Proof.
Denote by B r = { x ; (cid:107) x (cid:107) Σ −(cid:54) r } .
Within this ball, since (cid:107) x (cid:107) (Id+Σ
[Table]
(cid:54)
[Table]
Id + Σ)
[Table] [Table]
[Table]
x
[Table] [Table]
x
[Table]
Σ
[Table]
1 , we have 1 /d ( x ) (cid:54) | Id + Σ | 12 e r 2 / 2 .
We also recall that
[Table]
x
[Table]
x
[Table]
and
[Table]
cr
[Table]
Now we decompose, using
[Table]
[Equation]
Σ
[Equation]
Where the last line is obtained by choosing r = (cid:112) (8 / 5) log(1 /ε ) .
Then we use that for two random variables X and Y , (cid:107) E X − E Y (cid:107) (cid:54) (cid:113) E (cid:107)
X − Y (cid:107) 2 , and (cid:13)(cid:13)
E XY (cid:62) − E Y Y (cid:62) (cid:13)(cid:13) (cid:54) (cid:113) E (cid:107) Y (cid:107)
2 (cid:113) E (cid:107) X − Y (cid:107) 2 , and (cid:13)(cid:13)
E XX (cid:62) − E Y Y (cid:62) (cid:13)(cid:13) (cid:54) ( (cid:113) E (cid:107)
X
(cid:107)
+ (cid:113) E (cid:107) Y (cid:107) 2 ) (cid:113) E (cid:107) X − Y (cid:107) 2 , to conclude.
We are now ready to show Theorem
Proof of Theorem 4.
We proceed in two steps.
First, we use Lemma 1 to show that we can replace the x (1) by ϕ reg.
( x i ) in the computation of the risk.
Second, we use Lemma 3 to concentrate the ϕ reg.
( x i ) around their new expectations.
We deﬁne ˆ β ϕ and R ϕ by replacing Z ( k ) with Z ϕ =
X ϕ M where the rows of X ϕ are the ϕ reg.
( x i ) , in (3) and (4).
Since ϕ reg.
( x ) is bounded, it is a subgaussian vector.
We can therefore apply the same reasoning as in the proof of Theorem 6 and concentrate ( Z ϕ tr ) (cid:62) Z ϕ tr /n tr .
Using Lemma 4, for ε small enough, and n large enough, it is almost equal to M (cid:62) Σ (1) M , and thus λ min (( Z ϕ tr ) (cid:62) Z ϕ tr /n tr ) (cid:38) λ min ( M (cid:62) Σ (1) M ) .
Finally, using Lemma 2, for n large enough λ min (( Z (1) tr ) (cid:62) Z (1) tr /n tr ) (cid:62) λ min (( Z ϕ tr ) (cid:62) Z ϕ tr /n tr ) / 2 .
Since (cid:107) ϕ reg.
( x ) (cid:107) Σ − 1 (cid:46) 1 /ε , we bound
[Equation]
=
[Equation]
ϕ tr )
[Equation]
Z ϕ tr
[Equation]
tr )
[Equation] [Equation]
ϕ tr )
[Equation]
tr
[Equation]
Using the same bounds on 1 n te ( Z (1) te ) (cid:62) Y te and 1 n te ( Z (1) te ) (cid:62) Z (1) te , we get
[Equation]
Finally
[Equation]
we apply the same reasoning as in the proof of Theorem 3, we obtain
[Equation]
where Σ ϕ = E ϕ ( x ) ϕ
( x ) (cid:62) and Σ ϕ,x
=
E ϕ ( x ) x (cid:62) .
We use Lemma 3 and a union bound over all these inequalities to conclude the proof.
B Classiﬁcation of Gaussian mixtures B.1 Proof of Theorem
Proof.
The proof is similar to that of Theorem 3.
We begin by the concentration of the optimal ˆ β
=
( λ Id + Z (cid:62) tr
Z tr /n tr ) − 1 Z (cid:62)
tr Y tr /n tr .
We denote by I 1 , I − 1 ⊂ { 1 , . . .
, n tr } the indices of the x i respectively from the ﬁrst and second community
[Equation]
of size n 1 and n − 1 .
We have
[Equation]
Since the communities are balanced, by a simple application of Hoeffding’s inequality, with probability at least 1 − ρ we have n 1 /n tr = 1 / 2 + O (cid:16)(cid:112) log(1 /ρ ) /n
(cid:17) .
Then, as in the proof of Theoremby an application of [41, Corollary 5.50], with probability at least 1 − ρ ,
[Equation]
n and E N µ zz (cid:62) =
νν (cid:62) + Id .
We apply the same reasoning for I − 1 , and we obtain
[Equation]
n
In particular, for n large enough, λ min ( 1 n tr Z (cid:62) tr Z tr ) (cid:62) 1 / 2 .
Similarly,
[Equation]
Using the fact that (cid:107) x (cid:107) = sup (cid:107) u (cid:107) (cid:54) 1 u (cid:62) x and for such a u the variable u (cid:62) ( z − ν ) is unit Gaussian, applying Lemma 6 we get that with probability 1 − ρ
[Equation]
n and similarly for I − 1 , and therefore
[Equation]
At the end of the day
[Equation]
n
Moreover
[Equation]
ν is an eigenvector for ( λ + 1)Id + νν (cid:62) so the limit is actually:
[Equation]
Let us now compute the limit of the test risk:
[Equation]
By a reasoning identical to the one above on 1 n te Z (cid:62) te Y te ≈ ν and 1 n te Z (cid:62) te Z te ≈ Id + νν (cid:62) , and using (cid:13)(cid:13)(cid:13)
ˆ β lim (cid:13)(cid:13)(cid:13) (cid:54) (cid:107)
ν (cid:107) and (cid:13)(cid:13)
Id + νν (cid:62) (cid:13)(cid:13)
=
1 + (cid:107) ν (cid:107) 2 , we obtain n
[Equation]
Y te
[Equation]
(cid:62) te
[Equation]
We use a union bound over all these inequalities to conclude the proof.
B.2 Proof of Theorem
We start with the proof of Lemma 2.
Proof of Lemma 2.
The proof is similar to that of Lemma 1.
Here we denote by I 1 , I − 1 ⊂ { 1 , . . .
, n } the indices of the x i from the ﬁrst and second community, of size n 1 and n − 1 , from the whole sample set.
Again, since the communities are balanced, with probability 1 − ρ we have | I 1 | /n
≈
12 + O (cid:16)(cid:112) log(1 /ρ )
/n (cid:17) .
We decompose
[Equation]
+ d − µ ( x i ) ( xi − µ 2 ))
ε + 1 n (cid:80) j W g
( x i ,x j ) (cid:13)(cid:13)(cid:13)(cid:13)
+ (cid:13) (cid:13)(cid:13)(cid:13) 1 2 ( d µ ( x i ) ( xi + µ 2 ) + d − µ ( x i ) ( xi − µ 2 ))
ε + 1 n (cid:80) j W g
( x i ,x j ) − 1 2 ( d µ ( x i ) ( xi + µ 2 ) + d − µ ( x i ) ( xi − µ 2 ))
ε + 12 ( d µ ( x i )+ d − µ ( x i ))
(cid:13) (cid:13) (cid:13) (cid:13) (cid:54) 1 n (cid:13)(cid:13)(cid:13)(cid:80)
j x j (cid:13)(cid:13)(cid:13) + ε − 1 (cid:13)(cid:13)(cid:13) 1 n (cid:80) j W g
( x i , x j ) x
j − 12 (cid:0) d µ ( x i ) (cid:0) x i + µ 2 (cid:1) + d − µ ( x i ) (cid:0) x i − µ 2 (cid:1)(cid:1)(cid:13)(cid:13)(cid:13) + 14 ε 2 (cid:12)(cid:12)(cid:12) 1 n (cid:80) j W g ( x i , x j ) − d µ ( x i )+ d − µ ( x i )(cid:12)(cid:12)(cid:12) (cid:13)(cid:13)
d µ ( x i ) (cid:0) x i + µ 2 (cid:1) + d − µ ( x i ) (cid:0) x
i − µ 2 (cid:1)(cid:13)(cid:13)
For the ﬁrst term, with probability 1 − ρ we have
[Table]
Where we have used Lemma 7 with W = 1 and n 1 = n − n − 1 ≈ n/ 2 for the last line.
For the second term, similarly
[Table]
[Equation]
Using Lemma
[Equation]
we have E N W g ( x, X ) X
= d 0 ( x ) x 2 .
Hence, using the Lipschitz properties of the
Gaussian kernel
[Equation]
and since we can center
[Equation] [Equation]
x
[Equation]
x
[Equation]
and x j − µ ∼ N , using Lemma 7 we obtain
[Equation]
We proceed similarly for the second term, and again with the ﬁrst part of Lemma 7 to obtain
[Equation]
which gives us the ﬁrst result.
The second is obtained by simply decomposing and using (cid:107) ϕ cl.
( x ) (cid:107) (cid:46) 1 /ε .
We will need the following Lemma, similar to Lemma 3.
Lemma 4.
Let x ∼ N µ .
We have
[Equation]
/ 4) (cid:13)(cid:13)
(cid:46)
d 2 d/ 2 (cid:107) µ (cid:107) 2 ε 1 / 4 + (cid:107) µ (cid:107)√ d εe −(cid:107) µ (cid:107) 2 /
Proof.
Denote by B µ,r a ball of radius r around µ .
Within this ball, d µ ( x ) (cid:62) 2 − d/ 2 e − r 2 / 4 , while d − µ ( x ) (cid:54) 2 − d/ 2 e −(cid:107) µ (cid:107) 2 / 4 .
We also recall that (cid:82) B cµ,r N µ (cid:54) e − r 2 / 2 and (cid:82) B cµ,r (cid:107) x − µ (cid:107) 2 N µ (cid:46)
[Table]
e
Now we decompose
[Table]
[Equation]
Where the last line is obtained by choosing r = (cid:112) 3 log(1 /ε ) .
Then we use that for two random variables X and Y , (cid:107) E X − E Y (cid:107) (cid:54) (cid:113) E (cid:107)
X − Y (cid:107) 2 , and (cid:13)(cid:13)
E XX (cid:62) − E Y Y (cid:62) (cid:13)(cid:13) (cid:54) ( (cid:113) E (cid:107)
X
(cid:107)
+ (cid:113) E (cid:107) Y (cid:107) 2 ) (cid:113) E (cid:107) X − Y (cid:107) 2 to conclude.
We are now ready to prove Theorem 7.
Proof of Theorem 7.
We proceed as in the proof of Theorem 4.
We deﬁne ˆ β ϕ and R ϕ by replacing
Z ( k ) with Z ϕ =
X ϕ M where the rows of X ϕ are the ϕ cl.
( x i ) .
Since (cid:107) ϕ cl.
( x ) (cid:107) (cid:54) max (cid:16) (cid:107) x + µ (cid:107), (cid:107) x − µ (cid:107)(cid:17) , ϕ cl.
( x ) is a subgaussian vector with (cid:13)(cid:13) u (cid:62) ϕ cl.
( x ) (cid:13)(cid:13)
ψ 2 (cid:46) (cid:13)(cid:13) u (cid:62)
x (cid:13)(cid:13) ψ 2 (cid:46) 1 .
We can therefore apply the same reasoning as in the proof of Theorem 6 and concentrate ( Z ϕ tr ) (cid:62) Z ϕ tr /n tr .
Using Lemma 4, for ε small enough, and (cid:107) µ (cid:107) and n large enough, it is almost Id / 4 + νν (cid:62) , and thus λ min (( Z ϕ tr ) (cid:62) Z ϕ tr /n tr ) (cid:38) 1 .
Finally, using Lemma 2, for n tr large enough λ min (( Z (1) tr ) (cid:62) Z (1) tr /n tr ) (cid:62) λ min (( Z ϕ tr ) (cid:62) Z ϕ tr /n tr ) / 2 (cid:38) 1 .
Since (cid:107) ϕ cl.
( x ) (cid:107) (cid:54) 1 /ε , using Lemma 2 we bound (cid:13)(cid:13)(cid:13) ˆ β − ˆ β
[Equation]
=
[Equation]
tr ) (cid:62) Z ϕ tr /n tr ) − 1 ( Z ϕ tr ) (cid:62) Y tr /n tr (cid:13)(cid:13)(cid:13) (cid:54) (cid:13)(cid:13)(cid:13) (( λ Id + ( Z (1) tr ) (cid:62) Z (1) tr /n tr ) − 1 − ( λ Id + ( Z ϕ tr ) (cid:62) Z ϕ tr /n tr ) − 1 )( Z ϕ tr ) (cid:62) Y tr /n tr (cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13) ( λ Id + ( Z (1) tr ) (cid:62) Z (1) tr /n tr ) − 1 (( Z (1) tr ) (cid:62) Y tr /n tr − ( Z ϕ tr ) (cid:62) Y tr /n tr ) (cid:13)(cid:13)(cid:13) (cid:54)ε sup i (cid:13)(cid:13)(cid:13) x (1) i
( x (1) i ) (cid:62) − ϕ
( x i ) ϕ
( x i ) (cid:62) (cid:13)(cid:13)(cid:13) + sup i (cid:13)(cid:13)(cid:13) x
(1) i − ϕ
( x i ) (cid:13)(cid:13)(cid:13) (cid:46) poly (1 /ε ) log n ( √ d + (cid:112) log(1 /ρ ))
√ n
Using the same bounds on 1 n te ( Z (1) te ) (cid:62) Y te and 1 n te ( Z (1) te ) (cid:62) Z (1) te , and using (cid:13)(cid:13)(cid:13) ˆ β ϕ (cid:13)(cid:13)(cid:13) (cid:46) ε − 1 , we get (cid:12)(cid:12)(cid:12) R (1) −
[Equation]
Then, we apply the same reasoning as in the proof of Theorem 6, we obtain
[Equation]
where β ϕlim =
( λ Id + E z ϕ
( z ϕ ) (cid:62) )
− 1 ( E yz ϕ ) .
Finally, using Lemma 4, and computations similar to 6, we obtain
[Equation]
which concludes the proof.
C Technical Lemmas
This section gather some technical Lemmas used throughout the proofs.
We start by some derivations on Gaussian distributions, then details the chaining concentration inequalities used in this work.
C.1 Properties of Gaussians
Lemma 5 (Gaussian integral) .
Let W ( x, y ) =
e − 12 (cid:107) x − y (cid:107) Σ − 1 W be the Gaussian kernel with covariance Σ W .
We have
[Equation]
Proof.
We have the following when n → ∞ .
[Equation]
Since the convolution of two gaussians is a Gaussian.
And
[Equation]
using that the ﬁrst term in the sum is 0 since it is the integral of a derivative.
C.2 Chaining and subgaussian variables
A random variable X is said to be subgaussian if
[Equation]
A good reference on subgaussian random variables is [42, Chap. 2].
For a bounded random variable
X and subgaussian Y , we have immediately from the deﬁnition
[Equation]
Lemma 6 (Chaining on non-normalized kernels) .
Consider x i ∼ P ∈ P
( R m ) , a ball B r ⊂ R n with respect to a metric d , and a bivariate function F : R n × R m → R that satisﬁes:
For all z ∈ B r , F ( z, X ) is subgaussian with norm (cid:107) F ( z, X ) (cid:107) ψ 2 (cid:54) C 2.
For all z, z (cid:48) ∈ B r , (cid:107) F ( z, X ) − F ( z (cid:48) , X ) (cid:107) ψ 2 (cid:54) C L d
( z, z (cid:48) )
Then, with probability at least 1 − ρ ,
[Equation]
Proof.
Deﬁne
[Equation]
By [42, Lemma 2.6.8], we have (cid:107) Y z (cid:107) ψ 2 (cid:54) C .
Hence we can apply a generalized Hoeffding’s inequality for subgaussian variables: with probability at least 1 − ρ ,
[Equation]
n
For any z 0 , we have
[Equation]
The second term is bounded by the inequality above.
For the ﬁrst term, we are going to use Dudley’s inequality “tail bound” version [42, Thm 8.1.6].
We ﬁrst need to check the sub-gaussian increments of the process Y z .
For any z, z (cid:48) ∈ B r , we have
[Equation]
where we have used, from [42], Prop. 2.6.1 for the ﬁrst line, Lemma 2.6.8 for the second, and the properties of F for the last.
Now, we apply Dudley’s inequality [42, Thm 8.1.6] to obtain that with probability 1 − ρ ,
[Equation]
which concludes the proof
[Equation]
Lemma 7.
Let x 1 , . . .
, x n be iid N 0 , Σ on R d , and W be a 1 -bounded, C -Lipschitz kernel in the ﬁrst variable with respect to
the metric (cid:107)·(cid:107) Σ − 1 .
With probability at least 1 − ρ ,
[Equation]
With probability at least 1 − ρ ,
[Equation]
Proof.
By the properties of Gaussian variables and a union bound, with probability at least 1 − ρ ,
[Equation] [Equation]
Now, since W is bounded, W ( x, X ) is subgaussian for any x .
Applying Lemma 6 with F = W and considering that (cid:107)·(cid:107) ψ 2 (cid:54) (cid:107)·(cid:107) ∞ , we get that with probability at least 1 − ρ ,
[Equation]
Combining with (30), we get (28).
Now, we write
[Equation]
We aim to apply again Lemma 6 for the function F (( x, u ) , x (cid:48) )
=
W ( x, x (cid:48) ) u (cid:62) x
(cid:48) and the metric (cid:107) x (cid:107) Σ − 1 + (cid:107) u (cid:107) Σ .
First, for any u with (cid:107) u (cid:107) Σ (cid:54) 1 , u (cid:62) X is Gaussian with variance less than 1 , so (cid:13)(cid:13) W ( x, X ) u (cid:62) X
(cid:13)(cid:13)
ψ 2 (cid:54) (cid:107) W ( x, · ) (cid:107) ∞ (cid:13)(cid:13) u (cid:62)
X (cid:13)(cid:13)
ψ 2 (cid:46) 1 .
Similarly, ( u − u (cid:48) )
(cid:62) X is Gaussian with variance (cid:107) u − u (cid:48) (cid:107) Σ , so
[Equation]
Hence, we get sup
[Equation]
which concludes the proof
[Equation]