Generalization Analysis of Message Passing Neural Networks on Large Random Graphs
Anonymous Author(s)
AffiliationAddress email
Abstract
Message passing neural networks (MPNN) have seen a steep rise in popularity since theirintroduction as generalizations of convolutional neural networks to graph structured data,and are now considered state-of-the-art tools for solving a large variety of graph-focusedproblems.
We study the generalization error of MPNNs in graph classification and regression.
We assume that graphs of different classes are sampled from different random graph models.
We show that, when training a MPNN on a dataset sampled from such a distribution, thegeneralization gap increases in the complexity of the MPNN, and decreases, not only withrespect to the number of training samples, but also with the average number of nodes inthe graphs.
This shows how a MPNN with high complexity can generalize from a smalldataset of graphs, as long as the graphs are large.
The generalization bound is derived froma uniform convergence result, that shows that any MPNN, applied on a graph, approximatesthe MPNN applied on the geometric model that the graph discretizes.
Introduction
A graph is an abstract structure that represents a set of objects along with the connections that existbetween those objects.
In many important fields, such as chemistry, biology, social networks, orcomputer graphics, data can be described by graphs.
This has led to a tremendous interest in thedevelopment of machine learning models for graph-structured data in recent years.
A ubiquitous toolfor processing such data are graph convolutional neural networks (GCNNs), which extend standard
Euclidean convolutional neural networks (CNNs) to graph-structured data.
Most GCNNs used in practice can be described using the general architecture of Message Passing
Neural Networks (MPNNs) .
MPNNs generalize the convolution operator to graph domains by aneighborhood aggregation or message passing scheme.
By f ( t − 1) i denoting the feature of node i inlayer t − 1 and e j,i denoting edge features from node j to i , one layer in a message passing graphneural network is given by
[Equation]
where N ( i ) is the set of nodes connected to node i , AGG denotes a differentiable and permutationinvariant function, e.g., sum, mean, or max, and Ψ ( t ) and Φ ( t ) denote differentiable functions such as
MLPs (Multi-Layer Perceptrons)
[FL19].
MPNNs have shown state-of-the-art performance in many graph machine learning tasks such as nodeor graph classification.
As such, MPNNs had a tremendous impact to the applied sciences, withpromising achievements such as discovering a new class of antibiotics [SYS + 20], and has impactedthe industry with applications in social media, recommendation systems, and 3D reconstruction,among others (see, e.g., [YHC + 18, WHZ + 18, WZL + 18, MFE + 19, FML + 19]).
The practical success of MPNNs led to a significant boost in research aimed at understanding the theoretical propertiesof MPNNs.
See, e.g., the variational inference point of view of MPNNs [DDS16], and algorithmicalignment of MPNNs with combinatorial algorithms [XHLJ19, MRF + 19].
In this paper we study the generalization capabilities of MPNNs with mean aggregation in a graphclassification task.
Previous works developed generalization bounds that do not depend on anymodel of the data, namely, graphs in these works can be generated and labeled in any arbitrary way[STH18, GJJ20, LUZ21].
In this work, we consider a generative model for the graphs which istheoretically powerful and general on the one hand, and allows much tighter generalization boundson the other hand.
Formally, we are given pairs of graphs and graph signals x =
( G, f ) and a target output y , where( x , y ) are jointly drawn from a distribution µ G ( x , y ) .
The goal is to learn a MPNN Θ that approximates y by Θ( x ) .
For this, one uses a loss function L , which measures the discrepancy betweenthe true label y and the output of the MPNN Θ( x ) .
The aim of a machine learning algorithm is tominimize the statistical loss (also called expected loss)
[Equation]
In (data-driven) machine learning one has only access to a training set instead of knowing thedistribution µ G .
Namely, we consider a multi-graph setting, where the training set T =
( x i =( G i , f i ) , y i ) mi
=1 is a collection of m samples drawn i.i.d.
from the distribution µ G ( x , y ) .
Then,instead of minimizing the statistical loss, one minimizes the empirical loss, given by
[Equation]
The optimized MPNN then depends on the dataset, and is hence denoted by Θ T .
The generalizationerror is defined to be
[Equation]
One then usually bounds (2) by the uniform generalization error
[Equation]
where the supremum is taken over some space of MPNNs.
Bounds of GE typically take the form
GE 2 ≤ Cm q ( N ) , where C is a constant that describes the complexity of the model class (e.g., numberof parameters), m is the size of the training set, and q ( N ) is a constant that depends on the (average)size of the graphs.
For such bounds, see, e.g., VC-dimension based bounds [STH18], Rademachercomplexity based bounds [GJJ20], and PAC-Bayesian based bounds [LUZ21].
While in previous bounds from the literature q ( N ) either increases in N or in the average degree, inthis paper we develop a generalization bound that decays in the average number of nodes N .
Theidea is to treat the nodes of each graph as randomly sampled from some random graph model.
Inthis point of view, not only the different graphs x i are seen as random samples, but the union ofall nodes of all graphs comprise together the random samples of the empirical loss.
In the spirit of
Monte Carlo theory, such a point of view should lead to a decay of the error between the empiricaland statistical losses as N increases.
As opposed to graphs, nodes cannot be seen as independent, dueto the correlations entailed by the graph structure.
Hence, our analysis focuses on developing Monte
Carlo error bounds in a correlated nodes regime.
Since in our approach we model graphs as randomly sampled from underlying continuous models,we define the application of message passing neural networks, not only on graphs, but also onthe underlying space from which graphs are sampled.
We then formulate and prove the followingconvergence result, that we write here informally.
Let x =
( G, f ) be drawn from the model χ , thenwith high probability, we have for all MPNNs Θ
[Equation]
where N is the number of nodes in x and α > 0 .
Based on this convergence result, we are able provea generalization bound that decays in N .
1.1 Validity of the
Proposed Model
The random graph models in our work are graphons [Lov12] with associated graphon signals (see
Definition 2.3).
The main assumption in our analysis is that graphs that are sampled from the same graphon belong to the same class.
While this may seem like a limitation, it is actually a very mildand reasonable assumption.
It is well known that equivalence classes of isomorphic graphs can becharacterized by homomorphism densities [Lov67].
Namely, given two graphs G 1 , G 2 , if (and onlyif) for every simple graph F the number of homomorphisms from F to G 1 is equal to the numberof homomorphisms from F to G 2 , then G 1 is isomorphic to G 2 .
Graphon analysis relaxes thisobservation to a continuous similarity measure.
A sequence of graphs { G j } j ∈ N is said to convergein the graphon sense, if for every simple graph F the homomorphism densities of F in the graphs{ G j } j ∈ N converge to some value.
Graphs from such a sequence can be thought of as being similarin some sense which relaxes the combinatorial notion of graph isomorphism.
Moreover, for eachsuch converging sequence, there is a unique (up to some symmetry) limit object, called a graphon.
This graphon is also seen as a generative model for graphs in the respective sequence, where graphsare generated by randomly sampling the graphon (see Definition 2.3).
Now, since it is well knownthat MPNNs cannot distinguish between isomorphic graphs, it is also unreasonable to expect them toseparate two graphs that are sampled from the same graphon.
We hence assume that two graphs thatare sampled from the same graphon belong to the same class (but not necessarily vice versa).
Thisassumption allows us to derive a generalization bound that is much tighter than previously proposedbounds (see Figure 1 for comparison).
From a practical stance, our graphon assumption is reasonable since many graph models are specialcases of graphons, like Erd ˝ os–Rényi, stochastic block model, and random geometric graphs [Pen03].
Moreover, the decoder of a graph variational autoencoder [KW16] can be seen as a graphon.
1.2 Related Work
In this subsection we briefly survey different approaches for studying the convergence and generalization capabilities of GCNNs that were introduced in previous contributions.
We give a comparisonwith our results in Section 3.
In [LHB + 21], the authors introduce the notion of GCNN transferability – the ability to transfer a
GCNN between different graphs, which is closely related to generalization.
For example, [LIK19,
GBR20, KTD21] show that the output of spectral-based GCNNs is linearly stable with respect toperturbations of the input graphs.
[LHB + 21] prove that spectral-based methods are transferable undergraphs and graph signals that are sampled from the same latent space.
[KBV20, RGR21, RWR21,
MLK21] show that spectral-based GCNNs are transferable under graphs that approximate the samelimit object – the so called graphon.
In [STH18], the authors provide generalization bounds that are comparable to VC-dimension boundsknown for CNNs.
These bounds are improved in [GJJ20], which provides the first data dependentgeneralization bounds for MPNNs with sum aggregation that are comparable to Rademacher boundsfor recurrent neural networks.
[LUZ21] derive a generalization bound via a PAC-Bayesian approachthat is governed by the maximum node degree and spectral norms of the weights.
[VZ19a] considergeneralization abilities of single-layer spectral GCNNs for node-classification task and provide ageneralization bound that is directly proportional to the largest eigenvalue of the graph Laplacian.
Another paper of this flavor is [YFM + 21], showing that certain MPNNs (with sum aggregation) donot generalize from small to large graphs.
1.3 Main Contributions
We follow the route of [KBV20] and consider graphs as discretizations of continuous spaces in ouranalysis, called random graph models (RGM, see Definition 2.3).
We introduce a continuous versionof message passing neural networks – the realization of MPNNs on random graph models, whichwe call cMPNNs.
Such cMPNNs are seen as limit objects of graph MPNNs, when the number ofgraph nodes goes to infinity.
We prove, up to our knowledge, the first convergence result of the graph
MPNN to the corresponding cMPNN as the number of nodes increases, which is uniform in thechoice of the MPNN.
For the generalization analysis, we assume that the data distribution µ G represents graphs which arerandomly sampled from a collection of template RGMs, with a random number of nodes.
Using ourconvergence results, we can then prove that the generalization error between the training set and thetrue distribution is small.
Here, we give the following informal version of Theorem 3.3.
Theorem 1.1 (Informal version of Theorem 3.3) .
Consider a graph classification task with mtraining samples T =
( x i =
( G i , f i ) , y i ) mi =1 drawn i.i.d.
from the data distribution µ G ( x , y ) on ametric-measure space χ of dimension D χ .
Suppose that the size N of each graph in T is drawn froma distribution ν .
Then
[Equation]
The constant C represents the complexity of the hypothesis space of the network, via the Lipschitzconstants of the message and update functions and the depth of the MPNNs.
Theorem 3.3 shows how we can use fewer graphs m than model complexity C when training MPNNsif the graphs are sufficiently large.
Preliminaries
A weighted graph G =
( V, W , E ) with N nodes is a tuple, where V = { 1 , . . .
, N } is the node set.
The edge set is given by E ⊂ V × V , where ( i, j ) ∈
E if node i and j are connected by an edge.
W = ( w k,l ) k,l is the weight matrix, assigning the weight w i,j to the edge ( i, j ) ∈ E , and assigningzero if ( i, j ) is not an edge.
The degree d i of a node i is defined as d i = (cid:80) N j =1 w i,j .
If G is a simplegraph, i.e., a weighted graph with W ∈ { 0 , 1 } N × N , the degree d i is the number of nodes connectedto node i by an edge.
We define a graph signal f : V → R F as a function that maps nodes to theirfeatures in R F , where F ∈ N is the feature dimension.
The signal f can be represented by a matrixf = ( f 1 , . . .
, f N ) ∈ R N × F , where f i ∈ R F is the feature at node i .
We also call f a (graph) featuremap .
For a random variable Y distributed according to κ , and a function F of Y , we denote by E Y ∼ κ [ F ( Y )]the expected value of F ( Y ) .
Similarly, we denote by Var Y ∼ κ [ F ( Y )] the variance of F ( Y ) .
2.1 Message Passing Graph Neural Networks
Message passing graph neural networks (gMPNNs) are defined by realizing an architecture of amessage passing neural network (MPNN) on a graph.
MPNNs are defined independently of aparticular graph.
Definition 2.1.
Let T ∈ N denote the number of layers.
For t = 1 , . . .
, T , let Φ ( t ) :
R 2 F t − 1 → R H t − 1and Ψ
( t ) :
R F t − 1 + H t − 1 → R F t be functions that we call the message and update functions, where
F t ∈ N is called the feature dimension of layer t .
The corresponding message passing neural network(MPNN)
Θ is defined to be the sequence
[Equation]
The message and the update function in Definition 2.1 are often defined as multi-layer-perceptrons(MLPs).
In a MPNNs, messages are sent between nodes and aggregated.
An aggregation scheme isa permutation invariant function that takes the collection of features in the edges of each node andcomputes a new nodes feature.
In this paper, we consider MPNNs with mean aggregation .
Then, agMPNN processes graph signals by realizing a MPNN on the graph as follows.
Definition 2.2.
Let G =
( V, W ) be a weighted graph and Θ be a MPNN, as defined in Definition2.1.
For each t ∈ { 1 , . . .
, T } , we define the gMPNN Θ ( t ) G as the mapping that maps input graphsignals f
= f (0) ∈ R N × F 0 to the features in the t -th layer byΘ ( t ) G : R N × F 0 → R N × F t , f (cid:55)→ f ( t ) =
( f ( t ) i )
Ni =1 , where f ( t ) ∈ R N × F t are defined sequentially bym ( t )
i := 1 d i N (cid:88)
j = w i,j Φ ( t ) ( f ( t − 1) i , f ( t − 1) j ) f ( t ) i :=
Ψ ( t ) ( f ( t − 1) i , m ( t ) i ) , for every i ∈ V .
We call Θ G :=
Θ ( T ) G a message passing graph neural network (gMPNN) .
Given a MPNN Θ as defined in Definition 2.1, the output
Θ G ( f ) ∈
R N × F T is a graph signal.
Ingraph classification or regression, the network should output a single feature for the whole graph.
Hence, the output of a gMPNN after global pooling is a single vector Θ PG ( f ) ∈ R F T , defined byΘ PG ( f )
= 1 N N (cid:88) i = Θ G ( f ) i .
For brevity, in this paper we typically do not distinguish between a MPNN and its realization on agraph.
2.2 Random Graph Models
Let ( χ, d, µ ) be a metric-measure space, where χ is a set, d is a metric and µ is a probability Borelmeasure.
A kernel (also called a graphon ), is a measurable mapping W : χ × χ → R .
The points x ∈ χ ofthe metric space are seen as the nodes of a continuous model, and the kernel is seen as a continuousversion of a weight matrix.
Kernels are treated as generative graph models using the followingdefinition.
Definition 2.3.
A random graph model (RGM) on ( χ, d, µ ) is defined as a pair ( W, f ) of a kernel
W : χ × χ → R and a measurable function f : χ → R called a metric-space signal .
We definea random graph with corresponding node features ( G, f ) by sampling N i.i.d.
random points
X 1 , . . .
, X N from χ , with probability density µ , as the nodes of G .
The weight matrix W = ( w i,j ) i,jof G is defined by w i,j = W ( X i , X j ) for i, j = 1 , . . .
, N .
The graph signal f is defined byf i =
f ( X i ) .
We say that ( G, f ) is drawn from W , and denote ( G, f ) ∼ ( W, f ) .
2.3 Continuous Message Passing Neural Networks
Given a MPNN, we define continuous message passing neural networks (cMPNNs) that act on kernelsand metric-space signals f : χ → R F , by replacing the graph node features and the aggregationscheme in (2.2) by continuous counterparts.
Let W be a kernel.
We define the kernel degree of W atx ∈ χ by
[Equation]
Consider a message signal U : χ × χ → R H , where U ( x, y ) is interpreted as a message sent fromthe point y to the point x in χ .
We define the continuous mean aggregation of U by
[Equation]
Given the messages U ( x, y ) =
Φ( f ( x ) , f ( y )) , where Φ : R 2 F → R H , we have
[Equation]
By abuse of notation, we often denote in short Φ( f, f ) :=
Φ (cid:0) f ( · ) , f ( ·· ) (cid:1) .
By replacing mean aggregation by continuous mean aggregation in Definition 2.2, the same messageand update functions that define a graph MPNN can also process metric-space signals.
Definition 2.4.
Let W be a kernel and Θ be a MPNN, as defined in Definition 2.1.
For eacht ∈ { 1 , . . .
, T } , we define Θ ( t ) W as the mapping that maps the input signal to the signal in the t -thlayer by
[Equation]
where f ( t ) are defined sequentially by
[Equation]
W [Equation] [Equation] (6) and f (0) =
f : χ → R F 0 is the input metric-space signal.
We call Θ W :=
Θ ( T ) W a continuousmessage passing neural network (cMPNN) .
As with graphs, the output of a cMPNN Θ W on a metric-space signal f : χ → R F 0 is anothermetric-space signal
Θ W ( f ) : χ → R F T .
The output of a cMPNN after global pooling is a singlevector Θ PW ( f ) ∈ R F T , defined by Θ PW ( f ) =
(cid:82) χ Θ W ( f )( x ) dµ ( x ) .
2.4 Data Distribution for Graph Classification Tasks
In the following, we consider a training data T = (cid:0) x i =
( G i , f i ) , y i (cid:1) m i =1 of graphs G i , graph signalsf i , and corresponding values
y i that can represent the classes of the graph-signal pairs.
The trainingdata is assumed to be drawn i.i.d.
from a distribution µ G ( x , y ) that we describe next.
In this paper, we focus on classification tasks.
More precisely we have classes j = 1 , . . .
, Γ , eachrepresented by a RGM ( W j , f j ) on a metric-measure space ( χ j , d j , µ j ) .
In fact, we suppose thateach class corresponds to a set of metric spaces.
For example, a graph representing a chair can besampled from a template of either an office chair, a garden chair, a bar stool, etc., and each of theseis represented by a metric space.
For simplicity of the exposition, we however treat every templatemetric space as its own class.
This does not affect our analysis.
The distribution µ G ( x , y ) is defined via the following procedure of data sampling.
For sampling onegraph, first, choose a class with probability γ j , i.e., for ( x , y ) ∼ µ G and j = 1 , . .
, Γ , γ j = P ( y = j ) .
Independently of the choice of the class, choose the number of nodes N ∼ ν , where ν is a discretedistribution on N ∈ N .
After choosing a class y ∈ { 1 , . . .
, Γ } and the graph size N , a random graph( G, f ) ∼ ( W y , f y ) with N nodes is drawn from the space χ y with probability density of the nodes( µ y ) N .
The notation T ∼ µ m G describes a dataset T consisting of m samples ( x 1 , y 1 ) , . . .
, ( x m , y m )
drawni.i.d.
from µ G .
We refer to Subsection C.1 in the appendix for a detailed definition of the distributionµ G .
Convergence and Generalization of MPNNs
In this section, we provide our main results on convergence (Subsection 3.1) and generalization(Subsection 3.2) of MPNNs.
For z ∈ R F , we define ∥ z ∥ ∞ =
max j =1 ,...,F | z j | .
Given a metricspace ( Y , d Y ) , we define the infinity norm of a vector valued function g : Y → R F by ∥ g ∥ ∞ =max j =1 ,...,F ess sup y ∈Y | ( g ( y ))
j | .
The function g is called Lipschitz continuous if there exists aconstant L g ∈ R such that for all y, y ′ ∈ Y ,
[Equation]
If the domain Y is Euclidean, we always endow it with the L ∞ -metric.
We measure the error between the output of a continuous MPNN and a gMPNN after pooling asfollows.
Given a graph signal f ∈ R N × F and a metric-space signal f : χ → R F , both the graph andthe continuous MPNN map to the same output space, i.e, Θ PW ( f ) , Θ PG ( f ) ∈ R F T .
Namely, the outputdimension of Θ P is independent of the random graph model it is realized on and also independent ofthe graph.
Hence, we define the error to be the supremum norm ∥ Θ PW ( f ) − Θ PG ( f ) ∥ ∞ .
We define theε -covering numbers of the metric space χ , denoted by C ( χ, ε, d ) , as the minimal number of balls ofradius ε required to cover χ .
For every j = 1 , . . .
, Γ , we make the following assumptions, which hold for the remainder of thepaper.
We assume that there exist constants C χ j , D χ j > 0 such that
[Equation]
for every ε > 0 .
Denote D χ = max j D χ j and C χ = max j C χ j Such constants exist for everymetric space with finite Minkowski dimension (see Appendix A).
We assume that diam( χ j ) :=
sup x,y ∈ χ j { d ( x, y ) } ≤ 1 .
Further, we only consider kernels W j such that there exists a constantd min > 0 satisfying
[Equation]
where the kernel degree d W j is defined in (4).
We moreover assume that W j ( x, · ) and W j ( · , x )are Lipschitz continuous (with respect to its second and first variable, respectively) with Lipschitzconstant L W j for every x ∈ χ .
We also assume that the metric-space signal f j : χ → R F is
Lipschitz continuous.
Since the diameter of χ j is finite, this means that f j ∈ L ∞ ( χ ) .
We considerthe following class of MPNNs
[Equation]
3.1 Convergence
In this subsection we show that the error between the cMPNN and the according gMPNN decayswhen the number of nodes increases.
Theorem 3.1.
Let W : χ 2 → R be a Lipschitz continuous kernel with Lipschitz constant L W ,where the metric space χ satisfies (7) with respect to the constants C χ , D χ > 0 , and W satisfies (8).
Consider a graph ( G, f ) ∼ ( W, f ) with N nodes X 1 , . .
, X N drawn i.i.d.
from χ with probabilitydensity µ .
Then, for every Lipschitz continuous f : χ → R F ,
E
[Equation]
where C ′ is defined in Subsection B.2 of the appendix.
Remark 3.2.
The constant C ′ in Theorem 3.1 depends polynomially on the Lipschitz constants
L Φ ( l ) and L Ψ ( l ) of the message and update functions Φ ( l ) and Ψ ( l ) , on the so called formal biases∥ Φ ( l ) (0 , 0)
∥ ∞ and ∥ Ψ ( l ) (0 , 0) ∥ ∞ , on ∥ W ∥ ∞ , on the Lipschitz constant L W of W , on (cid:112) log( C χ ) + 261 (cid:112)
D χ , and on 1 d min , where the degree of the polynomial is T .
A regularization of these constants canalleviate the exponential dependency of the bound on T .
The proof of Theorem 3.1 is given in Subsection B.2 of the appendix.
Discussion and Comparison to other Convergence Results The work closest related to ourconvergence results is [KBV20], where the authors show convergence of a fixed spectral GCNN toits continuous counterpart with comparable regularity assumptions as in Theorem 3.1.
Our resultholds for MPNNs, which are more general than spectral GCNNs.
Moreover, our bound is uniformin the choice of the MPNN Θ .
This last property is essential for leveraging the convergence resultto derive a generalization error.
Indeed, using the bound from [KBV20], for each MPNN Θ thereis a different high probability event E Θ where the convergence error is small.
However, the trained
MPNN Θ = Θ T depends on the dataset T and cannot be fixed in the analysis.
Hence, we wouldneed to intersect all events (cid:84) Θ E Θ to guarantee a small convergence error of the trained network Θ T ,which would not result in an event of high probability.
3.2 Generalization
In this subsection, we state the main result of our paper, which provides a non-asymptotic bound onthe generalization error of MPNNs, as defined in (3).
We consider a graph classification task witha training set T =
( x i =
( G i , f i ) , y i ) mi =1 and Γ classes.
The graphs and graph features in T aredrawn i.i.d.
from a probability distribution µ G ( x , y ) as described in Subsection 2.4.
We recall thatthe distribution that samples the size of the graph is denote by ν .
Given a MPNN with pooling, Θ P , and its output dimension R F T , we consider a non-negative lossfunction L : R F T × { 1 , . . .
, Γ } → [0 , ∞ ) .
Additionally, we assume that L is Lipschitz continuouswith Lipschitz constant L L .
Note that although the cross-entropy loss, a popular choice for lossfunction in classification tasks, is not Lipschitz-continuous, cross-entropy composed on softmax is.
Theorem 3.3.
There exists a constant C > 0 such that
[Equation]
[Table]
where C is specified in Subsection C.2 of the appendix.
The proof of Theorem 3.3 is given in Subsection C.2 of the appendix.
Remark 3.4.
The constant C in Theorem 3.3 represents the complexity of the class Lip L,Band can be bounded similarly to the constant C ′ from Theorem 3.1, as described in Remark3.2.
We summarize its dependencies on the parameters of the MPNN and the RGM by √ C ≲
[Equation]
+ [Equation] L W j ∥ W j ∥ T ∞ and refer to Subsection C.3 of the appendix for more details. Similarly to Remark 3.2 the exponential dependency of the constant C in
Theorem 3.3 on the depth T and the polynomial dependency on the uniform Lipschitz bound L canbe alleviated by regularizing the latter. We also note that the exponential dependency on the numberof classes Γ in Theorem 3.3 can be eliminated by assuming that the data is representative, i.e., if thenumber of training samples that fall into class j = 1 , . . . , Γ is deterministically γ j m .
The term
[Equation]
in Theorem 3.3 does not depend on the model complexity and is typically muchsmaller than the second term.
Hence, it does not affect bias–variance tradeoff considerations, and canbe ignored in the situation where m ≫ C E N ∼ ν
[log( N ) N − 1 Dχ +1 ] ≫ 1 .
Theorem 3.3 allows us tothink not just about graphs as samples, but also about individual nodes as samples.
However, nodesare correlated with their neighbors, and the higher the dimension D χ is, the larger the neighborhoodsare.
This is why the dependency on the number of nodes is N − 1 2( Dχ +1) and not N − 1 / 2 .
Still, thisdependency of the bound on N explains one way in which we train on less graphs than modelcomplexity and still generalize well.
Another insight is that the generalization bound becomes smallerthe smaller the Lipschitz constants of the message and update functions (see Remark 3.4).
Thisindicates that regularization methods like weight decay promote generalization.
Comparison to other generalization bounds in graph classification We compare our generalization bound with other generalization bounds derived by bounding the VC-dimension [STH18],the Rademacher complexity [GJJ20], and using a PAC-Bayesian approach [LUZ21].
We do notcompare with [VZ19b] since they derive generalization bounds for single-layered MPNNs in nodeclassification tasks.
Hence, the role of depth is unexplored.
Furthermore, their bound scales as
O ( λ 2 T max /m ) , where T is the number of SGD steps and λ max is the largest eigenvalue of the graph
Laplacian.
Hence, the generalization bound can increase monotonically for increasing T (see[LUZ21] for more details).
We summarize the comparison in Table 1 and provide more details,specially on the comparability, in Subsection C.4 of the appendix.
Our analysis derives a generalization bound on MPNNs that has essentially the same dependencyon the sample size m (up to a logarithmic factor), but does not directly depend on the number ofhidden units.
We emphasis that our bound depends on negative moments of the expected node size N .
In contrast, the VC-dimension based bound [STH18] scales as O (log( N ) N ) , the Rademachercomplexity based bound [GJJ20] scales as O ( d T − 1 (cid:112) log( d 2 T − 3 )) , and the PAC-Bayesian approachbased bound [LUZ21] scales as O ( d T − 1 ) , where d denotes the maximum node degree. [Figure]
Numerical Experiments
We give empirical evaluations of our generalization bound in comparison to the PAC-Bayesianbased bound [LUZ21] and the Rademacher complexity based bound [GJJ20].
We note that the VCdimension bound of [STH18] is written in O notations and hence cannot be quantitatively evaluated. We experiment on a synthetic dataset of 100K random graphs of 50 nodes, sampled from threedifferent RGMs: the Erdös-Rényi model (ERM) with edge probability 0 .
4 , a smooth version ofa stochastic block model (SBM), based on the kernel K ( x, y ) = sin(2 πx ) sin(2 πy ) / 2 π + 0 .
25on
[0 , 1] 2 , and a geometric graph with kernel K ( x, y ) = exp( −| x − y | 2 ) .
The correspondingsignals are given in Appendix D.2.1.
Each RGM represents one class in three binary classificationproblems, comparing all pairs of RGMs. For the MPNN we consider GraphSAGE [HYL17] withmean aggregation, and number of layers T = 1 , 2 or 3, implemented using Pytorch Geometric [FL19].
We consider a maximal hidden dimension of 128 . In Appendix D we give more details and alsoconsider synthetic data sampled from additional RGMs.
Our generalization bound becomes smaller the smaller the Lipschitz constants of the message andupdate functions are.
To control the Lipschitz constants, we consider two learning settings.
First, wetrain with weight decay regularization, which decreases the Lipschitz bounds, and second, we trainwith no regularization. For each setting (each choice of the number of layers and regularization) wetrain the MPNN, and read the resulting Lipschitz constants of the network.
We then plug all constantsinto our generalization bound formula (see Theorem C.7 in the appendix for the full formula), andinto the generalization bound formulas of the PAC-Bayes and Rademacher bounds (see Appendix C.4for the formulas).
The results are reported in Figure 1.
We observe that our generalization bounds areorders of magnitude smaller than the other works. In fact, theoretical generalization bounds typicallyteach us about the asymptotic behavior of generalization, and about the hyperparameters that affectgeneralization, but rarely give realistic numerical bounds (less than 1) that guarantee generalization.
Nevertheless, in one of the scenarios (one layer MPNN) our theory gives the bounds 0.08911 and0.13299 (respectively in the two datasets of Figure 1), which guarantees generalization in practice.
Conclusion
In this paper we proved that MPNNs with mean aggregation generalize from training to test datain classification tasks, if the graphs are sampled from RGMs that represent the different classes.
This follows from the fact that the MPNN on sampled graphs converges to the MPNN on the RGMwhen the number of nodes goes to infinity.
Our generalization bounds become smaller the larger thegraphs, which gives one explanation to how MPNNs with high complexity can generalize well from arelatively small dataset of large graphs.
We observe two main limitations of our current model.
First,the dependency of the generalization bound on the size of the graph N is O ( N − 1 2( Dχ +1) ) , which istypically slower than the observed decay in experiments (See Appendix D.1).
One potential futuredirection is to improve this dependency using a more sophisticated models of the trained networkand of the message and update functions.
Secondly, our model of the data is somewhat limited.
Onefuture direction is to allow deformations of the RGMs, to consider a continuum of RGMs instead of afinite set, and to consider sparse graphs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the conclusion in Subsection
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is
theoretical work with no foreseeable societal consequences
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
Yes
(b) Did you include complete proofs of all theoretical results?
[Yes] The appendix includescomplete proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
The experiments
are very simple which is why we decided to not include the code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
We did not perform training
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)?
[Yes] We report the standard error with respect to the randomness of our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
We did not perform training, i.e., the experiments were toy examples and not computation heavy.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We refer to [FL19]
for using Pytorch Geometric.
We do not use any existing code, data or model.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We do not use real data.
Only toy experiments on synthetic
data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
We do not use real data.
Only toy experiments on synthetic data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not crowdsource or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]