A PAC-Bayesian Generalization Bound for Equivariant Networks
Abstract
Equivariant networks capture the inductive bias about the symmetry of the learning task by building those symmetries into the model.
In this paper, we study how equivariance relates to generalization error utilizing PAC Bayesian analysis for equivariant networks, where the transformation laws of feature spaces are determined by group representations.
By using perturbation analysis of equivariant networks in Fourier domain for each layer, we derive norm-based PAC-Bayesian generalization bounds.
The bound characterizes the impact of group size, and multiplicity and degree of irreducible representations on the generalization error and thereby provide a guideline for selecting them.
In general, the bound indicates that using larger group size in the model improves the generalization error substantiated by extensive numerical experiments.
Introduction
Equivariant networks are widely believed to enjoy better generalization properties than their fullyconnected counterparts by including an inductive bias about the invariance of the task.
A canonical example is a convolutional layer for which the translation of the input image (over R 2 ) leads to translation of convolutional features.
This construction can be generalized to actions of other groups including rotation and permutation.
Intuitively, the inductive bias about invariance and equivariance helps to focus on features that matter for the task, and thereby help the generalization.
Recently, many works provide theoretical support for this claim mainly for general equivariant and invariant models.
In this work, however, we focus on a class of equivariant neural networks built using representation theoretic tools.
As in Shawe-Taylor and Wood [1996], Cohen and Welling [2016b], Weiler et al. [2018a], this approach can build general equivariant networks using irreducible representations of a group.
An open question is about how to design feature spaces and combine representations.
Besides, the question of impact of these choices on generalization remain.
On the other hand, while it is rather well understood how to build a G equivariant model when G is ﬁnite, the choice of the group G to consider is not always obvious.
Indeed, using a subgroup H < G smaller than G is usually sufﬁcient to achieve signiﬁcant improvements over a non-equivariant baseline.
Moreover, the symmetries G of the data are often continuous, which means G -equivariance can not be built into the model using a group convolution design as in Cohen and Welling [2016a].
In such cases, the most successful approaches approximate it with a discrete subgroup H < G , e.g. see Weiler and Cesa [2019] for the G = O (2) case.
In addition, the choice of the equivariance group H can affect either the complexity or the expressiveness of the model and, therefore, its generalization capability, depending on how the architecture is adapted.
For instance, if one can not freely increase the model size, using a larger group H only increases the parameters sharing but reduces the expressiveness of the model.
For this reason, using the whole group G might not be beneﬁcial even when feasible.
In that line, we consider how the choice of group size affects generalization.
Contributions In this paper, we utilize PAC Bayes framework to derive generalization bounds on equivariant networks.
Our focus is on equivariant networks built following Weiler et al.
[2018a],
Cohen and Welling [2016b].
We combine the representation theoretic framework of Weiler and
Cesa [2019] with PAC-Bayes framework of Neyshabur et al.
[2018] to get generalization bounds on equivariant networks as a function of irreducible representations (irreps) and their multiplicities.
Different from previous PAC-Bayes analysis, we derive the perturbation analysis in Fourier domain.
Without this approach, as we will show, the naive application of Neyshabur et al.
[2018] lead to degenerate bounds.
As part of the proof, we derive a tail bound on the spectral norm of random matrices characterized as direct sum of irreps.
Note that for building equivariant networks, we need to work with real representations and not complex ones, which are conventionally studied in representation theory.
The obtained generalization bound provide new insights about the effect of design choices on the generalization error veriﬁed via numerical results.
To the best of our knowledge, this is the ﬁrst generalization bound for equivariant networks with an explicit equivariant structure.
We conduct extensive experiments to verify our theoretical insights, as well as some of the previous observations about neural networks.
Related Works
Equivariant networks.
In machine learning, there has been many works on incorporating symmetries and invariances in neural network design.
This class of models has been extensively studied from different perspectives Shawe-Taylor [1993, 1989], Kondor and Trivedi [2018], Cohen et al. [2018], Mallat [2012], Dieleman et al.
[2016], Cohen and Welling [2016a,b], Worrall et al.
Weiler et al.
[2018b,a], Thomas et al.
[2018], Bekkers et al.
[2018], Weiler and Cesa [2019], Bekkers [2020], Defferrard et al.
[2019], Finzi et al.
[2020], Weiler et al.
[2021], Cesa et al.
[2022] to mention only some.
Indeed, the most recent developments in the ﬁeld of equivariant networks suggest a design which enforces equivariance not only at a global level Laptev et al.
[2016] but also at each layer of the model.
An interesting question is how a particular choice for designing equivariant networks affect its performance and in particular generalization.
A related question is whether including this inductive bias helps the generalization.
The authors in Sokoli´c et al.
[2017a] consider general invariant classiﬁers and obtain robustness based generalization bounds, as in Sokoli´c et al.
[2017b], assuming that data transformation changes the inputs drastically.
For ﬁnite groups, they reported a scaling of generalization error with 1 / p | H | where | H | is the cardinality of underlying group.
In Elesedy [2022], the gain of invariant or equivariant hypotheses was studied in PAC learning framework, where the gain in generalization is attributed to shrinking the hypothesis space to the space of orbit representatives.
A similar argument can be found in Sannai et al.
[2021], where it is shown that the invariant and equivariant models effectively operate on a shrunk space called Quotient
Feature Space (QFS), and the generalization error is proportional to its volume.
They generalize the result of Sokoli´c et al.
[2017a] and relax the robustness assumption, although their bound shows suboptimal exponent for the sample size.
Similar to Sokoli´c et al.
[2017a], they focus on scaling improvement of the generalization error with invariance and do not consider computable architecture dependent bounds.
The authors in Lyle et al.
[2020] use PAC-Bayesian framework to study the effect of invariance on generalization, although do not obtain any explicit bound.
Bietti and Mairal [2019] studies the stability of models equivariant to compact groups from a RKHS point of view, relating this stability to a Rademacher complexity of the model and a generalization bound.
The authors in Elesedy and Zaidi [2021] provided the generalization gain of equivariant models more concretely and reported VC-dimension analysis.
Subsequent works also considered the connection of invariance and generalization Zhu et al.
In contrast with these models, we consider the equivariant networks with representation theoretic construction.
Beyond generalization, we are interested in getting design insights from the bound.
Generalization error for neural networks.
A more general study of generalization error in neural networks, however, has been ongoing for some time with huge body of works on the topic.
We refer to some highlights here.
A challenge for this problem was brought up in Zhang et al.
[2017], where it was shown using the image-net dataset with random labels, that the generalization error can be arbitrarily large for neural networks, as they achieve small training error but naturally cannot generalize because of random labels of images. Therefore, any complexity measure for neural networks should be consistent with the above observation.
As a result, uniform complexity measures like VC-dimension do not satisfy this requirement as they provide a uniform measure over the whole hypothesis space.
In this light, recent works related the generalization errors to quantities like margin and different norms of weights for example in, among others, Wei and Ma [2019], Sokoli´c et al.
[2017b], Neyshabur et al.
[2018], Arora et al.
[2018], Bartlett et al.
[2017], Golowich et al.
[2018], Dziugaite and Roy [2018a], Long and Sedghi [2019], Vardi et al.
[2022], Ledent et al.
[2021], Valle-Pérez and Louis [2020].
Some of these bounds are still vacuous or dimension-dependent (see Jiang et al.
[2020] for detailed experimental investigation and Nagarajan and Kolter [2019], Koehler et al.
[2021], Negrea et al.
[2021] for follow-up discussions on uniform complexity measures).
Class of convolutional neural networks are particularly relevant as they can be considered as a special case of equivariant models.
These models are discussed in Pitas et al. [2019], Long and Sedghi [2019], Vardi et al.
Ledent et al.
We choose PAC Bayesian framework for deriving generalization bounds.
There are many works on PAC Bayesian bounds for neural networks Neyshabur et al.
[2018], Biggs and Guedj [2022], Dziugaite and Roy [2017, 2018b,a], Dziugaite et al.
[2020] ranging from randomized and deterministic bounds to choosing priors for non-vacuous bounds.
Our method combines the machinery of Neyshabur et al.
[2018] with representation theoretic tools.
Background 3.1 Preliminaries and notation
We ﬁx some notations ﬁrst.
We consider a classiﬁcation task with input space X and output space
Y .
The input is assumed to be bounded in ` 2 -norm by B .
The data distribution, over X ⇥ Y , is denoted by
D .
The hypothesis space, H , consists of all functions realized by a L -layer general neural network with 1 -Lipschitz homogeneous activation functions 3 .
The network function, f W ( · ) with ﬁxed architecture, is speciﬁed by its parameters W :=
( W 1 , . . .
, W L ) .
The operations on W are extended from the underlying vector spaces of weights.
For any function f ( · ) , the k ’th component of the function is denoted by f ( x )[ k ] throughout the text.
For a loss function L :
H ⇥ X ⇥ Y !
R , the empirical loss is deﬁned by ˆ L ( f ) :
= 1 m
P mi =
[Equation]
L ( f, x i , y i )) and the true loss is deﬁned by
L ( f ) =
E ( x ,y )
⇠ D [ L ( f, x , y ))] .
For classiﬁcation, the margin loss is given by
[Equation]
The true loss of a given classiﬁer is then given by L ( f W ) =
L 0 ( f W ) .
The empirical margin loss is similarly deﬁned and denoted by ˆ L  .
In this work, we consider invariance with respect to transformations modeled as the action of a compact group G .
See Supplementary 7 for a brief introduction to group theory and the concepts that we will use throughout this paper.
3.2 Equivariant Neural networks
We provide a concise introduction to equivariant networks here with more details given in Supplementary 8.
We adopt a representation theoretic framework for building equivariant models.
Example.
Consider a square integrable complex valued function f with bandwidth B deﬁned on 2D-rotation group SO (2) and represented using its Fourier series as f ( ✓ ) =
P
Bn =0 a n e in ✓ .
Consider an equivariant linear functional of f mapping it to another function on SO (2) .
In Fourier space, the linear transformation is given by Wa with a as the Fourier coefﬁcient vector a = ( a 0 , . . .
, a B ) .
The action of rotation group on the input and output space is simply given by f ( ✓ ) !
f ( ✓  ✓ 0 ) with ✓ 0 as the rotation angle, or equivalently in Fourier space as a linear transformation a ! ⇢ ( ✓ 0 ) a with ⇢ ( ✓ 0 ) =
diag (1 , e  i ✓ 0 . . .
e  iB ✓ 0 ) .
Since it holds for all a , Equivariance condition means
W ⇢ ( ✓ 0 ) = ⇢ ( ✓ 0 ) W for all ✓ 0 , which implies that W should be diagonal.
A general feature space can be built by stacking multiple linear transformations W , and more ﬂexibly, by letting the transformations acting only on some of the frequencies (for example, a transformation that acts only on
B for frequency machinery equivariant networks compact irreducible representations irreps
SO
General Model.
Consider an Multi-Layer Perceptron (MLP) with L layers with the layer l 2 [ L ] given by the matrix W l
[Equation]
The action of the group H on layer l is determined by a linear transformation ⇢ l , which is called the group representation on R c l (see section 8 for more details).
A group element h 2 H linearly transforms the layer l by the matrix ⇢ l ( h ) .
The matrix W l is equivariant w.r.t.
the representations ⇢ l and ⇢ l  1 acting on its input and output if and only if for all h 2 H , we have:
[Equation]
Representation theory provides a way of characterizing equivariant layers using Maschke’s theorem and Schur’s lemma.
The key step is to start from characterizing the irreducible representations (irreps) { } of the group H .
As we will see, irreps are closely related to generalization of Fourier analysis for functions deﬁned on H .
Maschke’s theorem implies that the representation ⇢ l decomposes into direct sum of irreps as ⇢ l = Q l ⇣L L m l, i = ⌘
Q  1 l , where m l, is the number of times the irrep is present in the representation ⇢ l , that is, its multiplicity.
Each is a dim ⇥ dim matrix, and the direct sum is effectively a block diagonal matrix with matrix irreps on the diagonal.
Through this decomposition, we can parameterize equivariant networks in terms of irreps, that is in Fourier space.
Deﬁning
[Equation]
l  1 , the equivariance condition writes as
[Equation]
The block diagonal structure of [Equation] induces a similar structure on c
W l (see Figure 6).
By c
W l
[Equation]
2 , j [Equation] 1 , i ) denote the block in c
W that relates i ’th instance of 1 to j ’th instance of 2 , with i 2 [ m l  1 , 1 ] and j 2 [ m l, 2 ] , as:
[Equation]
Schur’s lemma helps us to characterize the equivariant kernels.
Note for neural network implementation, we need to work with a version of Schur’s lemma for real-valued representations given in
Supplementary 8. Schur’s lemma implies that if 1 6 = 2 , the block needs to be zero to guarantee equivariance.
Otherwise, it needs to have one of the three forms in Supplementary 8, depending on the type of 1 .
To simplify the notation, we write c
W l ( , j, , i ) as c
W l ( , j, i ) .
As it is shown in
Supplementary 8, the matrix c
W l ( , j, i ) can be written as the linear sum c
[Equation]
[Equation]
where { [ w l,i,j ( ) k } c k =1 are learnable parameters with ﬁxed matrices B l, ,i,j,k .
The value of c is either 1,2 or 4.
The structure of B l, ,i,j,k changes according to each type c .
Each layer is parameterized by the matrices c
W l ( ) , which include c parameters [ w l,i,j ( ) 2 R c with i 2 [ m l  1 , ] and j 2 [ m l, ] .
Therefore, for each layer, there will be
P m l, m l  1 , c parameters with the width of the layer l given as c l =
P m l, dim .
Note that, if x l = W l x l  1 , then b x l = c
W l b x l  1 , where b x l = Q  1 l x l .
The non-linearities should satisfy equivariance condition to have full end-to-end equivariance.
An important question is which representation ⇢ l of H should be used for each layer.
For the input, the representation is ﬁxed by the input space and its structure.
However, the intermediate layers can choose ⇢ l by selecting irreps and their multiplicity.
We explain how this choice impacts generalization bounds.
We consider a general equivariant network denoted by f W with the parameters of l ’th layer given by [ w l,i,j ( ) for all irreps , i 2 [ m l  1 , ] and j 2 [ m l, ] .
PAC-Bayesian bound
We derive PAC-Bayes bounds for equivariant networks f W presented in the previous section.
Although the proof follows a similar strategy as in Neyshabur et al.
[2018], it differs in two important aspects.
First, the weights of equivariant MLPs have speciﬁc structure requiring reworking proof steps.
Second, we carry out the analysis in Fourier domain.
In Supplementary 11, we also show that naively applying norm-bounds of Neyshabur et al.
[2018] cannot explain the generalization behaviour of equivariant networks.
A simple version of our result is given below.
Theorem 4.1 (Homogeneous Bounds for Equivariant Networks) .
For any equivariant network, with high probability we have:
[Equation]
where ⌘ 2 (0 , 1) and
[Equation]
The complete version of the theorem is given in Theorem 9.1.
We comment brieﬂy on the proof steps.
PAC-Bayesian generalization bounds start with deﬁning a prior P and posterior Q over the network parameters.
The network is drawn randomly using Q , which is also used to evaluate the average loss and generalization error.
The PAC-Bayesian bound provides a bound on the average generalization error of networks randomly drawn from Q (Section 4.1).
The next step is to de-randomize the bound and to obtain the generalization error for a speciﬁc instance f W 2 H .
Among different strategies, we follow the perturbation based method of Neyshabur et al.
The main idea is to carefully deﬁne a posterior Q , such that the networks randomly drawn from Q have small distance with the speciﬁc f W , and therefore the loss averaged over Q can be used to bound the loss of f W .
One way to choose Q is to deﬁne a Gaussian distribution around the parameters W and choose the variance to control the output perturbation (Section 4.2).
In what follows we provide more details about these steps.
4.1 PAC-Bayesian Bounds for Randomized Networks
The starting point is the PAC-Bayes theorem Langford and Shawe-taylor [2002], McAllester [1998], which provides a generalization bound on randomized classiﬁers.
If the function is chosen randomly from a distribution Q , then deﬁne the average losses as L ( Q ) =
E f ⇠ Q L ( f ) and ˆ L ( Q ) =
E f ⇠ Q ˆ L ( f ) .
We use the following version from Germain et al.
Lemma 4.2.
For any hypothesis space H , let P be a probability distribution on H .
Then for all  2 (0 , 1] , with probability 1   , for all Q on H , we have:
[Equation]
where ⇠ ( m ) :=
[Equation]
In PAC-Bayes vernacular, the distributions P and Q are called a prior and a posterior distribution on H .
This indicates that the distribution P and Q are chosen independently with Q typically chosen based on the training data and the obtained model, and P chosen independently based on any information available before training.
Choosing an appropriate prior plays a central role in getting non-vacuous bounds.
For instance in Dziugaite and Roy [2017], the prior P is chosen using a separate dataset not used during training.
Lemma 4.2 provides immediately a generalization bound for randomized equivariant classiﬁers.
The hypothesis space is parameterized by the network parameters W , which contains kernel parameters [ w l,i,j ( ) .
We choose a prior distribution P as zero-mean normal with covariance matrix  2 I over the parameters.
The posterior Q is chosen as normal distribution with the ﬁnal parameters [ w l,i,j ( ) as mean value and the same covariance matrix  2 I .
The KL-divergence in the PAC-Bayesian theorem is then given by:
[Equation]
The generalization bound depends depends on the sum of the norm of kernels.
In the next step, we de-randomize the bound to get a bound on the generalization error of f W .
4.2 De-Randomization and Perturbation Bounds
Our de-randomization technique follows closely that of Neyshabur et al.
We provide the sketch of derivations here, and the full proof is relegated to the supplementary materials.
To derandomize the previous bound, a common step is to choose Q such that the probability of margin violation is controlled.
Let S W be deﬁned as:
[Equation]
Any function on this set can change the margin of f W at most by  / 2 .
Therefore, for any h ,
L ( f W )  L  / 2 ( h ) , which implies L ( f W )  L  / 2 ( Q ) if Q is supported only on this set.
Similarly ˆ L  / 2 ( Q )  ˆ L  ( f W ) .
To choose Q , ﬁrst, we characterize the output perturbation of equivariant networks for a given input perturbation.
Next, we determine the variance  such that the output perturbation is bounded by  / 4 with probability 4 1 / 2 .
The ﬁrst step is the perturbation analysis.
Lemma 4.3 (Perturbation Bound) .
For a neural network f W ( · ) with input space of ` 2 -norm bounded by B , and for any weight perturbations U
= ( U 1 , . .
, U L ) , we have:
[Equation]
For equivariant networks, where the weights are parameterized as in eq.
2, the spectral norm of the perturbation U l is bounded as
[Equation]
The ﬁrst inequality is already given in Neyshabur et al.
The inequality 8 will be shown in supplementary materials.
The perturbation bound already suggests the beneﬁt of equivariant Kernels.
As it will be shown in
Supplementary materials, the equivariant kernels w.r.t.
satisfy
[Equation]
[Equation] which is the smallest possible norm (compare with general relation for dim ⇥ dim matrices: k W
[Equation]
1 p dim k W k F ). Therefore, the term 1 dim c
U l ( , j, i )
F in the perturbation bound is already tight.
The perturbation bound helps deﬁning a posterior Q by adding zero-mean Gaussian perturbations [ u l,i,j ( ) k of variance  2 to the learnable parameters [ w l,i,j ( ) k .
The output perturbation is given by the above theorem in terms of norm of U i .
The following lemma controls the norm of this random matrix, and thereby the output perturbation.
Lemma 4.4.
Consider a random equivariant matrix U l deﬁned by i.i.d.
Gaussian N (0 ,  2 ) choice of [ u l,i,j ( ) k in eq.
We have:
[Equation]
The proof is given in supplementary materials.
With Lemma 4.4, we can choose the variance  2 such that the output perturbation does not violate the intended margin.
Using an union bound over all layers, one ﬁnds that by choosing t = log ⇣
[Equation]
l
[Equation]
⌘ , with probability at least 12 the following inequality holds for every layer l 2 [ L ] :
[Equation]
We can use this bound jointly with the perturbation bound eq.
7 to choose  such that the margin is below  / 4 with high probability.
Note that the  chosen in this way is a function of learned weights.
This cannot be the case because of the prior P should be ﬁxed prior to learning.
A trick for circumventing this issue is to select many priors that can adequately cover the space of possible weights, and take the union bound over it.
The complete proof of the theorem is given in Supplementary 9, where we also consider special cases of this generalization bound for group convolutional networks Cohen and Welling [2016a].
Remarks on the generalization error from the theory.
The generalization error of Theorem 4.1 contains multiple terms.
First, the term ⌘ is a hyperparameter, which will be ﬁxed to 1 / 2 for the rest.
The term M ( l, ⌘ ) encodes the multiplicity and type of used irreps and their impact on the generalization error.
Let the hidden layer dimension c l =
P dim m l, be ﬁxed to a constant.
For Abelian groups, all real irreps are at most 2-dimensional, therefore, restricting the network to 2-dimensional irreps, we have:
[Equation]
The bound can be controlled further by appropriate choice of m l  1 , , m l, , c , which favors smaller multiplicity and type c .
Therefore, the bound favors using different irreps instead of repeating them.
For non-abelian cases, irreps can have larger dim , which keeping the layer dimension c l ﬁxed, amounts to smaller multiplicity.
This can improve the sum
P
Ll
[Equation]
P m l, in M ( l, ⌘ ) and indicates a potential improved generalization for non-abelian groups.
For dihedral groups, which is in general non-abelian, the real irreps are also at most two dimensional, however, of type c = 1 .
Note that for ﬁnite groups, there is a connection between irrep degrees and group order | H | =
P dim 2 /c .
The impact of group size manifests itself in the whole generalization error term.
We numerically verify this in the experimental results.
The inverse dependence on margin  is expected and similar to other works for instance Neyshabur et al.
The rest of the terms contain different norms of neural network kernels.
We discuss them more in connection with Neyshabur et al.
Comparison to Neyshabur et al.
There are two main differences with the bound in Neyshabur et al.
First, their bound has a norm dependence of ˜ O ⇣
[Equation]
⌘ , while in our result, the Frobenius norm includes an additional scaling of 1 / dim in
[Equation]
F / dim .
This term is strictly smaller in our bound for dim > 1 .
The second difference is the term (
P l M [Equation] l, ⌘ )) 2 , which is replaced with L 2 h
ln( Lh ) where h is the layer width.
In our case, the layer width is c l =
P dim m l, .
Choosing all layer widths c l equal to h , we recover the ln( Lh ) term from Neyshabur et al.
[2018] in our bound.
The remaining term is O [Equation] l p m l  1 , m l, c  ⌘ .
As an example, assume the same multiplicity m l, = m l and same dimension dim = dim for all irreps, that is, h
= N dim m l with N being the number of used irreps.
Then, we get a term as O  h 2 L 2 / ( N dim) .
As long as the multiplicity m l is smaller than N dim , i.e. number of used irreps times their dimension, our bound is strictly tighter.
Intuitively, this means that using more diverse irreps leads to better generalization.
See Section 11 for more detailed discussions.
Experiments
In this section, we also validate the theoretical insights derived in the previous section.
More experiments and discussions are included in Supplementary 12.
We have used datasets based on natural images and synthetic data.
In all experiments, we consider data symmetries G which are subgroups of the group O(2) of planar rotations and reﬂections.
This includes a number of ﬁnite/continuous and commutative/non-commutative groups, in particular C
N ( N rotations), D N ( N rotations and reﬂections), SO(2) (continuous planar rotations) and O(2) itself (see Supplementary 7).
Moreover, all models are based on group convolution over a ﬁnite subgroup H < G Cohen and Welling [2016a].
This means that the features of the layer l have shape | H | ⇥ c
Hl , that is, c
Hl channels of dimension | H | .
For each layer l , we keep the total number of features (approximatively) constant when changing the group H .
Models are trained until 99% of the training set is correctly classiﬁed with at least a margin  .
We used  = 10 in the synthetic datasets and  = 2 in the image ones.
First, we compare our result with the bound derived in Supplementary 11 using naive application of
Neyshabur et al.
In Fig.
2 and Fig.
3, we compare these two bounds on 6 synthetic datasets with O(2) and SO(2) symmetries (one per column).
Each dataset lives on high dimensional tori and O(2) (or SO(2) ) rotates the circles composing them, with different frequencies.
We consider different maximum frequencies F 2 { 1 , 2 , . .
, 6 } in the experiments.
For example, in Fig.
2, the ﬁrst plot on the left corresponds to a synthetic dataset with O(2) symmetry generating only using frequency 1.
See Supplementary 12.1 for more details and visualization of the synthetic data.
The results in Fig.
2 and 3 indicate that the bound built using the strategy from Neyshabur et al.
[2018] cannot explain the effect of equivariance on generalization.
Conversely, the bound in Theorem 4.1, which uses the parametrization in the Fourier domain, correlates with the measured generalization error, suggesting that it can account for the effect of different groups on generalization.
In addition, note that it also correlates with the saturation effect observed when using larger discrete groups H to approximate the continuous group G = O(2) .
Indeed, on low frequency datasets, the bound tends to saturate faster as we increase the group size | H | .
Conversely, on high frequency datasets, choosing a larger group H results in larger improvements in the estimated bound.
Remark 5.1 . As it can be seen in our experiments in Fig 2 and 3, our bound is larger that the one in
Neyshabur et al. This is partially due to coarse approximation of Frobenius norm and the presence of the term h 2 in our bound.
This dependency could be alleviated if we increase the number of used irreps in the group convolutional network, namely the larger group size. It is an interesting research direction to explore the optimal dependency on the width for equivariant networks.
In Fig.
4, we perform a larger study on the transformed MNIST datasets to investigate the simultaneous effect of the group size | H | , data augmentation and training set size m .
For each training data size, the generalization error improves with larger equivariance group H .
We study the relation between the term
P l p
M ( l, ⌘ ) and the generalization error observed.
Note that this term depends only on the architecture design, i.e. it is data and training agnostic.
In Fig.
5, we show the correlation between these two quantities for two synthetic datasets. Both datasets have G = O(2) symmetry, but the data rotates with frequencies up to F = 1 in the ﬁrst case and F =in the second. We observe that the term P l p
M ( l, ⌘ ) correlates strongly with the generalization in the high-frequency dataset ( F = 6 ), where the generalization beneﬁts from increasing the size of the equivariance group.
Conversely, in the low-frequency case ( F = 1 ), we observe a saturation of the performance, as increasing the group size does not improve generalization.
In this setting, the term
P l p M ( l, ⌘ ) is only a weak predictor of generalization. From these observations, we interpret this term as an indicator of the expressiveness of a model. Particularly, when the irreps are chosen compatible with the dataset symmetries, the term
P l p M ( l, ⌘ ) provides a good guideline for the architecture choice.
However, similar to general neural networks, the generalization error is dependent on other factors as well, which can be for example norms of network weights.  Conclusion
We provided learning theoretic frameworks to study the effect of invariance and equivariance on the generalization error and provide guidelines for design choices.
The obtained bounds provide
[Figure]
[Figure]
useful insights and guidelines for understanding these architectures.
As limitation of our work, we show experimentally in supplementary materials that the obtained bounds do not decrease with larger training sizes m as expected.
Nevertheless, for a ﬁxed training set, the trend of the generalization bound correlates with the generalization error.