G ROUP S YMMETRY IN PAC L EARNING
A BSTRACT
In this paper we show rigorously how learning in the PAC framework with invariant or equivariant hypotheses reduces to learning in a space of orbit representatives.
Our results hold for any compact group, including inﬁnite groups such as rotations.
In addition, we show how to use these equivalences to derive generalisation bounds for invariant/equivariant models in terms of the geometry of the input and output spaces.
To the best of our knowledge, our results are the most general of their kind to date.
I NTRODUCTION
There is renewed interest in engineering models, such as neural networks, to be invariant or equivariant to the action of a group on their inputs.
The intuition being that if the relevant aspects of the task are unchanged (or change predictably) by the action of the group, then models that are hard-coded to be invariant (or equivariant) to this group will require fewer examples to learn.
Work in this area has garnered much attention (Cohen & Welling, 2016; Cohen et al., 2019) and enjoys application to domains where the symmetry is known a priori, such as quantum mechanics (Pfau et al., 2019).
We approach the study of symmetry in machine learning from a theoretical perspective, using concepts from statistical learning theory to give sample complexity guarantees.
We ﬁnd that, when the symmetry of the model is correctly speciﬁed (is also a symmetry of the target)
there is a reduction in sample complexity relative to an arbitrary model.
Moreover, the reduction in sample complexity (or, equivalently, the improvement in generalisation) for invariant/equivariant models is in terms of the geometry of a set of orbit representatives and a canonical collection of outputs, relative to that of the entire space of inputs and outputs.
Informally, if these canonicalised spaces are much smaller, then we expect a large sample complexity improvement from invariant/equivariant models.
This agrees with intuition: if the set of orbit representatives is much smaller than the whole space, then there is a lot of information in the group transformations that is not relevant to the task.
Contributions We begin in Section 2 with our technical setup and notation.
In Proposition 3.1, we show how to make precise the common intuition that learning with invariant hypotheses is equivalent to a reduced problem on a set of orbit representatives.
We go further in Section 4, giving the ﬁrst of our main results, Theorem 4.1, in which we show that learning with equivariant hypotheses depends only a space of canonicalised inputs and targets.
The second of our main results, Theorem 5.2, allows us to use the aforementioned equivalences to derive sample complexity improvements for equivariant models in terms of a measure of the geometrical quantities we alluded to previously.
All of this holds for arbitrary compact (possibly inﬁnite) groups.
Proofs are deferred to the appendix.
P RELIMINARIES
Notation and Technicalities We write X and Y for input and output spaces respectively and set Z
= X
× Y .
There will be a background probability space (Ω , σ Ω , P ) that is assumed to be rich enough to support our analysis.
The input and output spaces are assumed to be standard Borel spaces, which are Polish spaces with the Borel σ -algebra.
When dealing with real vector spaces, we use (cid:107)·(cid:107) 2 for the Euclidean norm and (cid:107)·(cid:107) ∞ for the component-wise max magnitude.
On a function f : X → R d , (cid:107) f (cid:107) L ∞ =
sup x ∈X (cid:107) f
( x ) (cid:107) ∞ .
Throughout the paper, G will represent an arbitrary Hausdorff, second countable and compact topological group that has measurable actions on both X and Y .
For an action φ of G on X this means that the map φ : G × X → X is measurable, the same goes for an action ψ on Y .
The action of G on Z is deﬁned by g ( x, y ) =
( φ ( g ) x, ψ ( g ) y ) .
We often write gx as a shorthand for φ ( g ) x and similarly for actions on Y and Z .
Our supervised learning setup will be deﬁned by ( X, Y ) , where X (resp.
Y ) is random element of X (resp.
Y ).
Deﬁnition 2.1.
A task is a tuple T =
( X, Y, (cid:96) )
where X is a random element of X , Y is a random element of Y and (cid:96) :
Y × Y → R + (the loss function) is integrable.
PAC Learning The PAC framework (Shalev-Shwartz & Ben-David, 2014, Deﬁnition 3.3), originally due to Valiant (1984), provides a precise deﬁnition of learning from data.
We use a distribution dependent relaxation of the agnostic formulation from Haussler (1992).
What we arrive at can be considered a form of uniform learning (Vapnik & Chervonenkis, 2015).
Let H be a user-speciﬁed class of measurable functions h : X → Y called the hypothesis class.
For example, H could be a neural network architecture with real parameters.
Let the loss function (cid:96) :
Y × Y → R + be integrable.
An algorithm alg : ∪ i ∈ N Z i
→ H is a measurable map that associates with each ﬁnite sequence of points of
Z a hypothesis from H .
We say that alg learns H with respect to a task T =
( X, Y, (cid:96) )
if ∃ m : (0 , 1) 2 → N such that ∀ (cid:15), δ ∈ (0 , 1) , if n ≥ m ( (cid:15), δ ) then P ( E [ (cid:96) ( h S ( X ) , Y ) | S ] ≤ inf h ∈H E [ (cid:96) ( h ( X ) , Y )]
+ (cid:15) ) ≥ 1 − δ where h S = alg
( S ) and S ∼ ( X, Y ) n i.i.d.
Suppose alg learns H with respect to T and let the set of all m satisfying the above be M , we deﬁne the sample complexity of alg on D as the point wise minimum m alg , T ( (cid:15), δ )
=
min m ∈M m ( (cid:15), δ ) .
Invariance and Equivariance
A function f : X → Y is G -invariant if f ( gx ) =
f ( x ) ∀ x ∈ X ∀ g ∈ G .
A function f : X → Y is G -equivariant if f ( gx ) =
gf ( x ) ∀ x ∈ X ∀ g ∈ G .
A hypothesis class is G -(invariant/equivariant) if all of its members are G -(invariant/equivariant).
A measure µ on X is G -invariant if ∀ g ∈ G and any µ -measurable B ⊆
X the pushforward of µ by the action G equals µ , i.e. ( g ∗ µ )( B ) = µ ( B ) .
This means that if X ∼ µ then gX ∼ µ ∀ g ∈ G .
When the group is clear from the context we will sometimes just say invariant/equivariant.
Invariant Algorithms
An algorithm alg is G -invariant if alg ( { ( g 1 x 1 , g 1 y 1 ) , . . .
, ( g n x n , g n y n ) } ) = alg ( { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } ) ∀ ( x i , y i )
∈ Z , ∀ g i ∈ G and ∀ n ∈ N .
If the hypothesis class is G -invariant and G acts trivially on Y , then any algorithm that depends on the training inputs only through the values of the hypotheses satisﬁes this property.
For instance, any form of ERM is covered by this.
Similarly, if the hypothesis class is G -equivariant and the algorithm depends on the data only through a loss that is preserved by G , meaning (cid:96) ( gy, gy (cid:48) )
= (cid:96) ( y, y (cid:48) )
∀ g ∈ G ∀
y, y (cid:48) ∈ Y , then it will be an invariant algorithm.
Once again, any form of ERM is covered.
Highlighting two cases: if the action of G is via an orthogonal representation, then (cid:96) ( y, y (cid:48) )
=
l ( (cid:104) y, y (cid:48) (cid:105) ) is preserved by G for any l .
The same goes for (cid:96) ( y, y (cid:48) )
=
l ( (cid:107) y − y (cid:48)
(cid:107) 2 ) .
Deﬁnition 2.2 ( ( H , G ) -equivalent) .
We say the tasks T and T (cid:48) are ( H , G ) -equivalent
if, for any G -invariant algorithm alg , alg learns H with respect to T if and only if alg learns H with respect to T (cid:48) , and the sample complexities are equal, i.e. m alg , T = m alg , T (cid:48) .
Intuitively, if two tasks are ( H , G ) -equivalent then they are equally ‘difﬁcult’ to learn with H .
2. O RBIT R EPRESENTATIVES
In the following, σ X is the Borel σ -algebra on X .
Deﬁnition 2.3 (Measurable cross-section (Eaton, 1989, Deﬁnition 4.1)) .
Let G be a group acting measurably on X .
The set X π ⊆
X is a measurable cross-section of X with respect to G if the following conditions hold: 1) X π is measurable.
2) X π contains exactly one element from each orbit of each point x ∈ X , say x π . 3)
The function π : X → X π deﬁned by π ( x ) =
x π is σ X -measurable when X π has the subspace σ -algebra { B ∩ X π : B ∈ σ X } .
We call π the projection .
A measurable cross-section provides a natural way of identifying a collection of orbit representatives with suitable measurability properties.
If G is compact and acts measurably on X (with respect to a Borel σ -algebra), then a measurable cross-section always exists (Bloem-Reddy & Teh, 2020).
Measurable cross-sections are not necessarily unique.
L EARNING WITH I NVARIANT M ODELS
Proposition 3.1 is a rigorous version of the common intuition that learning with a G -invariant model is equivalent to learning on a space of orbit representatives.
In addition, we show that these learning problems have the same sample complexity.
The class of orbit representatives is often of much smaller dimension or complexity than the input space, resulting in an improvement in sample complexity for invariant models.
Finally, note that our result requires no conditions on the data distribution, in particular there is no constraint for the input distribution to be invariant.
Proposition 3.1.
Let G act measurably on X and trivially on Y .
Let X π be a measurable cross-section of X and let π be its projection.
Let H be a hypothesis class of G -invariant functions, let alg : ∪ i ∈ N Z i → H be a G -invariant algorithm and let (cid:96) :
Y × Y → R + be and integrable loss function.
Then ( X, Y, (cid:96) ) and ( X π , Y, (cid:96) ) are ( H , G ) -equivalent, where
X π
= π ( X ) with π as in Deﬁnition 2.3.
Example 3.2.
Let X
= R 2 and Y = { 0 , 1 } .
Proposition 3.1 tells us that learning with a rotationally invariant hypothesis class, such as discs about the origin H =
{ ( x, y ) (cid:55)→ 1 (cid:8) x 2 + y 2 ≤ r (cid:9) : r ∈ R + } , is equivalent to learning on the reduced space X
= R + .
Example 3.3 (Deep Sets, (Zaheer et al., 2017)) .
Zaheer et al.
; Bloem-Reddy & Teh (2020) consider learning functions f : [0 , 1] M → R that are S M -invariant, where S M is the group of permutations on M elements.
It is shown that any such continuous function must be of the form f ( T )
=
ρ (cid:0)(cid:80) t ∈ T φ ( t ) (cid:1) where T ∈ [0 , 1] M and ρ, φ : R → R .
By Proposition 3.1 we can see, as was shown by Sannai et al.
(2021), that learning permutation invariant functions is equivalent to learning the same class of functions restricted to the domain { x 1 ≥ x 2 ≥ · · · ≥ x M ; x i ∈ [0 , 1] } .
In high dimensions this is a much smaller space than [0 , 1] M , it is a factor of M ! smaller in volume.
Example 3.4 (G-CNN (Cohen & Welling, 2016)) .
Cohen & Welling (2016) present a convolutional layer that is equivariant to various discrete groups of transformations.
These layers are used to generate features for an invariant classiﬁer.
Consider the simple case of the group p 4 of rotations about the origin in R 2 through an angle of π 2 .
Proposition 3.
shows that training a G-CNN based classiﬁer (e.g. by gradient descent on the empirical loss) that is p 4 -invariant is equivalent to learning the network restricted to a single quadrant of the plane.
L EARNING WITH E QUIVARIANT M ODELS
Before we present our ﬁrst main result, we introduce some additional assumptions, which we discuss in Appendix A.
Assumption 1 (Structure of the marginal P X ) .
We assume the existence of a probability distribution ν on G such that
X d = GX π where X ∼ P X , X π
= π ( X ) , G ∼ ν and G ⊥⊥ X π .
Essentially, this says that the orbit that X ∼ P X belongs to and where it is in the orbit are independent.
For intuition, if G = SO (2) acts by rotation on R 2 then any density f ( r, θ ) that is separable in polar co-ordinates as f ( r, θ ) =
f rad ( r ) f ang
( θ ) gives the required independence.
Assumption 2 (Functional representation of Y ) .
We assume that there exists a measurable, G -equivariant function f such that Y d = f ( X, η ) where η ∼ Unif[0 , 1] and η ⊥⊥ X .
2 Equivariance of f refers to its ﬁrst argument only f ( gX, η ) =
gf ( X, η ) .
A special case of this is an equivariant target function with independent additive noise ξ such that gξ d = ξ ∀ g ∈ G (e.g. orthogonal representation and Gaussian noise).
This is inspired by noise-outsourcing , which typically appears with almost sure equality rather than equality in distribution.
It is also known as transfer , see Kallenberg (2006, Theorem 6.10) and Bloem-Reddy & Teh (2020) for an application to groups.
Theorem 4.1.
Let G act measurably on both X and Y .
Let H be a hypothesis class of G -equivariant functions and let alg : ∪ i ∈ N Z i → H be a G -invariant algorithm.
Let X π be a measurable cross-section of X and let π be its projection.
Let ( X, Y ) satisfy Assumption 1 and Assumption 2.
Then the tasks ( X, Y, (cid:96) ) and ( X π , Y π , ¯ (cid:96) ) are ( H , G ) -equivalent,
where X π = π ( X ) , Y π
= f ( X π , η ) with f and η as in Assumption 2, and ¯ (cid:96) ( y, y (cid:48) )
= (cid:82) G (cid:96) ( gy, gy (cid:48) )
d ν ( g ) .
The loss function ¯ (cid:96) is the average of (cid:96) over the orbits of G , weighted by the probability of each g ∈ G .
In any instance in which (cid:96) is preserved by G , we have ¯ (cid:96) =
(cid:96) (see Section 2).
One can think of Y π as the canonical target corresponding to the canonical input X π .
The task ( X π , Y π , (cid:96) ) can be thought of as a canonical version of the original task, with any nuisance information from the group transformations removed.
Naturally, this canonical task depends the choice of cross-section.
Example 4.2 (Deep Sets, Zaheer et al.
Returning to Zaheer et al.
, the authors consider neural network layers f : R M → R M with f ( x ) =
σ (Θ x ) , where Θ ∈ R M × M and σ is an element-wise non-linearity.
It is shown that f is S M -equivariant if and only if Θ ij = aδ ij + b for scalars a, b ∈ R .
In this case X = Y = R M and S M acts on each space by permutation.
Assume that the marginal distribution on the inputs is exchangeable (we will see in Appendix A that this means that ν exists and is uniform on S M ), then Theorem 4.1 says that learning the class of equivariant f is equivalent to learning on the restricted domain { x 1 ≥ x 2 ≥ · · · ≥ x M ; x i ∈
X
} with the averaged loss function ¯ (cid:96) ( y, y (cid:48) )
= 1 M !
(cid:80) τ ∈ S M (cid:96) ( y τ , y (cid:48) τ ) where ( y τ ) i
=
y τ
( i ) and the same for y (cid:48) .
I MPLICATIONS FOR S AMPLE C OMPLEXITY
Theorem 5.2 links the generalisation error to the geometries of the input and output spaces.
Using invariant or equivariant hypotheses reduces the learning problem to one on a cross-section X π .
Often, X π will be smaller than X , and because the sample complexity depends a notion of the size of the input space, we get a reduction for invariant/equivariant models.
We use covering numbers for this notion of size.
Deﬁnition 5.1 (Covering, covering number) .
Let ( T, d ) be a pseudo-metric space and let U ⊂ T .
K ⊂ T is an (cid:15) -cover of U (with respect to d ) if ∀ u ∈ U ∃ k ∈ K with d ( u, k ) ≤ (cid:15) .
The (cid:15) -covering number of U is the smallest cardinality of all the (cid:15) -covers, i.e. Cov ( U ; d, (cid:15) )
=
inf K ∈K | K | where K is the set of all (cid:15) -covers of U .
Theorem 5.2.
Let X be a closed subset of a metric space ( T, ρ ) .
Let Y =
B d ( r ) be the closed Euclidean ball of radius r in R d .
Let H be a class of functions h : X → Y that are C -Lipschitz in the sense that (cid:107) h ( x ) − h
( x (cid:48) )
(cid:107) ∞
≤ Cρ ( x, x (cid:48) )
∀ x, x (cid:48) ∈ X .
Let S =
(( X 1 , Y 1 ) , . . .
, ( X n , Y n )) ∼ ( X, Y ) n be a training sequence drawn i.i.d.
for some random variables X, Y .
Then, for any (cid:15) ∈ (0 , 1) ,
[Equation]
where
[Equation]
The above result says, taking α = 1 / 2 for simplicity, that
[Equation]
examples are sufﬁcient to generalise.
The signiﬁcance of this for equivariant models is as follows.
Suppose that the loss is preserved by G , e.g. an orthogonal representation on Y and the Euclidean loss, and that ( X, Y ) satisfy Assumptionand Assumption 2.
In this setting, we can consider the change to the bound in Theorem 5.2 that arises from taking H to be G -equivariant.
Speciﬁcally, Theorem 4.1 tells us that the sample complexity of learning on the task T =
( X, Y, (cid:96) ) is the same as learning on the task T (cid:48)
=
( X π , Y π , (cid:96) ) .
This means that
[Equation]
examples are sufﬁcient to generalise for an equivariant model, potentially much less than above.
The quantity
D τ
( X , H ) is, in a sense, simultaneously measuring the size of X and the outputs of H at a scale τ .
The beneﬁt of equivariance depends on the geometry of X and X π .
Since X π ⊂ X we get D τ ( X π , H ) ≤ D τ
( X , H ) and, informally, the reduction will be large if X π is much smaller than X .
If H ( X π ) is much simpler than H ( X ) then the same applies.
R ELATED W ORK
We build on concepts from Eaton (1989); Bloem-Reddy & Teh (2020), which discuss group invariance in statistics and the representation of invariant/equivariant functions.
Implementations and Applications While there has been a recent surge in interest, symmetry has a long history in machine learning and it is not clear where it found its ﬁrst implementation.
Recent literature is dominated by neural networks, but other methods do exist: e.g. kernels (Haasdonk et al., 2005), support vector machines (Sch ¨ olkopf et al., 1996) or feature spaces such as polynomials (Schulz-Mirbach, 1994; 1992).
The engineering of invariant neural networks dates back at least to Wood & Shawe-Taylor (1996), in which ideas from representation theory are applied to ﬁnd weight tying schemes that result in group invariant architectures; similar themes are present in Ravanbakhsh et al.
There is much recent work that follows in this vein, borrowing ideas from fundamental physics to construct invariant/equivariant convolutional architectures (Cohen & Welling, 2016; Cohen et al., 2018).
Correspondingly, a sophisticated theory of invariant/equivariant networks has arisen (Kondor & Trivedi, 2018; Cohen et al., 2019) including universal approximation results (Maron et al., 2019; Yarotsky, 2018).
Learning and Generalisation
It was noted by Abu-Mostafa (1993) that constraining a model to be invariant cannot increase its VC dimension.
A heuristic argument for reduced sample complexity is made by Mroueh et al.
(2015) in the case that the input space has ﬁnite cardinality.
The sample complexity of linear classiﬁers with invariant representations trained on a simpliﬁed image task is discussed brieﬂy by Anselmi et al.
(2014), the authors conjecture that a general result may be obtained using wavelet transforms.
Sokolic et al.
(2017) use the idea of robustness Xu & Mannor (2012) to formulate a generalisation bound for interpolating large-margin classiﬁers that are invariant to a ﬁnite set of transformations, their results contain an implicit margin constraint on the training data.
Sannai et al.
(2021) consider the generalisation of models invariant or equivariant to ﬁnite permutation groups.
Each of Lyle et al.
(2019; 2020) cover the PAC Bayes approach to generalisation of invariant models, the latter also considers the relative beneﬁts of feature averaging and data augmentation.
Recently, Elesedy & Zaidi (2021) proved the ﬁrst strict beneﬁt in generalisation for equivariant models by taking a function space approach.
This was extended to kernel methods in Elesedy (2021).
Following a different approach, Mei et al.
(2021) study the generalisation of invariant random features and kernel methods by projecting into a space of high degree polynomials.
Zhu et al.
(2021) study the generalisation beneﬁt of invariance in terms of coverings of the training set.
Finally, the concurrent work of Shao et al.
(2022) also considers invariance in the PAC framework, but takes a different approach.
A CKNOWLEDGEMENTS
We thank Sheheryar Zaidi for helpful discussion in earlier stages of this project.
We thank Varun Kanade and Yee
Whye Teh for help and support through this and other projects.
We also thank Akiyoshi Sannai and Benjamin BloemReddy for helpful clariﬁcations of their work.
The author receives support from the UK EPSRC CDT in Autonomous
Intelligent Machines and Systems (grant reference EP/L015897/1)
A D ISCUSSION OF A SSUMPTIONS
First a remark on Assumption 1.
The distribution ν and the choice of cross-section are not independent.
In particular, under the map X π
(cid:55)→ g
X π we have ν
(cid:55)→ ν ◦ g − 1 .
The dependence goes away only if ν = ν ◦ g ∀ g ∈ G which by uniqueness means ν = λ , the Haar measure on G .
This fact is benign but worth being aware of.
We now justify Assumption 2.
First note that this assumption is much weaker than, for instance, the standard (equivariant) regression setup in which Y =
f ∗ ( X ) + ξ where ξ ⊥⊥ X , ξ is G -invariant and f ∗ is equivariant.
In particular, Assumption 2 allows for noise corruption that is not necessarily additive.
It is known that, under certain conditions, a functional representation for Y is equivalent to the conditional equivariance of Y , in the sense that gY | gX d = Y | X ∀ g ∈ G (Bloem-Reddy & Teh, 2020).
We describe this setting now.
Assume that the marginal for X is G -invariant, so X d = gX ∀ g ∈ G .
We adapt the following deﬁnition from BloemReddy & Teh (2020).
Deﬁnition (Representative Equivariant) .
Let G be a group acting freely on a set X .
A representative equivariant is an equivariant function τ : X → G .
That is, τ ( gx ) =
gτ ( x ) ∀ g ∈ G ∀ x ∈ X .
The following result from Bloem-Reddy & Teh (2020) then gives us Assumption 2.
This result is a form of noiseoutsourcing or transfer (Kallenberg, 2006, Theorem 6.10).
Theorem (Bloem-Reddy & Teh (2020, Theorem 9)) .
Let G be a compact group acting measurably on Borel spaces
X and Y , such that there exists a measurable representative equivariant τ : X → G (c.f. Appendix A).
Let X be a
G -invariant random element of X .
Then Y is conditionally G -equivariant
if and only if ∃ a measurable G -equivariant function f : X × [0 , 1] → Y such that Y a.s. =
f ( X, η ) where η ∼ Unif[0 , 1] and η ⊥⊥ X .
As it happens, if we assume that the marginal for X is G -invariant, so X d = gX ∀ g ∈ G , then Eaton (1989, Theorem 4.4) says that ∃ U ∼ λ such that X = UX π and U ⊥⊥ X π .
So this setup is sufﬁcient for Assumption 1 too.
B P ROOFS B. P ROOF OF P ROPOSITION 3.
Proposition (Proposition 3.1) .
Let G act measurably on X and trivially on Y .
Let X π be a measurable cross-section of X and let π be its projection.
Let H be a hypothesis class of G -invariant functions, let alg : ∪ i ∈ N Z i → H be a G -invariant algorithm and let (cid:96) :
Y × Y → R + be and integrable loss function.
Then ( X, Y, (cid:96) ) and ( X π , Y, (cid:96) ) are ( H , G ) -equivalent,
where X π = π ( X ) with π as in Deﬁnition 2.3.
Proof.
For any training set S , deﬁne S
π = { ( π ( x ) , y ) : ( x, y ) ∈ S } .
Clearly, if S ∼ ( X, Y ) n then S π ∼ ( X π , Y ) n .
Set h S = alg
( S ) and h S π = alg ( S π ) , which with the invariance of alg implies h S
=
h S π .
By invariance, h ( X ) =
h ( X π ) for any h ∈ H .
Together, this means that
[Equation]
We can then conclude that
[Equation]
which completes the proof.
In the last line, the probabilities are taken over the respective training samples.
B. P ROOF OF T HEOREM 4.
Theorem (Theorem 4.1) .
Let G act measurably on both X and Y .
Let H be a hypothesis class of G -equivariant functions and let alg : ∪ i ∈ N Z i → H be a G -invariant algorithm.
Let X π be a measurable cross-section of X and let π be its projection.
Let ( X, Y ) satisfy Assumption 1 and Assumption 2.
Then the tasks ( X, Y, (cid:96) ) and ( X π , Y π , ¯ (cid:96) ) are ( H , G ) - equivalent, where X π
= π ( X ) , Y π
= f ( X π , η ) with f and η as in Assumption 2, and ¯ (cid:96) ( y, y (cid:48) )
= (cid:82) G (cid:96) ( gy, gy (cid:48) )
d ν ( g ) .
Proof.
Let the training sets be S ∼ ( X, Y ) n and S π ∼ ( X π , Y π ) n .
We have from Assumption 1 that X d = GX π and from Assumption 2 that
[Equation]
So by the invariance of a we have h S :
= alg ( S ) =
alg ( S π ) =:
h S π and hence E [ (cid:96) ( h S ( X ) , Y )]
=
E [ (cid:96) ( h S π ( X ) , Y )] .
Then, for any equivariant hypothesis h ∈ H ,
[Equation]
where we applied Fubini’s theorem (Kallenberg, 2006, Theorem 1.27) and ¯ (cid:96) ( y, y (cid:48) )
= (cid:82) G (cid:96) ( gy, gy (cid:48) )
d ν ( g ) .
Note that the function (cid:96) y,y (cid:48) ( g ) =
(cid:96) ( gy, gy (cid:48) ) is measurable in g by Kallenberg (2006, Lemma 1.26), so ¯ (cid:96) exists.
Furthermore, the right hand side of the above must be ﬁnite because (cid:96) is integrable (c.f. the left hand side).
Putting everything together gives
[Equation]
which is exactly what’s required.
In the last line, the probabilities are taken over the respective training samples.
B. P ROOF OF T HEOREM 5.
Theorem (Theorem 5.2) .
Let X be a closed subset of a metric space ( T, ρ ) .
Let Y =
B d ( r ) be the closed Euclidean ball of radius r in R d .
Let H be a class of functions h : X → Y that are C -Lipschitz in the sense that (cid:107) h ( x ) − h
( x (cid:48) )
(cid:107) ∞ ≤
[Equation] ( x, x (cid:48) ) ∀ x, x (cid:48) ∈ X .
Let S =
(( X 1 , Y 1 ) , . . .
, ( X n , Y n )) ∼ ( X, Y ) n be a training sequence drawn i.i.d.
for some random variables X, Y .
Then, for any (cid:15) ∈ (0 , 1) ,
[Equation]
where
[Equation]
The basic idea of this proof is to use the Lipschitz property to bound coverings of the hypothesis class in terms of coverings of the input space.
Similar ideas appeared in (Sokolic et al., 2017; Sannai et al., 2021).
Proof.
Let E be a minimal δ -cover for X .
We construct a cover for H in terms of E .
Take h ∈ H and let h (cid:48) :
X → Y be C -Lipschitz.
Since X is closed, there is an x ∈ X such that (cid:107) h − h (cid:48)
(cid:107) L ∞
=
(cid:107) h
( x ) − h (cid:48)
( x ) (cid:107) ∞ .
Let x (cid:48) be an element of E such that d ( x, x (cid:48) ) ≤ δ , then
[Equation]
Deﬁne H ( x ) = { h ( x ) : x ∈ H} ⊂ Y .
For any x ∈ E let K x be a minimal κ -covering of H ( x ) in the vector norm (cid:107)·(cid:107) ∞ .
Let F x be the collection of constant functions
X → Y taking the values on K x
[Equation]
This means that the set { f ( x ) : x ∈ E, f ∈ F x } is a minimal κ -cover for H ( x ) in (cid:107)·(cid:107) ∞ .
Then the above derivations mean that (cid:83) x ∈ E F x is a minimal (2 Cδ + κ ) -cover for H in (cid:107)·(cid:107) L ∞ .
Thus,
[Equation]
Then for α ∈ (0 , 1) set δ = α(cid:15) 32 Crd and κ = α(cid:15) 16 rd and apply Proposition C.3 to get the conclusion.
C A DDITIONAL R ESULTS
We need an additional deﬁnition.
Deﬁnition C.1 (Packing, packing number) .
Let ( T, d ) be a pseudo-metric space and (cid:15) > 0 .
E ⊂ T is an (cid:15) -packing of
T (with respect to d ) if ∀ x, y ∈ E , d ( x, y ) > (cid:15) .
The (cid:15) -packing number is the largest cardinality of all the (cid:15) -packings, i.e. Pack ( T, d, (cid:15) ) =
sup E ∈E | E | where E is the set of all (cid:15) -packings.
If this supremum doesn’t exist, then we say the packing number is inﬁnite.
The following is a straightforward corollary of Proposition 3.1.
Corollary C.2 (Lipschitz classiﬁers with margin) .
Let X be a compact set and let F be a class of L -Lipschitz functions f : X → R .
The hypothesis class H is built by thresholding functions from F , e.g. by H
= { h = sign( f ) : f ∈ F} , and we insist that the output classiﬁer attains margin γ on any training set.
Then for any points x, x (cid:48) ∈ X with different labels we must have γ ≤ | f ( x ) − f
( x (cid:48) )
| ≤ L (cid:107) x − x (cid:48) (cid:107) which means that VC ( H ) ≤ Pack ( X , (cid:107)·(cid:107) , γ/L ) .
Alternatively, if H were invariant then we know from Proposition 3.1 that VC ( H ) ≤ Pack ( X π , (cid:107)·(cid:107) , γ/L )
≤ Pack ( X , (cid:107)·(cid:107) , γ/L ) , suggesting a distribution independent sample complexity improvement for invariant hypotheses.
C
C ONCENTRATION OF M EASURE
The following is an adaptation of a standard result.
Proposition C.3.
Let ( X , ρ ) be a metric space, Let Y =
B d ( r ) be the closed Euclidean ball of radius r in R d .
Let H be a class of functions h : X → Y .
Let (cid:96) ( y, y (cid:48) )
= (cid:107) y − y (cid:48)
(cid:107) 22 be the squared-error loss and let
S =
(( X 1 , Y 1 ) , . . .
, ( X n , Y n )) ∼ ( X, Y ) n be a training sequence drawn i.i.d.
for some random variables X, Y .
Then,
P
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
Proof.
The proof is based on Mohri et al.
(2018, Exercise 3.31).
For any probability measure µ on X × Y deﬁne
[Equation]
22 ] where ( A, B ) ∼ µ .
Then for any h, h (cid:48) ∈ H
[Equation]
Now let
[Equation]
By setting µ as the empirical measure on S and then as the distribution of ( X, Y ) , one ﬁnds that
[Equation]
Now let K be an κ -cover of H in (cid:107)·(cid:107) L ∞ .
Deﬁne the sets D ( k ) =
{ h ∈ H : (cid:107)
h − k (cid:107) L ∞ ≤ κ } .
Then
[Equation]
Set κ = α(cid:15) 8 rd for 0 < α < 1 .
Using the above, for any h ∈ D ( k ) we have L
[Equation]
h
[Equation]
, hence
P
[Equation]
sup
[Equation]
Then Hoeffding’s Inequality gives
[Equation]
where we used (cid:107) h
( X i ) − Y
i (cid:107) 22 ≤ 4 r 2 .