Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
1. Introduction
The ability to robustly categorize objects under conditions and transformations that preserve the object categories is essential to animal intelligence, and to pursuits of practical importance such as improving computer vision systems.
One way to achieve such robustness is equivariance.
When such transformations are restricted to be an algebraic group, the resulting equivariant representations have found significant success in machine learning starting with classical convolutional neural networks (CNNs) (Denker et al., 1989; LeCun et al., 1989) and recently being generalized by the influential work of Cohen and Welling (2016).
While it is clear that equivariance imposes a strong constraint on the geometry of representations, the implications of such constraints on model expressivity are not well understood.
In this work we take a step toward addressing this gap.
Our particular contributions are the following:
• We extend Cover’s function counting theorem (a measure of expressivity) to equivariant representations, finding that expressivity scales with the dimension of the subspace fixed by the group action.
• We demonstrate the applicability of our result to G -convolutional network layers – including pooling layers – through theory and verify through simulation.
This abstract is based on recently published work (Farrell et al., 2022).
2. Problem formulation
Suppose x abstractly represents an object and let r ( x ) ∈ R N be some feature map of x to an
N -dimensional space (such as an intermediate layer of a deep neural network).
We consider transformations of this object such that they form a group G in the algebraic sense of the word.
We denote the abstract transformation of x by element g ∈ G as g x .
Groups G may be represented by invertible matrices, which act on a vector space V (which themselves form the group GL ( V ) of invertible linear transformations on V ).
We are interested in feature maps r which satisfy the following group equivariance condition:
[Equation]
where π : G → GL ( R N ) is a linear representation of G which acts on feature map r ( x ).
Note that many representations of G are possible, including the trivial representation: π ( g )
=
I for all g .
We are interested in perceptual object manifolds generated by the actions of G .
Each of the P manifolds can be written as a set of points { π ( g ) r µ : g ∈ G } where µ ∈ [ P ] ≡ { 1 , 2 , . . .
, P } ; that is, these manifolds are orbits of the point r µ ≡ r ( x µ ) under the action of π .
We will refer to such manifolds as π -manifolds .
Each of these π -manifolds represents a single object under the transformation encoded by π ; hence, each of the points in a π -manifold is assigned the same class label.
To measure the expressivity of this representation, we consider a perceptron endowed with a set of linear readout weights w that attempts to determine the correct class of every point in every manifold.
The condition for realizing (i.e. linearly separating) the dichotomy { y µ } µ can be written as y µ w ⊤ π ( g ) r µ > 0 for all g ∈ G and µ ∈ [ P ], where y µ = +1 if the µ th manifold belongs to the first class and y µ = − 1 if the µ th manifold belongs to the second class.
The perceptron capacity is the fraction of dichotomies that can be linearly separated; that is, separated by a hyperplane.
The proofs for results in the main text are in Appendix A.2.
3. Separability of π -manifolds
We begin with a lemma which states that classifying the P π -manifolds can be reduced to the problem of classifying their P centroids.
Let π : G → GL( R N ) be an arbitrary linear representation of a compact group G .
1 We denote the average of π over G with respect to the Haar measure by ⟨ π ( g ) ⟩ g ∈ G ; for finite G this is simply 1 | G | (cid:80) g ∈ G π ( g ) where | G | is the order (i.e. number of elements) of G .
For ease of notation we will generally write ⟨ π ⟩ ≡ ⟨ π ( g ) ⟩ g ∈ G when the group G being averaged over is clear.
Lemma
A dataset { ( π ( g ) r µ , y µ ) } g ∈ G,µ
∈ [ P ] consisting of P π -manifolds with labels y µ is linearly separable if and only if the dataset { ( ⟨ π ⟩ r µ , y µ ) }
µ ∈ [ P ] consisting of the P centroids ⟨ π ⟩ r µ with the same labels is linearly separable.
3.1. Relationship with Cover’s Theorem
The fraction f of linearly separable dichotomies on a dataset of size P for datapoints in general position 2 in N dimensions was computed by Cover (1965) and takes the form f ( P, N )
= 2 1 − P (cid:80) N − 1 k =
(cid:0) P − 1 k (cid:1) where we take (cid:0) nm (cid:1) =
for m > n .
For fixed N , f ( P, N ) is a nonincreasing sigmoidal function of P with inflection point at P = 2 N .
We take this inflection point to be the capacity of the system, noting that as N → ∞ the sigmoid sharpens into a step function.
Theorem 2 Suppose the points ⟨ π ⟩ r µ for µ ∈ [ P ] lie in general position in the subspace
V
= range( ⟨ π ⟩ ) = {⟨ π ⟩ x : x ∈ V } .
Then V 0 is the fixed point subspace of π , and the fraction of linearly separable dichotomies on the P π -manifolds { π ( g ) r µ : g ∈ G } is f
( P, N 0 ) , where
N 0 = dim V 0 .
The fixed point subspace is the subspace W = { w ∈ V | gw = w , ∀ g ∈ G } .
The condition that the points ⟨ π ⟩ r µ be in general position essentially means that there is no prescribed special relationship between the r µ and between the r µ and ⟨ π ⟩ .
Taking the r µ to be drawn from a full-rank Gaussian distribution is sufficient to satisfy this condition.
3.2. The regular representation of Z m
First we illustrate the theory in the case of the cyclic group G =
Z m on m elements.
This group is isomorphic to the group of integers { 0 , 1 , ..., m − 1 } under addition modulo m , and this is the form of the group that we will consider.
An example of this group acting on an object is an image that is shifted pixel-wise to the left and right, with periodic boundaries.
Suppose π : Z m → GL( V ) is the representation of Z m consisting of the cyclic shift permutation matrices (this is called the regular representation of Z m ).
In this case
V = R m and π ( g ) is the matrix that cyclically shifts the entries of a length- m vector g places.
For instance, if m = 3 and v = (1 , 2 , 3) then π (2) v = (2 , 3 , 1).
The average of the regular representation matrix is ⟨ π ⟩ = 1 | G | 1 m 1 ⊤ m , indicating that ⟨ π ⟩ projects data along 1 m .
See Appendix A.3 for more illustrative examples of group representations and their capacities.
4. G-Equivariant Neural Networks
The proposed theory can shed light on the feature spaces induced by G -CNNs.
Consider a single convolutional layer feature map for a finite group G with the following activation:
[Equation]
for some nonlinear function ϕ .
For each filter i , and under certain choices of π , the feature map a i ( x ) ∈ R | G | exhibits the equivariance property a i ( g k x )
=
π ( g k ) a i ( x ).
We will let a ( x ) ∈ R | G | N denote a flattened vector for this feature map.
[Figure]
layer
[Figure]
In a traditional periodic convolutional layer applied to inputs of width W and length L the feature map of a single filter a i ( x ) ∈ R | G | is equivariant with the regular representation of the group G
=
Z W × Z L (the representation that cyclically shifts the entries of W × L matrices).
Here the order of the group is | G | =
W L .
Crucially, our theory shows that this representation contributes exactly one dimension to the fixed point subspace per filter (see
Appendix A.6.1).
Since the dimension of the entire collection of N feature maps a ( x ) is
N | G | for finite groups G , one might naively expect capacity to be P ∼ 2 N | G | for large N .
However, Theorem 2 shows that for G -invariant classification, only the fixed point subspace contribute to the classifier capacity.
Since the dimension of the fixed point subspace present in the representation is equal to the number of filters N , we have P ∼ 2 N (recall that
P = 2 N is the inflection point of f ( P, N )).
We show in Figure 1 that our prediction for f ( P, N ) matches that empirically measured by training logistic regression linear classifiers on the representation.
The convolutions in these networks are modified to have periodic boundary conditions while keeping the filters the same – see Appendix A.6.1 and Figure A.2 for more information and the result of using non-periodic convolutions, which impact the capacity but not the overall scaling with N 0 .
4.1. Pooling Operations
In CNNs, local pooling is typically applied to the feature maps which result from each convolution layer.
The impact of 2x2 maxpooling in CNNs is shown in Fig.
1, which reveals that maxpooling reduces capacity.
In Appendix A.4 we prove that maxpooling reduces capacity and bound this reduction.
We also prove that average pooling leaves the capacity unchanged.
4.2. Induced representations
Induced representations are a fundamental ingredient in the construction of general equivariant neural network architectures (Cohen et al., 2019).
Here we state our result, and relegate a formal definition of induced representations and the proof of the result to Appendix A.6.4.
Proposition 3 Let π be a representation of a finite group induced from representation ρ .
Then the fraction of separable dichotomies of π -manifolds is equal to that of the ρ -manifolds.
5. Discussion and Conclusion
Equivariance has emerged as a powerful framework to build and understand representations that reflect the structure of the world in useful ways.
In this work we take the natural step of quantifying the expressivity of these representations through the well-established formalism of perceptron capacity.
We find that the number of “degrees of freedom” available for solving the classification task is the dimension of the space that is fixed by the group action.
This has the immediate implication that capacity scales with the number of output channels in standard CNN layers, a fact we illustrate in simulations.
However, our results are also very general, extending to virtually any equivariant representation of practical interest – in particular, they are immediately applicable to GCNNs.
Appendix A. Appendix Appendix A.1 a glossary of definitions and notation. Appendix A.2 Proofs of Lemma 1 and Theorem 2. Appendix A.3 Additional example applications of Theorem 2. Appendix A.4 Proofs of local pooling results. Appendix A.5 a derivation of the irreps for the regular representation of the cyclic group Z m .
Appendix A.6 a description of the construction of the GCNNs used in the paper, the methods for empirically measuring the fraction of linearly separable dichotomies, a description of local pooling, and complete formal proofs of the fraction of linearly separable dichotomies for these network representations (including with local pooling).
This appendix also includes more description of the induced representation and a formal proof of the fraction of linearly separable dichotomies.
Appendix A.6.1 also contains an additional figure, Figure A.2.
A.1. Notation and Glossary
• x : an abstract notation for an input object • r ( x ): a feature map of the input to an N dimensional vector space.
• π : N dimensional linear representation of group G .
For each g ∈ G , π ( g ) ∈ GL ( R N ) is an N × N invertible real matrix.
• Equivariance property: r ( g x ) =
π ( g ) r
( x ) for all g ∈ G and all x . • Invariant measure: a measure µ : G → R + on G with µ ( gS ) = µ ( S ) = µ ( Sg ).
For finite groups, the uniform distribution.
For locally compact topological groups, the Haar measure.
• ⟨·⟩ g ∈ G : an average over the invariant measure of G • Irreducible representation (irrep): an irreducible representation ρ on vector space
V satisfies ρ ( g ) v ∈ V for all v ∈ V, g ∈ G .
• Character χ ( g ): the trace of the representation χ ( g ) =
Tr π ( g ).
• Fixed point subspace: the subspace V 0 for which π ( g ) v ∈ V 0 for all v ∈ V 0 .
• General position: a collection of P points in general position in an N dimensional vector space have the property that any subset of k ≤ N points are linearly independent.
These points are generic in the sense that they satisfy no more linear relationships than they must.
• Dichotomy: a particular binary labeling { y µ } of P points { x µ } .
• f ( P, N ): fraction of linearly separable dichotomies given by Cover’s function counting theorem (Cover, 1965).
• VC dimension: the largest possible integer P such that there exist P points where all possible dichotomies { y µ } can be realized by the model Abu-Mostafa et al.
• Capacity: the largest possible ratio α c = P/N where P points in general position can be linearly separated by a N dimensional perceptron with probability 1 in an asymptotic limit where P, N → ∞ with P/N = O N,P (1).
The classical result is α c = 2 (Gardner, 1987; Shcherbina and Tirozzi, 2003).
• P ( · ): a local pooling operation.
A.2. Proofs of Lemma 1 and Theorem
Lemma
A dataset { ( π ( g ) r µ , y µ ) } g ∈ G,µ
∈ [ P ] consisting of P π -manifolds with labels y µ is linearly separable if and only if the dataset { ( ⟨ π ⟩ r µ , y µ ) }
µ ∈ [ P ] consisting of the P centroids ⟨ π ⟩ r µ with the same labels is linearly separable.
Formally,
[Equation]
Proof The forward implication is obvious: if there exists a w which linearly separates the P manifolds according to an assignment of labels y µ , that same w must necessarily separate the centroids of these manifolds.
This can be seen by averaging each of the quantities y µ w ⊤ π ( g ) r µ over g ∈ G .
Since each of these quantities is positive, the average must be positive.
For the reverse implication, suppose y µ w ⊤ ⟨ π ⟩ r µ > 0, and define ˜ w = ⟨ π ⟩ ⊤ w .
We will show that ˜ w separates the P π -manfolds since
[Equation]
[Equation]
(Definition of ˜ w )
[Equation]
of
[Equation]
and linearity of π ( g )
[Equation]
of the Haar Measure µ ( Sg )
[Equation]
µ ( S ) for set S )
[Equation]
(Assumption that w separates centroids)
Thus, all that is required to show that ˜ w separates the π -orbits are basic properties of a group representation and invariance of the Haar measure to G -transformations.
Theorem
Suppose the points ⟨ π ⟩ r µ for µ ∈ [ P ] lie in general position in the subspace
V
= range( ⟨ π ⟩ ) = {⟨ π ⟩ x : x ∈ V } .
Then V 0 is the fixed point subspace of π , and the fraction of linearly separable dichotomies on the P π -manifolds { π ( g ) r µ : g ∈ G } is f
( P, N 0 ) , where
N 0 = dim V 0 .
Equivalently, N 0 is the number of trivial irreducible representations that appear in the decomposition of π into irreducible representations.
Proof By the theorem of complete reducibility (see Fulton and Harris (2004)), π admits a decomposition into a direct sum of irreducible representations ( irreps ) π ∼ = π k 1 ⊕ π k 2 ⊕ ...
⊕ π k M acting on vector space V =
V 1 ⊕ V 2 ⊕ ...
V M , where ∼ = denotes equality up to similarity transformation (see Serre (2014) for a definition of irreps).
The indices k j indicate the
[Figure]
type of irrep corresponding to invariant subspace V j .
The fixed point subspace V 0 is the direct sum of subspaces where trivial k j = 0 irreps act: V
= (cid:76) n : k n =0 V n .
By the Grand
Orthogonality Theorem of irreps (see Liboff (2004))
all non-trivial irreps average to zero ⟨ π k,ij ( g ) ⟩ g ∈ G ∝ δ k,
δ i,j .
Then, the matrix ⟨ π ⟩ simply projects the data to V 0 .
By Lemmathe fraction of separable dichotomies on the π -manifolds is the same as that of their centroids ⟨ π ⟩ r µ .
Since, by assumption, the P points ⟨ π ⟩ r µ lie in general position in V 0 , the fraction of separable dichotomies is f ( P, dim V 0 ) by Cover’s Theorem.
A.3. Example Applications of Equivariant capacity
The 2 × 2 discrete rotation matrices R
[Equation]
≡
[Equation]
where θ g = 2 πg/m and g ∈
Z m , are one possible representation of Z m ; in this case V = R 2 .
This representation is irreducible and nontrivial, which implies that the dimension of the fixed point subspace is(only the origin is mapped to itself by R for all g ).
Hence the fraction of linearly separable dichotomies of the π -manifolds by Theorem 2 is f ( P, 0).
This result can be intuitively seen by plotting the orbits, as in Figure A.1a for m = 4.
Here it is apparent that it is impossible to linearly separate two or more manifolds with different class labels, and that the nontrivial irrep R averages to the zero matrix.
The representation can be augmented by appending trivial irreps, defining π : G →
GL( R N ) by π ( g ) =
R ( θ g ) ⊕ I ≡ (cid:20) R ( θ g )
I (cid:21) where I is an ( N − 2) × ( N − 2)-dimensional identity matrix.
The number of trivial irreps is N − 2, so that the capacity is f ( P, N − 2).
This is illustrated in Figure A.1b for the case N = 3.
Here we can also see that the trivial irrep, which acts on the subspace span { (0 , 0 , 1) } , is the only irrep in the decomposition of π that does not average to zero.
This figure also makes intuitive the result of Lemma 1 that dichotomies are realizable on the π
-manifolds if and only if the dichotomies are realizable on the centroids of the manifolds.
Suppose π : Z m → GL( V ) is the representation of Z m consisting of the cyclic shift permutation matrices (this is called the regular representation of Z m ).
In this case
V = R m and π ( g ) is the matrix that cyclically shifts the entries of a length- m vector g places.
For instance, if m = 3 and v = (1 , 2 , 3) then π (2) v = (2 , 3 , 1).
In Appendix A.5 we derive the irreducible representations (irreps) of this representation, which consist of rotation matrices of different frequencies.
There is one copy of the trivial irrep π 0 ( g ) ≡ 1 corresponding with the fixed point subspace span { 1 m } where 1 m is the length- m all-ones vector.
Hence the fraction of separable dichotomies is f ( P, 1).
This is illustrated in Figure A.1c in the case where m = 3.
The average of the regular representation matrix is ⟨ π ⟩ = 1 | G | 1 m 1 ⊤ m , indicating that ⟨ π ⟩ projects data along 1 m .
A.3.1. Direct sums of Regular Representations
For our last example we define a representation using the isomorphism
Z m ∼ =
Z m 1 ⊕ Z mfor m =
m 1 m 2 and m 1 and m 2 coprime 3 .
Let π (1) :
Z m 1 → GL( R m 1 ) and π (2) :
Z m 2 →
GL( R m 2 ) be the cyclic shift representations (i.e. the regular representations) of Z m 1 and
Z m 2 , respectively.
Consider the representation π (1) ⊕ π (2) :
Z m → GL( R m 1 + m 2 ) defined by ( π (1) ⊕ π (2) )
( g ) ≡ π (1) ( g mod m 1 ) ⊕ π (2) ( g mod m 2 ), the block-diagonal matrix with π (1) ( g mod m 1 ) being the first and π (2) ( g mod m 2 ) the second block.
There are two copies of the trivial representation in the decomposition of π (1) ⊕ π (2) , corresponding to the one-dimensional subspaces span { ( 1 m 1 , 0 m 2 ) } and span { ( 0 m 1 , 1 m 2 ) } ,
where 0 m 1 is the length- k vector of all zeros.
Hence the fraction of separable dichotomies is f ( P, 2).
This reasoning extends simply to direct sums of arbitrary length ℓ , yielding a fraction of f ( P, ℓ ) .
These representations are used to build a novel G-convolutional layer architecture with higher capacity than standard CNN layers in Section A.6.3.
These representations are analogous to certain formulations of grid cell representations as found in entorhinal cortex of rats (Hafting et al., 2005), which have desirable qualities in comparison to place cell representations (Sreenivasan and Fiete, 2011).
Precisely, a collection { Z m k ×
Z m k } k of grid cell modules encodes a large 2-dimensional spatial domain
Z m × Z m where m = (cid:81)
k m k .
A.3.2. SO(3): A Non-Abelian Lie Group
The special orthogonal group SO(3) on 3 dimensions (rotation group), the 3 × 3 orthogonal matrices with determinant +1, can also be analyzed within our theory. G-convolutional
k after being divided by all divisors common among them.
Then Z ˜ m ∼ = ⊕ ℓk =1 Z ˜ m k and provided none of the ˜ m k are 1, one still gets a fraction of f ( P, ℓ ).
Place cells are analogous to standard convolutional layers. neural networks that are equivariant to SO(3) rotations have become of high interest in the physical sciences and computer vision where the objects of interest often respect these symmetries (Anderson et al., 2019;
Cohen et al., 2018; Esteves et al., 2018; Kondor et al., 2018)
The irreducible representations have the form B k m where B k m are (2 k m +1) ×
(2 k m +1) block matrices, known as Wigner D -matrices
(Wigner, 1931).
The trivial irreps correspond to the one-dimensional irreps with k = 0.
Thus, the SO(3)-invariant classification capacity merely counts the number of trivial irreps which have N 0 = (cid:80) m δ k m , 0 .
The capacity is again f ( P, N 0 ).
A.4. Pooling
In this section, we describe how our theory can be adapted for codes which contain such pooling operations.
We will first assume that π is an N -dimensional representation of G .
Let P ( r ) : R N → R N/k be a pooling operation which reduces the dimension of the feature map.
The condition that a given dichotomy { y µ } is linearly separable on a pooled code is
[Equation]
We will first analyze the capacity when P ( · ) is a linear function (average pooling) before studying the more general case of local non-linear pooling on one and two-dimensional signals.
A.4.1. Local Average Pooling
In the case of average pooling, the pooling function P ( · ) is a linear map, represented with matrix P which averages a collection of feature maps over local windows.
Using an argument similar to Lemma 1 (Lemma 4 and Theorem 6 below), we prove that the capacity of a standard CNN is not changed by local average pooling: for a network with N filters, local average pooling preserves the one trivial dimension for each of the N filters.
Consequently the fraction of separable dichotomies is f ( P, N ).
A.4.2. Local Nonlinear Pooling
Often, nonlinear pooling operations are applied to downsample feature maps.
For concreteness, we will focus on one-dimensional signals in this section and relegate the proofs for two-dimensional signals (images) to later in this section.
Let r ( x ) ∈ R N × D represent a feature map with N filters and length- D signals.
Consider a pooling operation P ( · ) which maps the D pixels in each feature map into new vectors of size D/k for some integer k .
Note that the pooled code is equivariant to the subgroup H =
Z D/k , in the sense that
P ( π ( h ) r ) =
ρ ( h ) P ( r ) for any h ∈ H .
The representation ρ is the regular representation of the subgroup H .
We thus decompose G into cosets of size D/k : g = jh , where j ∈ Z k and h ∈
Z D/k .
The condition that a vector w separates the dataset is
[Equation]
We see that there are effectively k points belonging to each of the P orbits in the pooled code.
Since P ( · ) is nonlinear, the averaging trick utilized in Lemma 1 is no longer available.
However, we can obtain a lower bound on the capacity, f ( P, ⌊ N/k ⌋ ), from a simple extension of Cover’s original proof technique as we show in Appendix A.6.2.
Alternatively, an upper bound f ≤ f ( P, N ) persists since a w which satisfies Equation 3 must separate ⟨ ρ ( h ) ⟩ h ∈ H P ( r µ ).
This upper bound is tight when all k points ⟨ ρ ( h ) ⟩ h ∈ H P ( π ( j ) r µ ) coincide for each µ , giving capacity f ( P, N ).
This is what occurs in the average pooling case where the upper bound f ≤ f ( P, N ) is tight.
Further, if we are only interested in the
H -invariant capacity problem, the fraction of separable dichotomies is f ( P, N ), since ρ is a regular representation of H .
First we prove an extension of Lemma 1 to equivariant linear maps.
This will be used to show that average pooling does not affect the capacity of the regular representation of
Z m .
Lemma 4 Let π be a representation of the group G and suppose the matrix P is equivariant with respect to the restriction of π to a subgroup H ⊆ G , so that for all h ∈ H P π ( h ) =
ρ ( h ) P for some representation ρ of H .
Let R denote a set of representatives of G/H .
Then we have the following equivalence.
[Equation]
Proof For the forward implication, we write the coset decomposition g
=
hg ′ of g and average over H to find
[Equation]
For the backward implication, suppose y µ w ⊤ P [Equation]
( h ) ⟩ h ∈ H π ( g ′ )
r µ > 0 for all representatives g ′ ∈ R , and define ˜ w = ⟨ ρ ( h ) ⟩ ⊤ h ∈ H w .
For any g ∈ G , take a coset decomposition g
=
hg ′ for h ∈ H and g ′ ∈ R .
We then have
[Equation]
The implication follows.
Lemma
For the regular representation of G = Z D , a local average pooling over windows of size k generates a matrix P which is equivariant with respect to the subgroup H =
Z D/k with the property that
[Equation]
where a > 0 is a positive constant.
Proof First, note that the new pooled code is the regular representation of H since shifts of size mk in the original feature map corresponds to shifts of length m in the pooled code.
Thus P is equivariant to H =
Z D/k .
Next we note the following two facts
[Equation]
a ′ and a ′′ are positive constants and 1 D and 1 D/k are the D and D/k dimensional vector of all ones, respectively.
Thus, we have that P ⟨ π ( h ) ⟩ h ∈ H = a P ⟨ π ( g ) ⟩ g ∈ G where a is a positive constant.
Theorem
The fraction of linearly separable dichotomies of a CNN pooling layer with
N filters after average pooling from feature maps with the size of the input image W × L to pooled feature maps of size W/k × L/k is f ( P, N ) , i.e. no capacity is lost due to local average pooling.
Proof The CNN layer before pooling is a regular representation π of the full group G =
Z W × Z L , applied to each of the M input channels via a direct a sum (cid:76) Mj =1 π .
The layer after pooling is a regular representation ρ of the subgroup H =
Z W/k × Z L/k , also applied to the output channels via a direct sum (cid:76) Nj =1 ρ .
Let R be a set of representatives of G/H .
Since P is equivariant to π and ρ over H we have by the previous two lemmas that
[Equation]
Thus the capacity is determined by the rank of ⟨ ρ ( h ) ⟩ h ∈ H , assuming the ⟨ ρ ( h ) ⟩ h ∈ H P r µ are in general position.
Since each ρ is a copy of the regular representation for H , the rank of ⟨ (cid:76) Nj =1 ρ ( h ) ⟩ h ∈ H is merely N .
Thus the fraction of linearly separable dichotomies is f ( P, N ), the same as the capacity before pooling.
Now we prove local pooling operations in a standard CNN preserve a regular representation of a subgroup of the cyclic group.
Lemma 7 Suppose P is a local pooling operation on two-dimensional signals (CNN feature maps), and that π is the regular representation of a group G =
Z W × Z L on code a ( x ) .
A pooled feature map r = P ( a ) which acts on k × k windows of a is a regular representation of the subgroup H =
Z W/k × Z L/k .
Proof Suppose an equivariant feature map a ( x ) ∈ R W × L × N has corresponding regular representation of the group G =
Z W × Z L for each of the N filters.
Consider any local pooling operation P ( · ) (such as average or maximum) which acts on k × k patches where k divides both W and L .
[Equation]
Note that for k > 1, r ( x ) is no longer equivariant to G since the representation does not satisfy the homomorphism property for shifts with length ℓ not divisible by k .
However, the new code is equivariant to a subgroup H =
Z W/k × Z L/k , namely vertical and horizontal shifts with length divisible by k .
Let π x nk,mk represent a vertical shift of the image x by nk pixels and horizontal shift by mk pixels.
Note that a ij,h ( π x nk,mk x ) =
a i + nk,j + mk ( x ) since a ( x ) is equivariant.
Then, the h -th pooled feature map transforms as r ij,h
[Equation]
x
[Equation]
,mk x [Equation] [Equation] is Equivariant to π
[Equation]
[Equation]
( x ) , (Definition of r )
We thus find that the code is a regular representation of the subgroup of the cyclic translations H
= { ( nk, mk ) } n ∈ [ H/k ] ,m ∈
[ W/k ] .
This new group G ′ has dimension | H | = 1 k 2 | G | .
A.5. Irreps for the cyclic group
Here we compute the irreducible representations (irreps) of representations π : Z m → GL( V ) of the cyclic group Z m over a real vector space V .
See Serre (2014) for a definition of irrep, and for a derivation of the irreps when V is a complex vector space.
The irreps can be used to determine the dimensionality of V 0 , as shown in the proof of Theorem 2 (Appendix A.2).
To find the irreps, one can use the form for the eigenvalues and eigenvectors for circulant matrices, since all the π ( g ) are circulant.
This results in the simultaneous diagonalization
[Equation]
where V is the real-valued version of the discrete Fourier transform matrix (the columns are proportional to cosines and sines of varying frequencies, along with a column proportional to 1 m ).
Note that the 2 x 2 rotation matrices R (2 πgk/m ) are irreps for k ̸ = m/ 2 and k ̸ = 0, since there is no one-dimensional subspace of R 2 that is invariant to R (2 πgk/m ) for all g .
The exception for k = m/ 2 if m is even, gives R (2 πgk/m ) =
( − 1)
g I , which corresponds to rotation of 180 degrees.
The subspace span { (1 , 0) } is invariant to this action, so that
R (2 πgk/m ) is not an irrep.
This representation can thus be reduced to an action on a onedimensional subspace, represented by ( − 1) g .
The case k = 0 gives the trivial representation.
A.6. G-equivariant convolutional layers A.6.1. Standard convolutional layers
A convolutional layer consists of a set of N k × k ′ filters F i that are convolved (technically cross-correlated) with a stack of M W × L input tensors.
Here M is the number of input channels and N the number of output channels.
The convolution runs each filter (i.e. takes the dot product at all possible positions) over each of the W × L input tensors, and the result is averaged across the M input channels to produce the output of one output channel.
In the positions where the filters approach the edges of the input tensor, different choices can be made about how to handle these edge conditions.
The standard choice is to pad the edges with some number of zeros depending on the desired shape of the output tensor and run the convolution out to the end of the padded image.
Another possible choice is to loop the edges of the input tensor together, so that the filter is applied to the other side of the input tensor as it runs off the edge.
This periodic boundary condition allows us to write the convolution formally in terms of group actions, and to apply our theory directly.
When convolutions are not periodic, the resulting capacity increases somewhat but still follows the P/N 0 scaling of the periodic convolutions (Figure A.2).
For the random convolutional layers of Figure 1a, the input tensors are size 10 ×and the number of input channels are M = 3, as for standard color images.
Each entry of these tensors is normally distributed with mean 0.
The filters are also of size 10 × 10 with periodic boundary conditions, and are initialized according to a normal Xavier distribution with parameters that are the default for Pytorch 1.9.
The convolution is implemented via the Pytorch 1.9 implementation of Conv2d with padding mode=“circular” and padding= in the case of periodic boundary conditions.
The bias term of the convolution is set to zero and the convolution is followed by a ReLU nonlinearity (blue line).
The resulting figures do not change appreciably for different choices of input tensor size, number of input channels, or size of filters (though note that the nonlinearity is essential for satisfying the general position condition of Theorem 2; otherwise, the capacity would be determined by the number of input channels rather than number of output channels).
The output of this convolution is then fed through a 2 × 2 max pooling layer (orange line in Figure 1a), provided by Pytorch 1.9’s MaxPool2d.
The pretrained VGG-11 layers used in Figure 1b and Figure A.2 are taken from liukuang (2017).
The first convolutional block (blue line) consists of 3 × 3 pretrained filters applied to CIFAR-10 image tensors randomly selected from the validation set and normalized in the same way they are normalized during training (see liukuang (2017) for details).
These images are of size 32 × 32 and have M = 3 input channels, followed by a batch normalization layer in evaluation mode (fixed parameters), and then followed by a ReLU nonlinearity.
The boundary conditions of these convolutions are set to be periodic in Figure 1b and nonperiodic with a zero padding sizes of 1 in Figure A.2, and the bias term is set to zero.
The batch normalization is an element-wise operation and so equivariant to the representations we consider – thus this operation is not expected and is not observed to affect the perceptron capacity.
This convolutional block is then followed by a 2 × 2 max pooling layer (orange line).
Finally, another convolutional block of 3 × 3 filters, batch normalization, and ReLU nonlinearity are applied (green line).
[Figure]
[Figure]
Details are exactly as in Figure 1b, but non-periodic convolutions with a zero padding of size
The fraction of linearly separable dichotomies is measured empirically by using the scikitlearn LogisticRegression implementation of logistic regression, with a tolerance value of tol=1e-18 and an inverse regularization value of C=1e8.
The maximum number of iterations is set to 500.
An intercept (i.e. bias) is not used for this fit.
To formally prove that the fraction of separable dichotomies is f ( P, N ) for standard periodic convolutional layers, first note that the convolution is equivariant with respect to cyclic permutation of the inputs and of the outputs.
The representation for cyclically permutating the output tensor can be written (cid:76) Nk
=1 π where π is the representation that cyclically permutes the entries of W × L matrices.
Since each copy of π contains one trivial irrep in its decomposition into a direct sum of irreps, the direct sum (cid:76) Nk
=1 π contains N trivial irreps in its decomposition.
The final step to use Theorem 2 is to argue that the centroids of the manifolds are in general position.
Since a nonlinearity (ReLU) is applied to the output of the convolution, and since there is no particular structure in the convolutional filters beyond possibly sparsity, we can generically expect this to be the case.
A.6.2. Lower Bound and Upper Bound on Capacity for Nonlinear Pooling
Theorem 8 Suppose a code which is equivariant to a finite group G is pooled to a new code which is equivariant to a finite subgroup H ⊆ G .
Suppose the number of trivial dimensions in the original G -equivariant code is N 0 .
Then the fraction of linearly separable dichotomies on the G -invariant problem for the pooled code is at least f ( P, ⌊ N 0 /k ⌋ ) where k = | G/H | .
Similarly the fraction is at most f ( P, N 0 ) .
Proof The pooled code, by assumption, has the property P ( π ( hg ′ ) r )
=
ρ ( h ) P ( π ( g ′ ) r ) for any h ∈ H and g ′ ∈ R , where R is a set of representatives of G/H .
The G -invariant separability condition amounts to the proposition
[Equation]
in other words, a solution on the right hand side affords a solution over all of the manifolds generated in the input space.
We see that, this requires considering if this particular dichotomy is linearly separble on the P k anchor points ⟨ ρ ( h ) ⟩ h ∈ H P ( π ( g ′ )
r µ ).
The simplest strategy to obtain an upper bound is to consider what happens when a single new manifold is added.
We see that when a single new base point r is added it corresponds to k new points in the orbit ⟨ ρ ⟩ P ( π ( g ′ ) r ) for all g ′ ∈ R .
Suppose that ⟨ ρ ( h ) ⟩ h ∈ H has rank N 0 .
Let
C ( P, N 0 ) represent the number of linearly separable dichotomies for P G -orbits in N 0 trivial dimensions.
Upon the addition of the k new points ( P → P + 1), we find that some of the pre-existing separable dichotomies give a new separable dichotomy.
This can be guaranteed to occur when a w separates the old dichotomy and has w ⊤ ⟨ ρ ⟩ · P ( π ( g ′ ) r ) =
for the new anchor point r (but this condition is not necessary for a new dichotomy to be separable).
This condition means that the original dichotomy is separable in the N 0 − k dimensional subspace { w :
[Equation]
By making infinitesimal adjustment to this w the correct label on this new orbit can be achieved without altering the labels on any other dichotomy.
Since this argument gives a sufficient but not necessary condition to generate a new dichotomy, we obtain the following inequality
[Equation]
Solving this recursion gives the capacity f Pooled ( P, N 0 ) ≥ f Cover ( P, ⌊ N 0 /k ⌋ ).
The greatest capacity occurs in the special case where ⟨ ρ ⟩ P ( π ( g ′ ) r ) = ⟨ ρ ⟩ P ( r ).
In this case, the usual counting theorem applies giving a fraction of separable dichotomies of f ( P, N 0 ).
This is achieved, for instance in average pooling as we showed in Theorem 6.
A.6.3. Direct sum equivariant convolutional layers
Here we describe how to build a convolutional layer architecture that is equivariant with respect to the regular representation in the input space and the direct sum representations introduced in Section A.3.1 in the output space.
For the following we assume that m 1 and m 2 are coprime, though see the footnote in Section A.3.1 for a discussion of how to loosen this requirement.
The input data is a W × L × M tensor where M is the number of input channels.
The first step is to simply take the output of a standard convolution (in our simulations we also apply a ReLU nonlinearity) applied to this input with periodic boundary conditions, resulting in a W × L ×
N tensor where N is the number of output channels.
The next step is to, for each of the output channels, take an average (or maximum) between entries spaced m 1 entries apart horizontally or vertically in the matrix, resulting in an m 1 × mmatrix.
In our simulations we took averages rather than maximums.
This is repeated for the other number m 2 , resulting in an m 2 × m 2 matrix.
This is repeated for every output channel, resulting in N matrices of size m 1 × m 1 and N matrices of size m 2 × m 2 .
Finally, the resulting matrices are flattened and appended into an ( m 21 + m 22 ) × N matrix, and the result is passed through a nonlinearity (ReLU).
As the input tensor is cyclically permuted according to a regular representation π of
Z m 1 m 2 , the output of this equivariant convolutional layer permutes according to the representation π (1) ⊕ π (2) where π (1) is the regular representation of Z m 1 and π (2) is the regular representation of Z m 2 .
The proof that the fraction of separable dichotomies is given by f ( P, 2 N ) follows the same proof as for the standard periodic convolutions in Appendix A.6.1.
Instead of a direct sum (cid:76) Nk
=1 π
we get a direct sum (cid:76)
Nk =
( π (1) ⊕ π (2) ).
Each of the π (1) ⊕ π (2) contain two trivial irreps in its decomposition, so that the final fraction is f ( P, 2 N ).
A.6.4. Induced representations
First we state the definition of an induced representation.
Let H be a subgroup of a finite group G and let ρ : H → GL( W ) be a representation of H .
Let V be the vector space of functions f : G → W such that f ( gh )
=
ρ ( h ) f
( g ) for all h ∈ H and g ∈ G .
We now define the induced representation π : G → GL( V ) to be the representation which satisfies ( π ( g ′ )
f )( g ) =
f
( gg ′ ).
For intuition, note that every element of g can be written g
= rh where r is a representative for a coset in G/H and h ∈ H .
This is because the cosets G/H partition G and the action of H stays within a coset; hence r selects out the coset, and h goes to the desired element of the coset: g = rh .
With this decomposition, the action of π is then π ( rh ) f
( g ) =
ρ ( h ) f
( gr ).
Hence the r component under π has the effect of permuting to the new coset that gr belongs in, and the h component under π then has the effect ρ ( h ) on the resulting vector f ( gr ) that we originally specified.
This is the most natural way to get a representation of G from a representation of H .
In the case of finite groups, one can think of the r component as permuting a set of isomorphic copies of V , each copy corresponding to a different coset.
To compute the capacity of induced representations, and so prove Proposition 3, we use
Frobenius reciprocity of characters.
Recall that the character θ of a representation π : G →
GL( V ) is the map θ : G → C induced by the trace: θ ( g ) =
Tr( π ( g )).
Now let θ be the character of ρ : H → GL( V ) and let θ G be the character of the induced representation.
Then ⟨ θ G ( g ) ⟩ g ∈ G = ⟨ θ ( h ) ⟩ h ∈ H by Frobenius reciprocity of characters (Mackey, 1970).
The average of the character is the number of trivial representations contained in the decomposition of the representation (see Serre (2014)).
Hence the capacity of the induced representation is equal to the capacity of ρ .
The existence of extensions beyond finite groups is not clear to the authors, but we welcome information if such exists.