C OHEN W ELLING BASES & SO (2) -E QUIVARIANT CLASSIFIERS USING T ENSOR NONLINEARITY . Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations.
We use the learned CW-basis to build a rotation equivariant classiﬁer to classify images.
The autoencoder and classiﬁer architectures use only tensor product nonlinearity.
The model proposed by
Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations.
We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously.
Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a).
Our classiﬁer has very good accuracy and we use fewer parameters.
Even when the sample complexity to learn a good CW-basis is low we learn classiﬁers which perform impressively.
We show that a coupled CW-bases in one scale can be deployed to classify images in a classiﬁer trained and tested on images in a different scale with only a marginal dip in performance.
I NTRODUCTION
A fundamental problem in vision is understanding how the human eye sees objects and images as being the same even when they undergo transformations.
To obtain such a behaviour in a machine learning set-up, a natural idea is to construct representations of the object which remain the same even when the object undergoes transformations.
This ’handcrafting’ of representations of the object invariant to transformations was the preferred method of the vision community for a long time.
Since the pathbreaking achievements of modern convolutional neural networks beginning with the seminal work of Krizhevsky et al.
the focus has changed - the interest is more now on developing algorithms that learn to construct these invariant representations from transformed examples.
Goodfellow et al.
(2009) were among the ﬁrst to address the question of invariance in deep networks.
Cohen & Welling (2014) were among the ﬁrst to use representation theory of compact Lie groups to give it a sound mathematical framework.
Developing on on ideas from earlier works of Rao & Ruderman (1999) and Sohl-Dickstein et al.
(2010) on the Lie group model, Cohen and
Welling build a model for the abelian group of transformations SO (2) , the group of rotations of the plane.
Using their model they learn a nice basis for the underlying vector space in which the images sit, which allows them to read off the types and multiplicities of the irreducible representations of
SO (2) which occur in images under rotations.
In this basis exposing irreducible representations 1 , the action of an element of SO (2) on an image is very easy to describe.
From projections of images onto this basis, 2 Cohen and Welling obtain features of images which they use for classiﬁcation.
A number of experts attribute the impressive performance of modern deep convolutional networks (CNN’s) to the fact that more complex and abstract features are learned as one moves down the network.
Nonlinearity seems to be essential to learn such features.
Deep networks achieve their nonlinearity by the use of activation functions such as RELU, Krizhevsky et al.
Convolution networks are by design invariant to translations.
Given the incredible success of these deep networks the quest is now to build networks that have invariance to a larger group of symmetries.
In Bruna & Mallat (2013) the authors show that the ﬁrst two layers of the scattering network yield a powerful representation of images which is invariant to geometric transformations.
Gens & Domingos (2014) proposed deep symmetry networks, symnets, that form feature maps over large symmetry groups.
They showed that Symnets over the afﬁne group have smaller sample complexity.
More recently, Group Convolutional Networks (GCNN’s) were introduced in Cohen & Welling (2016a) and Harmonic nets were introduced by Worrall et al.
Both
GCNN’s and Harmonic nets are designed to learn representations of images invariant to a larger set of symmetries than translations.
The resulting networks perform impressively - the error rate reported by GCNN’s on the MNIST-rot(MNIST Variations) data set is only 2.28%.
Harmonic nets are now the state of the art on MNIST-rot with a reported 1.62% error.
More recently Cohen & Welling (2016b) introduced Steerable CNN’s based on more sophisticated ideas from group representation theory.
This puts constraints on the network weights and architecture, and results in a reduction in the number of parameters to be learned.
Steerable CNN’s were shown to outperform ResNets, He et al.
(2016), and achieve state of the art results on CIFAR 10,100.
In both GCNN’s as well in in steerable CNN’s equivariance to the symmetries of a group G is achieved by a generalized form of convolution.
The standard convolution in CNN’s is replaced with the convolution of functions from G to C .
Kondor & Trivedi (2018b) proved the converse - they showed that any neural network which achieves equivariance with respect to the action of a group
G , necessarily implements such a generalized convolution of functions on G .
If one were to factor out depth, it appears that it is the nonlinearity of the activation functions used in deep networks that is responsible for their impressive performance.
But nonlinear functions like
RELU are mathematically hard to analyze.
It would be nice to design networks which construct features, just as deep networks seem to do in their various layers, with more mathematically amenable activation functions.
This is the ﬁrst motivation of this study.
Our other motivation comes from what is done in Pattern theory, Mumford (1997), where to learn images a stochastic model of the images to be learned is ﬁrst built.
To build a stochastic model of natural images it is important to view images under different scales simultaneously (Mumford (1997, Chapter 6)).
Scattering networks address this issue because they use a wavelet basis for images.
This leads us to the following motivating questions.
Can one use a mathematically amenable nonlinearity like tensor-product nonlinearity to design neural nets for image classiﬁcation, leveraging ideas from group representation theory?
How does one obtain complex abstract features from simple features using such a nonlinearity?
Does it help to view images under different scales simultaneously to build such neural nets?
In this paper we make a modest attempt to answer such questions.
We combine ideas from
Cohen & Welling (2014) and Kondor (2018a), and design neural networks leveraging notions of tensor products of group representations.
We decouple the classiﬁcation process in two steps - ﬁrst discover Cohen-Welling bases and then deploy them to train classiﬁers using ideas from Kondor.
P RELIMINARIES
We start with some deﬁnitions which set the stage for the work of Cohen & Welling (2014).
Let G be a group and let V be a ﬁnite-dimensional vector space over a ﬁeld - we will always assume that the underlying ﬁeld is the ﬁeld of complex numbers C , or the ﬁeld of real numbers R .
Let GL ( V ) denote the group of invertible linear transformations of V .
Deﬁnition
We say G acts on V if there is a group homomorphism ρ from G to GL ( V ) .
So g ∈ G acts on a vector v ∈ V by sending v to ρ ( g ) v .
If V has dimension d , then with respect to a basis of V , ρ ( g ) is given by a d × d matrix.
The action is then given by multiplying ρ ( g ) with v .
We denote this by g · v .
Deﬁnition
We say a subspace W of V is G -invariant under the action of G if for all g ∈ G and all w ∈ W , ρ ( g ) w ∈ W .
If V has no proper subspace which is G -invariant
we say V is irreducible.
The restriction of the action of G to a G -invariant subspace W is called a subrepresentation.
Suppose V has an invariant subspace W , and we can also ﬁnd a G -invariant subspace U such that
W ⊕ U = V .
Then one can choose a basis of W and of U giving us a basis of V .
In this basis each ρ ( g ) will be a block diagonal matrix consisting of two blocks of size dim ( W ) and dim ( U ) .
For certain groups (called reductive groups) there is always such a U , and the group of rotations of the plane, SO (2) , is an example of such a group.
For reductive groups we can continue this process till we have a decomposition of V into irreducibles, V = U 1 ⊕ U 2 ⊕ . . .
⊕ U k .
Deﬁnition
If V and W are representations of a group G , a G -morphism (aka a G -equivariant map) from V to W is a linear map φ from V to W such that φ ( g · v ) =
g · φ ( v ) .
If V and W are vector spaces over complex numbers Schur’s lemma, Serre (1977), places restrictions on the dimension of the space of G -morphisms.
First, if V and W are irreducible representations of
G , Schur’s lemma states the space of G -morphisms between them is either one dimensional (i.e every
G -morphism between them is given by a scalar matrix) or zero dimensional (i.e the zero morphism is the only G -morphism between them).
This allows us to group irreducible representations into types, with two of them being of the same type when there is a nonzero G -morphism between them.
Given a decomposition of a vector space with a G -action into a direct sum of irreducibles as above, we collect all irreducibles of the same type together and write V = m 1 S t 1 ⊕ m 2 S t 2 · · · ⊕ c k S tk .
Here tk is an indexing set for types, S ti is an irreducible G -representation of type ti , and S ti and
S tj are of different types when i 6 =
j .
We then say S ti occurs in V with multiplicity m i in the decomposition of V .
Now assuming that V = ⊕ i
= k i =
1 m i S ti and W =
⊕ i
= k
i =1 n i S ti , it follows that the dimension of the space of G -morphisms between V and W is
P i m i n i .
Cohen & Welling (2014) considered the problem of supervised learning of images under the group
SO (2) .
Viewing an N × N image as an N 2 dimensional vector, an element of SO (2) acts on an N 2 dimensional vector by taking the vector representing the image to the vector representing the rotated image.
SO (2) is an abelian group (rotating an image by θ 1 and then by θ 2 or rotating it ﬁrst by θ
and then θ 1 yields the same image).
Hence, over complex numbers, the irreducible representations of SO (2) are one-dimensional (see, Kanatani (1990)).
When the underlying space is a real vector space (as was considered in Cohen & Welling (2014)), the types are parametrized by nonnegative integers n (over complex numbers the irreducible representations are parametrized by all integers).
The irreducible representation corresponding to type n = 0 , S 0 is one-dimensional and is an invariant under SO (2) .
For n > 0 , the irreducible representation S n is two dimensional (over complex numbers this splits into two one-dimensional representations parametrized by ± n ).
It follows, say, from Kanatani (1990, 2.3.7, 2.4.7) that there is a change of basis of the underlying vector space V with respect to which the action of the rotation group is given by a block diagonal matrix D with blocks of size 1 and 2 .
It is easy to describe these blocks.
Blocks of size 1 correspond to invariants, of type n = 0 , and have entry 1 .
The number of such blocks is called the multiplicity of the space of invariants in V .
The block corresponding to nonzero n is the elementary rotation matrix (cid:20) cos
( nθ )
−
sin( nθ
) sin( nθ ) cos( nθ ) (cid:21) .
It follows that the number of such blocks is the multiplicity of irreducible type n in V .
So there exists an N × N orthogonal matrix W and a diagonal matrix D , as above, with the property that when an image x is rotated by θ the transformed image is given by WDW t x .
In
Cohen & Welling (2014), the authors address the question of determining the matrix W and the multiplicity with which a given n appears.
The input is collection of pairs ( x, y ) , of an image x
and it’s transformation y when it is rotated by an unknown angle θ .
The authors ﬁnd W using expectation-maximization over an interesting prior.
The W their algorithm outputs is almost always a small subspace of the image space, spanning only some of the irreducible subspaces (similar to an
SVD procedure which just outputs only the top few singular vectors).
So strictly speaking we should be calling this a CW-subbasis since it may not span the image space, but we ignore this technicality.
An image x is then projected onto the W constructed and the vector of projected values, W T x , were used by Cohen and Welling as elementary features of the image for classiﬁcation.
Recently Kondor (2018a) proposed a neural network for learning the behaviour and properties of complex many-body physical systems.
The neurons in this system operate entirely in the Fourier space.
The neurons compose activations following Schur’s lemma, thereby ensuring that the activations are covariant to rotations.
We pose our motivating questions more precisely using the language of Cohen-Welling.
Can we obtain a CW-basis of images and their rotations using tensor product nonlinearity?
How can elementary features obtained using a CW-basis be combined to get more complex abstract features for building good classiﬁers?
Can one discover CW-bases of images in two scales in tandem, with one inﬂuencing the discovery of the other?
Do such coupled bases perform better in classiﬁcation?
Can one use these coupled CW-bases interchangeably?
Our contributions We give an afﬁrmative answer to all the questions.
We give several architectures to construct such bases.
We use Kondor’s framework to construct a classiﬁer which takes elementary features and combines them using simple ideas from the representation theory of SO (2) .
We give a simple coupled autoencoder architecture which answers question 3 above - we use tensor product nonlinearity crucially.
While such bases are not better at classiﬁcation we show that they can be interchangeably used with a marginal drop in performance.
2. N OTIONS FROM REPRESENTATION THEORY
To describe our set up and explain our experiments we will need a few more notions from representation theory.
Throughout we will assume that the underlying group G is a reductive group, in fact the reader may think of G as SO (2) .
If V is a representation of a group G then the dual vector space V ∗ acquires a natural G action given by ( g · f )( v ) =
f ( g − 1 v ) , for f ∈ V ∗ , v ∈ V .
If V, U are representations of a reductive group then V ⊗ U is a representation of the product group G × G .
Furthermore, since G is a subgroup of G × G under the diagonal embedding ( g mapping to ( g, g ) ), V ⊗ U is a representation of G .
And by what we described above V ⊗ U splits into a direct sum of irreducible representations of G identiﬁed by their types and multiplicities.
It also follows that the tensor algebra, T ( V ) =
⊕
i V ⊗ i of V , is an (inﬁnite dimensional, algebraic) representation of G .
Remark 4 When G is SO (2) and 0
= i <
j then S i ⊗ S j splits into a direct sum S i + j ⊕ S j − i .
When i = [-
-]0 , S i ⊗ S j is isomorphic to S j , (Kanatani (1990)).
To make precise the question about coupled bases and features from a coupled bases we need one more deﬁnition.
Deﬁnition
We say G -representations V and W are coupled if W is the image of a G -morphism of a ﬁnite dimensional subrepresentation of the tensor algebra of V , and V is the image of a Gmorphism of a ﬁnite dimensional subrepresentation of the tensor algebra of W .
The above deﬁnition is equivalent to requiring that W can be realized as a subrepresentation (or a quotient) of the tensor algebra of V and V can realized as a subrepresentation (or a quotient) of the tensor algebra of W .
Deﬁnition 6 Let V be the vector space of 14 × 14 images and let W be the vector space of 28 ×images.
Assume the rotation group SO (2) acts on both.
We say a Cohen-Welling basis X of V is coupled to a Cohen-Welling basis Y of W if the vector space dual of the subspace spanned by X (with its SO (2) -action) is coupled to the vector space dual of the subspace spanned by Y .
Recall that by projecting an image onto a CW-basis we get elementary features of the image which are used for classiﬁcation.
These elementary features are elements of the vector space dual of the space spanned by the CW-basis, hence the need for the word dual in the above deﬁnition.
Motivated by the above deﬁnition we say features obtained from coupled CW-bases are coupled features .
The deﬁnition above is motivated by our question of whether one can build CW-bases in two different scales with one inﬂuencing the discovery of the other and vice-versa.
Our deﬁnition suggests , but use the rotation group SO (3) - for completeness we have included their results in
Table 1.
that in order to generate coupled CW-bases we will need to view the images at different scales simultaneously, and use tensor product nonlinearity to generate them, thereby forcing the inﬂuence we are looking for. {+Remark 
+}
In the appendix we describe our problem and solution in a language that is more familiar to the ML community - as that of seeking ﬁlters in the Fourier space of images.
Remark 8 One last remark is in order before we describe our setup and experiments.
We are working over the ﬁeld of real numbers, so Schur’s lemma does not apply in our situation.
Nevertheless when we look for SO (2) -morphisms between two irreducible vector spaces with an SO (2) -action we will assume that the morphisms are of the kind Schur’s lemma dictates - either we have no morphism or the morphism is given by a scalar matrix rId , r a real number and Id being the identity matrix.
E XPERIMENTS AND RESULTS
We use the notation W ℓ for a CW-basis of images of size ℓ × ℓ .
So W ℓ is a ℓ 2 × d matrix for some d (usually smaller than l 2 ).
Then, rotation of an ℓ × ℓ image x by an angle θ is the ℓ 2 -size vector W ℓ D ℓ ( θ ) W Tℓ x , where D ℓ is the block diagonal matrix described in section 2, of size d × d .
If a column of W ℓ represents a basis vector which is invariant, the corresponding block is 1 , and if a pair of columns in W ℓ is a basis for an irreducible representation of type n , the block in D ℓ for this pair is the elementary rotation matrix described in section 2.
If there is no confusion we ignore the subscripts.
We give three neural net architectures to discover CW-bases of images.
3. L EARNING CW- BASES
An autoencoder(AE) architecture.
The autoencoder(AE) architecture is given as Alogrithm 1.
An equivalent block diagram is given in the Appendix (see ﬁgure 6). Details of computing Z i below are given in Appendix, Section E. Algorithm
Input: X i ∈ R 784 , i ∈ { 1 , 2 , . . . }
Hyper-parameters : a j , b j , β
Let d a = a 0 +
P j =1 2 × a j ; d b = b 0 +
P j =1 2 × b j
Output: W 28 ∈ R 784 × d a
Output: φ ∈ R d b × d a // Equivariant linear maps (Block diagonal
Output: ψ ∈ R d a × ( d b + d b 2 ) // matrices) as dictated by Remarkfor i = 1 , 2 , . . .
do θ i ∈
U [0 , 360) //
Choosen uniformly at random
Y i =
Rotate ( X i , θ i ) //
Rotate is the standard image rotation
ˆ Y i
=
W T
Y
i
Z i =
ψ ( φ ( ˆ Y i ) ⊗ φ ( ˆ Y i )
⊕ φ ( ˆ Y i ))
ˆ X i =
W 28 D ( − θ i )
Z i //
Unrotate and reconstruct the image end
L =
P i |
X i − [Equation] X i |// Loss function
W 28 , φ, ψ = min W 28 ,φ,ψ ( L + β | W T 28 W 28 − Id d a × d a | 1 ) return W 28 , φ, ψ 2.
Coupled Autoencoder(CAE) architecture .
In this setup, Figure 1, we learn W 14 and W 28 in tandem.
We feed both, an image X and a scaled down version of the image x to the network.
The network on top takes x and produces ˆ X a 28x28 image.
The bottom network takes X and produces ˆ x a 14x14 image.
The two networks are connected and we use the same W 28 , and W 14 in the top and bottom layer.
Hyperparameters are multiplicities of SO (2) -irreducible representations in W 28 and W 14 .
We learn the W 14 , W 28 and two SO (2) -equivariant maps φ and ψ .
The bottom network takes X , rotates it by θ and projects the resulting Y on the current
W 28 , to get ˆ Y = W T 28 Y .
Applying φ we get φ ( ˆ Y ) .
We unrotate φ ( ˆ Y ) using D 14 ( − θ ) to get z .
We generate a 14x14 image using the current W 14 to get ˆ x = W 14 z .
The top network takes x and rotates it by the same θ .
This is projected onto the current W 14 to get ˆ y .
We apply ψ to ˆ y ⊕ (ˆ y ⊗ ˆ y ) and unrotate the resulting vector using D 28 ( − θ ) to get Z .
We use the current Wto get ˆ X
=
W
Z .
We minimize the reconstruction error | X − ˆ X | 2 + | x − ˆ x | 2 .
[Figure]
Bootstrapping W 14 to W 28 , W 28 to W 14 .
The appendix describes an architecture in which we bootstrap, starting with a W 14 learned in experiment 1 or 2 and learn a W 28 .
[Figure]
Figure 2 shows a 2 layer classiﬁcation network for 28 × 28 images - this can be extended to any number of layers, and other image sizes.
This is a logical extension of the auto encoder network.
It is a CNN in Fourier domain using tensor product nonlinearity as in Thomas et al.
Assume the dimension of W 28 is d W .
We project the image x onto W 28 to get W T 28 x .
Let L 1 be a vector space with an SO (2) -action - hyperparameters are multiplicities l 1 j of irreducibles SO (2) -representations
S j in L 1 .
We learn an SO (2) -equivariant map φ 1 from the image space of W T 28 to L 1 .
The SO (2) - decomposition of l 1 = φ 1 ( W T 28 x ) ⊕ ( φ 1 ( W T 28 x ) ⊗ φ 1 ( W T 28 x )) is computed.
This is the input to the next layer.
There we choose a vector space L 2 with a SO (2) -action which splits into irreducibles
S j with multiplicities l 2 j (hyperparameters).
We learn a second SO (2) -equivariant map φ 2 from
L 1 ⊕ ( L 1 ⊗ L 1 ) to L 2 and compute φ 2 ( l 1 ) .
We compute the tensor product of φ 2 ( l 1 ) with itself and produce l 2 =
φ 2 ( l 1 ) ⊕ ( φ 2 ( l 1 ) ⊗ φ 2 ( l 1 )) .
This is the input to the next layer.
So we learn as many
SO (2) -equivariant maps as there are layers.
In the ﬁnal layer, l f is projected onto the invariant space (the multiplicity of S 0 in L f ⊕ ( L f ⊗ L f ) and fed to a soft max classiﬁer.
3. R ESULTS 1. Construction of CW-basis
We construct CW-basis W 14 for 14 × 14 images and CW-basis W 28 for 28 × 28 in the AE and CAE architectures.
We evaluate these bases in terms of image reconstruction error (MSE).
We also used rotation reconstruction error by comparing with scikits-image
[Figure]
rotation.
The data set for which these errors are reported is MNIST-rot.
We choose [10 , 5 , 5 , 5 , 5 , 4 , 4 , 4 , 4 , 4 , 3 , 3 , 3 , 3 , 3 , 2 , 2 , 2 , 2 , 2 , 1 , 1 , 1 , 1 , 1] as our multiplicities for the a i ’s (hyperparameters) and [8 , 4 , 4 , 4 , 4 , 3 , 3 , 3 , 3 , 3 , 2 , 2 , 2 , 2 , 2 , 1 , 1 , 1 , 1 , 1] for the b i ’s (hyperparameters).
Figure 3 and Figure 4 shows graphs of the error functions for W 28 constructed in
AE, CAE architectures as a function of the number of samples given to learn the W 28 .
Even when the number of samples is as small as 50, we discover CW-bases which are good at rotation and reconstruction in both architectures. In Figure 7, in the Appendix we visualize a few basis elements learned.
Results on Classiﬁcation, MNIST In Figure 5 we plot the accuracy of these various W 28 obtained above when deployed for classiﬁcation.
We plot the accuracy of the classiﬁer as a function of the number of samples used to construct the CW-bases.
When the number of samples is as low as 50 a CAE- W 28 performs better than an AE- W 28 .
Beyond 100 samples the difference is insigniﬁcant.
Plots using a bootstrapping CW-basis are similar to that of the AE and omitted.
In Table 1 we give the mean accuracy(and stdev) of our classiﬁer when trained and tested on combinations using MNIST(NR) and MNIST-rot(R).
The R/NR column for example denotes the accuracy when trained on MNIST-rot and tested on MNIST (LeCun et al.).
For comparison we use Planar, Spherical CNN accuracies given in Cohen et al.
We also compare with FFS2CNN, (see Kondor et al.
(2018c)).
The fourth row shows the accuracy of our classiﬁer that used a CAE- W 28 .
This was obtained using CAE architecture which was given 12000 training samples of MNIST-rot to discover the
W 28 .
Our classiﬁer performs better in all the scenarios.
Coupling interchangeability To test how coupled the CAE- W 28 and CAE- W 14 are, we did the following experiment - the classiﬁer trained in row 4 with CAE- W 28 was presented with down sized 14x14 images for classiﬁcation.
No additional training was done.
Instead we use the top half of the coupling network from Figure 1 - given a test image y we compute
ˆ y = W T
y using the CAE- W 14 .
Then ψ ((ˆ y ⊗ ˆ y ) ⊕ ˆ y ) is fed to the trained classiﬁer of row 4.
These results are reported in row 5 as [28/14 Tensor].
For comparison we took the 14x14 images and scaled them up to 28x28 and then fed them to the trained classiﬁer of row 4.
These results are reported in Row 6 as [28/14 Scale].
Using CAE- Wis better, indicating that a coupled bases retains scale information well.
We repeated the same experiment when the coupled network is given only 500 samples to learn the CAE- W 28 and CAE- W 14 .
As row 8 shows, performance is still very good.
[Table]
[Table] Results on Classiﬁcation, Fashion-MNIST . In Table 2 we report mean classiﬁcation accuracy(and stdev) on the Fashion-MNIST data set(Xiao et al.
We rotate each data point around the origin by an angle chosen uniformly between 0 and 2 π to create F-MNIST-rot. The CW basis was learned in the AE architecture with Fashion-MNIST as input.
The classiﬁer is a four layer network (96K parameters). We compared with a Depth 5 CNN (102K parameters). [Table]
[Table]
I MPLEMENTATION
Our autoencoders and classiﬁers were implemented as neural nets in Python (TensorFlow). We used the Adam optimizer.
C ONCLUSION AND FUTURE WORK
We answer in the afﬁrmative all our motivating questions - we design architectures which learn
CW-bases and classify using only tensor product nonlinearity.
We obtain some state of the art results.
It appears that coupling bases do retain scale information and can be interchangeably used for classiﬁcation albeit with a small drop in performance.
There is no appreciable advantage in using coupled bases for classiﬁcation excepting when the bases are learned using small samples of inputs.
Tensor product nonlinearity allows us to decouple the classiﬁcation problem in two steps.
One advantage of decoupling the process is that the basis can be obtained ofﬂine.
Our ideas can be extended to other groups.
We have begun experiments with the action of the S n on images.
We are testing our architectures on CIFAR-10.
One issue we run into when using architectures with a lot of depth, is that accuracy increases and then drops.
A similar issue is resolved in RES-nets
He et al.
by computing residual functions.
In ongoing work we proceed similarly since the residual function is SO (2) -equivariant.
Robustness of classiﬁers to noise is an active area of research, (see Goodfellow et al.
We believe our classiﬁers built from bases learnt in a CAE architecture should be robust to noise - our belief comes from the fact that coupling bases can be used interchangeably.
We plan to investigate this.
A CKNOWLEDGEMENTS
The authors would like to thank Infosys foundation for a generous grant.
Appendices A C ONVOLUTION IN THE F OURIER DOMAIN
A CNN is equivariant to translations - i.e, whether we translate the image and convolve it with a ﬁlter or we convolve the image ﬁrst and then translate it we get the same result. Thus the ﬁlters are naturally equivariant to the group of translations. We would like obtain such an equivariance with respect to the group of rotations.
To do so it is easier to work in the Fourier space of the group
SO (2) .
However a vector space basis in the Fourier space is not readily available - (this is what we call CW basis - Cohen and Welling compute this in their paper).
Now every CW basis vector comes indexed by a non-negative integer (Paragraph 5 on Page 3).
There could be multiple basis vectors indexed with the same integer which is called the multiplicity (Paragraph 6 of page 3).
So in the Fourier space an image is a linear combination of CW basis vectors with coefﬁcients (which of course depend upon the image).
Lets call them CW coefﬁcients.
Denoting the CW basis by W , ˆ x = W T x are the CW-coefﬁcents of x .
We ﬁrst compute the CW basis given a reasonable number of samples of images x and their rotations.
The CW basis does not necessarily span the entire Fourier space, we need to ﬁnd enough basis vectors which give a good approximation (Paragraph 7, page 3).
Convolution of an image with a SO (2) -equivariant ﬁlter means that whether we rotate an image and then convolve, or ﬁrst convolve and then rotate we should end up transforming the image in the same way.
This translates to the following in the Fourier domain - taking linear combinations of CW coefﬁcients of CW-basis vectors of the same type m, thereby getting an equivariant linear map.
The entries of this linear map L is what we seek to ﬁnd.
So if we have a space V with a CW basis having m i basis vectors indexed by integer i and another space W with n i basis vectors indexed by integers i , the search space for SO (2) -equivariant ﬁlters in the Fourier domain has dimension
P i m i n i , corresponding to block diagonal matrices, the i -th block being of size n i m i (see paragraph 4 on page 3).
The natural nonlinearity in the Fourier space is multiplication of CW coefﬁcients.
When we multiply the coefﬁcient of a basis vector of type m and the coefﬁcient of a basis vector of type n, we get two coefﬁcients, for basis vectors of type m+n and m-n (content of Remark 4).
These are quadratic functions, of the starting CW coefﬁcients - since this nonlinearity is obtained by taking tensor products of irreducible representations, we call it tensor product nonlinearity.
Our learning and classiﬁcation takes place in the Fourier world.
B P ROOFS
Here we prove that the bases obtained in the coupling architecture satisfy the deﬁnition of coupling.
The proof is straightforward assuming that we have zero loss.
Let W 28 and W 14 be the ﬁnal CW-bases learned.
Following the bottom half of the network describing the coupling architecture, we see that under the morphism φ , the image space of W T 28 surjects onto the image space of W T 14 .
Surjection follows since the dotted arrow going from W 14 on the top to W 14 below shows that we can start with any linear combination of the basis elements of Wat the top and obtain it as the image of φ applied to an appropriate linear combination of basis elements of W 28 .
So the image space of W T 14 is isomorphic to a quotient of the image space of W T 28 .
On the other hand following the top half of the network it is clear that ψ maps the image space of
W T 14 ⊕ ( W T 14 ⊗ W T 14 ) surjectively onto the image space of W T 28 .
Surjectivity follows by reasoning as above.
Hence the bases obtained in the coupling architecture satisfy deﬁnition 6.
C L EARNING CW- BASIS
AutoEncoder(AE) architecture We describe this using ﬁgure Figure 6 (this architecture works for 14x14 images, 56x56 images).
The hyperparameters are multiplicities with which SO (2) irreducible representations occur in the images viewed as vectors in a 784 dimensional vector space with an action of SO (2) .
We choose nonzero multiplicities a j for representations S j with 0 ≤ j ≤ 24 giving us a subspace of the image space of dimension d W = a {+
+} +
P j 2 ∗ a j .
We denote this subspace by W 28 and learn a CW-basis also denoted W 28 , a 784 × d W matrix.
We initialize W 28 randomly.
We rotate a 28 x 28 image x by a random angle θ , to get y and project y onto W 28 to get ˆ y = W T 28 y .
We learn an SO (2) -equivariant map φ from the image space of W T 28 to a vector space L with an SO (2) -action, with multiplicities (hyperparameters) b j .
Consider φ (ˆ y ) ∈ L .
Now L ⊗ L gets a natural SO (2) action.
Recall that the tensor product of an SO (2) irreducible representation of type m and an SO (2) -irreducible representation of type type n yields irreducible representations of types m + n and m − n , (w.l.o.g m > n ).
We use this to compute the decomposition of φ (ˆ y ) ⊗ φ (ˆ y ) .
The last set of parameters to be learned are an SO (2) - equivariant map ψ from L ⊕ ( L ⊗ L ) to the image space of W T 28 .
Let z
=
ψ ( φ (ˆ y ) ⊕ ( φ (ˆ y ) ⊗ φ (ˆ y ))) .
We unrotate z using D ( − θ ) .
Then we reconstruct the image ˆ x = W 28 D ( − θ )
z We expect this to be close to x
so we minimize the reconstruction error | ˆ x − x | 2 with a regularizer to ensure that W T 28 W 28 is identity.
[Figure]
⊕ φ (ˆ y )) D ( − θ ) z ReconstructImage 28x28Image
ˆ x φ (ˆ y )
W W 2. Bootstrapping architecture In this experiment we start with a known W 14 obtained using experiment 1 for 14x14 images, to learn a W 28 .
The setup to go from W 14 to W 28 is exactly the part of the network in Figure 1 going from X to ˆ X along the top.
The hyperparameters are the multiplicities of SO (2) -irreducible representations in W 28 .
We learn the SO (2) -equivariant ψ and W 28 by minimizing | X − ˆ X | 2 . Similarly starting with a known W 28 we can learn W 14 by following the path in Figure 1 from x to ˆ x along the bottom. D V ISUALIZATION OF CW B ASIS Figure 7 shows a few basis elements learned using AE architucture on MNIST-rot.
E E XAMPLE
Let us see an example of how Z i is calculated in Algorithm 1.
Let d a and d b be as given in the
Algorithm.
φ ∈ R d b × d a is a block diagonal matrix with blocks of sizes b 0 × a 0 , 2 b 1 × 2 a 1 , 2 b 2 × 2 a 2 , . . .
The ﬁrst block has ( b 0 ∗ a 0 ) entries ,the second block has ( b 1 ∗ a 1 ) entries with the following structure.
  α α . . .
α a 1  α α. . .
α a 1 1 α α . . . α a 1  α α. . .
α a 1 2 .
α 1 b α 2 b . . . α a 1 b  α 1 b α 2 b. . .
α a 1 b 
The multiplicities in ˆ Y are a = [2 , 1 , 2] and that of φ ( ˆ Y ) are b = [0 , 1 , 1] .
Hence d a = 8 , d b =and φ with arbitrary chosen values is given below. [Equation]
 0 0 1 0 0 0 00 0 0 1 0 0 00 0 0 0 2 0 30 0 0 0 0 2 0 
Boxes indicate the block for type 1 and type 2.
Let φ ( ˆ Y ) =
( p, q, r, s ) T
When we tensor ( p, q ) T of type n with ( r, s ) T of type m (m > n), we get ( pr − qs, ps + qr ) T of type (m+n) and ( pr + qs, ps − qr ) of type (m-n) (see Remark 4).
In this example, when we tensor φ ( ˆ Y ) with itself, we get [Equation]

[Equation]

[Equation] [Equation] [Equation]     with the following types : multiplicities - type 0 : 4, type 1 : 2, type 2 : 1, type 3 : 2, type 4 : 1.
During implementation we avoid duplicates and zeros and we use the following:
[Equation]

Also, φ ( ˆ Y ) ⊗ φ ( ˆ Y ) ⊕ φ ( ˆ Y ) after grouping the types, is given below:
[Equation]

The linear map ψ ∈ R d c × d a is similar to φ .
Only the input and output multiplicities are different.
In the example we have c = [2 , 2 , 2 , 1 , 1] and a = [2 , 1 , 2] .
A particular ψ with some ﬁxed values is given below, ψ =     1 2 0 0 0 0 0  0 0 03 4 0 0 0 0 0  0 0 00 0 5 0 6 0 0  0 0 00 0 0 5 0 6 0  0 0 00 0 0 0 0 0 7  0 0 00 0 0 0 0 0 0  0 0 00 0 0 0 0 0 9 0 0 0 00 0 0 0 0 0 0 10 0 0 0   
And Z i = ψ ( φ ( ˆ Y ) ⊗ φ ( ˆ Y )
⊕ φ ( ˆ Y ))