C OHEN W ELLING BASES & SO (2) -E QUIVARIANT CLASSIFIERS USING T ENSOR NONLINEARITY . Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations.
We use the learned CW-basis to build a rotation equivariant classiﬁer to classify images.
The autoencoder and classiﬁer architectures use only tensor product nonlinearity.
The model proposed by
Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations.
We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously.
Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a).
Our classiﬁer has very good accuracy and we use fewer parameters.
Even when the sample complexity to learn a good CW-basis is low we learn classiﬁers which perform impressively.
We show that a coupled CW-bases in one scale can be deployed to classify images in a classiﬁer trained and tested on images in a different scale with only a marginal dip in performance.
I NTRODUCTION
A fundamental problem in vision is understanding how the human eye sees objects and images as being the same even when they undergo transformations.
To obtain such a behaviour in a machine learning set-up, a natural idea is to construct representations of the object which remain the same even when the object undergoes transformations.
This ’handcrafting’ of representations of the object invariant to transformations was the preferred method of the vision community for a long time.
Since the pathbreaking achievements of modern convolutional neural networks beginning with the seminal work of Krizhevsky et al.
the focus has changed - the interest is more now on developing algorithms that learn to construct these invariant representations from transformed examples.
Goodfellow et al.
(2009) were among the ﬁrst to address the question of invariance in deep networks.
Cohen & Welling (2014) were among the ﬁrst to use representation theory of compact Lie groups to give it a sound mathematical framework.
Developing on on ideas from earlier works of Rao & Ruderman (1999) and Sohl-Dickstein et al.
(2010) on the Lie group model, Cohen and
Welling build a model for the abelian group of transformations SO (2) , the group of rotations of the plane.
Using their model they learn a nice basis for the underlying vector space in which the images sit, which allows them to read off the types and multiplicities of the irreducible representations of
SO (2) which occur in images under rotations.
In this basis exposing irreducible representations 1 , the action of an element of SO (2) on an image is very easy to describe.
From projections of images onto this basis, 2 Cohen and Welling obtain features of images which they use for classiﬁcation.
A number of experts attribute the impressive performance of modern deep convolutional networks (CNN’s) to the fact that more complex and abstract features are learned as one moves down the network.
Nonlinearity seems to be essential to learn such features.
Deep networks achieve their nonlinearity by the use of activation functions such as RELU, Krizhevsky et al.
Convolution networks are by design invariant to translations.
Given the incredible success of these deep networks the quest is now to build networks that have invariance to a larger group of symmetries.
In Bruna & Mallat (2013) the authors show that the ﬁrst two layers of the scattering network yield a powerful representation of images which is invariant to geometric transformations.
Gens & Domingos (2014) proposed deep symmetry networks, symnets, that form feature maps over large symmetry groups.
They showed that Symnets over the afﬁne group have smaller sample complexity.
More recently, Group Convolutional Networks (GCNN’s) were introduced in Cohen & Welling (2016a) and Harmonic nets were introduced by Worrall et al.
Both
GCNN’s and Harmonic nets are designed to learn representations of images invariant to a larger set of symmetries than translations.
The resulting networks perform impressively - the error rate reported by GCNN’s on the MNIST-rot data set is only 2.28%.
Harmonic nets are now the state of the art on MNIST-rot with a reported 1.62% error.
More recently Cohen & Welling (2016b) introduced Steerable CNN’s based on more sophisticated ideas from group representation theory.
This puts constraints on the network weights and architecture, and results in a reduction in the number of parameters to be learned.
Steerable CNN’s were shown to outperform ResNets He et al. and achieve state of the art results on CIFAR 10,100.
In both GCNN’s as well in in steerable CNN’s equivariance to the symmetries of a group G is achieved by a generalized form of convolution.
The standard convolution in CNN’s is replaced with the convolution of functions from G to C .
Kondor & Trivedi (2018b) proved the converse - they showed that any neural network which achieves equivariance with respect to the action of a group
G , necessarily implements such a generalized convolution of functions on G .
If one were to factor out depth, it appears that it is the nonlinearity of the activation functions used in deep networks that is responsible for their impressive performance.
But nonlinear functions like
RELU are mathematically hard to analyze.
It would be nice to design networks which construct features, just as deep networks seem to do in their various layers, with more mathematically amenable activation functions.
This is the ﬁrst motivation of this study.
Our other motivation comes from what is done in Pattern theory, Mumford (1997) where, to learn images, a stochastic model of the images to be learned is ﬁrst built.
To build a stochastic model of natural images it is important to view images under different scales simultaneously (Mumford (1997, Chapter 6)).
Scattering networks address this issue because they use a wavelet basis for images.
This leads us to the following motivating questions.
Can one use a mathematically amenable nonlinearity like tensor-product nonlinearity to design neural nets for image classiﬁcation, leveraging ideas from group representation theory?
How does one obtain complex abstract features from simple features using such a nonlinearity?
Does it help to view images under different scales simultaneously to build such neural nets?
In this paper we make a modest attempt to answer such questions.
We combine ideas from
Cohen & Welling (2014) and Kondor (2018a), and design neural networks leveraging notions of tensor products of group representations.
We decouple the classiﬁcation process in two steps - ﬁrst discover Cohen-Welling bases and then deploy them to train classiﬁers using ideas from Kondor.
P RELIMINARIES
We start with some deﬁnitions which set the stage for the work of Cohen & Welling (2014).
Let G be a group and let V be a ﬁnite-dimensional vector space over a ﬁeld - we will always assume that the underlying ﬁeld is the ﬁeld of complex numbers C , or the ﬁeld of real numbers R .
Let GL ( V ) denote the group of invertible linear transformations of V .
Deﬁnition
We say G acts on V if there is a group homomorphism ρ from G to GL ( V ) .
So g ∈ G acts on a vector v ∈ V by sending v to ρ ( g ) v .
If V has dimension d , then with respect to a basis of V , ρ ( g ) is given by a d × d matrix.
The action is then given by multiplying ρ ( g ) with v . We denote this by g · v .
Deﬁnition
We say a subspace W of V is G -invariant under the action of G if for all g ∈ G and all w ∈ W , ρ ( g ) w ∈ W .
If V has no proper subspace which is G -invariant
we say V is irreducible.
The restriction of the action of G to a G -invariant subspace W is called a subrepresentation.
Suppose V has an invariant subspace W , and we can also ﬁnd a G -invariant subspace U such that
W ⊕ U = V .
Then one can choose a basis of W and of U giving us a basis of V .
In this basis each ρ ( g ) will be a block diagonal matrix consisting of two blocks of size dim ( W ) and dim ( U ) .
For certain groups (called reductive groups) there is always such a U , and the group of rotations of the plane, SO (2) , is an example of such a group.
For reductive groups we can continue this process till we have a decomposition of V into irreducibles, V = U 1 ⊕ U 2 ⊕ . . .
⊕ U k .
Deﬁnition
If V and W are representations of a group G , a G -morphism (aka a G -equivariant map) from V to W is a linear map φ from V to W such that φ ( g · v ) =
g · φ ( v ) .
If V and W are vector spaces over complex numbers Schur’s lemma, Serre (1977), places restrictions on the dimension of the space of G -morphisms.
First, if V and W are irreducible representations of
G , Schur’s lemma states the space of G -morphisms between them is either one dimensional (i.e every
G -morphism between them is given by a scalar matrix) or zero dimensional (i.e the zero morphism is the only G -morphism between them).
This allows us to group irreducible representations into types, with two of them being of the same type when there is a nonzero G -morphism between them.
Given a decomposition of a vector space with a G -action into a direct sum of irreducibles as above, we collect all irreducibles of the same type together and write V = m 1 S t 1 ⊕ m 2 S t 2 · · · ⊕ c k S tk .
Here tk is an indexing set for types, S ti is an irreducible G -representation of type ti , and S ti and
S tj are of different types when i 6 =
j .
We then say S ti occurs in V with multiplicity m i in the decomposition of V .
Now assuming that V = ⊕ i
= k i =
1 m i S ti and W =
⊕ i
= k
i =1 n i S ti , it follows that the dimension of the space of G -morphisms between V and W is
P i m i n i .
Cohen & Welling (2014) considered the problem of supervised learning of images under the group
SO (2) .
Viewing an N × N image as an N 2 dimensional vector, an element of SO (2) acts on an N 2 dimensional vector by taking the vector representing the image to the vector representing the rotated image.
SO (2) is an abelian group (rotating an image by θ 1 and then by θ 2 or rotating it ﬁrst by θ
and then θ 1 yields the same image).
Hence, over complex numbers, the irreducible representations of SO (2) are one-dimensional (see, Kanatani (1990)).
When the underlying space is a real vector space (as was considered in Cohen & Welling (2014)), the types are parametrized by nonnegative integers n (over complex numbers the irreducible representations are parametrized by all integers).
The irreducible representation corresponding to type n = 0 , S 0 is one-dimensional and is an invariant under SO (2) .
For n > 0 , the irreducible representation S n is two dimensional (over complex numbers this splits into two one-dimensional representations parametrized by ± n ).
It follows, say, from Kanatani (1990, 2.3.7, 2.4.7) that there is a change of basis of the underlying vector space V with respect to which the action of the rotation group is given by a block diagonal matrix D with blocks of size 1 and 2 .
It is easy to describe these blocks.
Blocks of size 1 correspond to invariants, of type n = 0 , and have entry 1 .
The number of such blocks is called the multiplicity of the space of invariants in V .
The block corresponding to nonzero n is the elementary rotation matrix (cid:20) cos ( nθ ) − sin( nθ
) sin( nθ ) cos( nθ ) (cid:21) .
It follows that the number of such blocks is the multiplicity of irreducible type n in V .
So there exists an N × N orthogonal matrix W and a diagonal matrix D , as above, with the property that when an image x is rotated by θ the transformed image is given by WDW t x .
In
Cohen & Welling (2014), the authors address the question of determining the matrix W and the multiplicity with which a given n appears.
The input is collection of pairs ( x, y ) , of an image x
and it’s transformation y when it is rotated by an unknown angle θ .
The authors ﬁnd W using expectation-maximization over an interesting prior.
The W their algorithm outputs is almost always a small subspace of the image space, spanning only some of the irreducible subspaces (similar to an
SVD procedure which just outputs only the top few singular vectors).
So strictly speaking we should be calling this a CW-subbasis since it may not span the image space, but we ignore this technicality.
An image x is then projected onto the W constructed and the vector of projected values, W T x , were used by Cohen and Welling as elementary features of the image for classiﬁcation.
Recently Kondor (2018a) proposed a neural network for learning the behaviour and properties of complex many-body physical systems.
The neurons in this system operate entirely in the Fourier space.
The neurons compose activations following Schur’s lemma, thereby ensuring that the activations are covariant to rotations.
We pose our motivating questions more precisely using the language of Cohen-Welling.
Can we obtain a CW-basis of images and their rotations using tensor product nonlinearity?
How can elementary features obtained using a CW-basis be combined to get more complex abstract features for building good classiﬁers?
Can one discover CW-bases of images in two scales in tandem, with one inﬂuencing the discovery of the other?
Do such coupled bases perform better in classiﬁcation?
Can one use these coupled CW-bases interchangeably?
Our contributions We give an afﬁrmative answer to all the questions.
We give several architectures to construct such bases.
We use Kondor’s framework to construct a classiﬁer which takes elementary features and combines them using simple ideas from the representation theory of SO (2) .
We give a simple coupled autoencoder architecture which answers question 3 above - we use tensor product nonlinearity crucially.
While such bases are not better at classiﬁcation we show that they can be interchangeably used with a marginal drop in performance.
2. N OTIONS FROM REPRESENTATION THEORY
To describe our set up and explain our experiments we will need a few more notions from representation theory.
Throughout we will assume that the underlying group G is a reductive group, in fact the reader may think of G as SO (2) .
If V is a representation of a group G then the dual vector space V ∗ acquires a natural G action given by ( g · f )( v ) =
f ( g − 1 v ) , for f ∈ V ∗ , v ∈ V .
If V, U are representations of a reductive group then V ⊗ U is a representation of the product group G × G .
Furthermore, since G is a subgroup of
G × G under the diagonal embedding ( g mapping to ( g, g ) ), V ⊗ U is a representation of G .
And by what we described above V ⊗ U splits into a direct sum of irreducible representations identiﬁed by their types and multiplicities.
It follows that the tensor algebra T ( V )
=
⊕
i V ⊗ i of V is an (inﬁnite dimensional, algebraic) representation of G .
Remark 4 When G is SO (2) and 0
= i <
j then S i ⊗ S j splits into a direct sum S i + j ⊕ S j − i .
When i =
S i ⊗ S j is isomorphic to S j , (Kanatani (1990)).
To make precise the question about coupled bases and features from a coupled bases we need one more deﬁnition.
Deﬁnition
We say G -representations V and W are coupled if W is the image of a G -morphism of a ﬁnite dimensional subrepresentation of the tensor algebra of V , and V is the image of a Gmorphism of a ﬁnite dimensional subrepresentation of the tensor algebra of W .
The above deﬁnition is equivalent to requiring that W can be realized as a subrepresentation (or a quotient) of the tensor algebra of V and V can realized as a subrepresentation (or a quotient) of the tensor algebra of W .
Deﬁnition 6 Let V be the vector space of 14 × 14 images and let W be the vector space of 28 ×images.
Assume the rotation group SO (2) acts on both.
We say a Cohen-Welling basis X of V is coupled to a Cohen-Welling basis Y of W if the vector space dual of the subspace spanned by X (with its SO (2) -action) is coupled to the vector space dual of the subspace spanned by Y .
Recall that by projecting an image on to a CW-basis we get elementary features of the image which are used for classiﬁcation.
These elementary features are elements of the vector space dual of the space spanned by the CW-basis, hence the need for the word dual in the above deﬁnition.
Motivated by the above deﬁnition we say features obtained from coupled CW-bases are coupled features .
The deﬁnition above is motivated by our question of whether one can build CW-bases in two different scales with one inﬂuencing the discovery of the other and vice-versa.
Our deﬁnition suggests group SO (3) - for completeness we have included their results in
Table 1. that in order to generate coupled CW-bases we will need to view the images at different scales simultaneously, and use tensor product nonlinearity to generate them, thereby forcing the inﬂuence we are looking for.
Remark 7 One last remark is in order before we describe our setup and experiments.
We are working over the ﬁeld of real numbers, so Schur’s lemma does not apply in our situation.
Nevertheless when we look for SO (2) -morphisms between two irreducible vector spaces with an SO (2) -action we will assume that the morphisms are of the kind Schur’s lemma dictates - either we have no morphism or the morphism is given by a scalar matrix rId , r a real number and Id being the identity matrix.
E XPERIMENTS AND RESULTS
We use the notation W ℓ for a CW-basis of images of size ℓ × ℓ .
So W ℓ is a ℓ 2 × d matrix for some d (usually smaller than l 2 ).
Recall that if W ℓ is a CW-basis is is easy to describe how to rotate an ℓ × ℓ image x by an angle θ - it is the ℓ 2 -size vector W ℓ D ℓ ( θ ) W
Tℓ x , where D ℓ is the block diagonal matrix described in section 2, of size the cardinality of W ℓ .
If a column of W ℓ represents a basis vector which is invariant the corresponding block is 1 and if a pair of columns in W ℓ is a basis for an irreducible representation of type n , the block in D for this pair is the elementary rotation matrix described in section 2.
When there is no confusion we will ignore the subscripts.
We give three neural net architectures to discover CW-bases of images.
To construct a CW-basis we use MNIST-rot images as inputs.
In Cohen & Welling (2014) inputs are images generated using a standard Gaussian distribution with mean zero and images are rotated by an angle between 0 and 2 π .
3. L EARNING CW- BASES
An autoencoder(AE) architecture.
The schema is as given in Figure 1 (this architecture works for 14x14 images, 56x56 images).
We use MNIST-rot images as input.
The hyperparameters are multiplicities with which SO (2) irreducible representations occur in MNIST-rot images viewed as vectors in a 784 dimensional vector space with an action of SO (2) .
We choose nonzero multiplicities a j for representations S j with 0 ≤ j ≤ 24 giving us a subspace of the image space of dimension d W = a 0 +
P j 2 ∗ a j .
We denote this subspace by W 28 and learn a CW-basis also denoted W 28 , a 784 × d W matrix.
We initialize W 28 randomly.
We rotate a 28 x 28 image x by a random angle θ , to get y and project y onto W 28 to get ˆ y = W T 28 y .
We learn an SO (2) - equivariant map φ from the image space of W T 28 to a vector space L with an SO (2) -action, with multiplicities (hyperparameters) b j .
Consider φ (ˆ y ) ∈ L .
Now L ⊗ L gets a natural SO (2) action.
Recall that the tensor product of an SO (2) irreducible representation of type m and an SO (2) - irreducible representation of type type n yields irreducible representations of types m + n and m − n , (w.l.o.g m > n ).
We use this to compute the decomposition of φ (ˆ y ) ⊗ φ (ˆ y ) .
The last set of parameters to be learned are an SO (2) -equivariant map ψ from L ⊕ ( L ⊗ L ) to the image space of W T 28 .
Let z
=
ψ ( φ (ˆ y ) ⊕ ( φ (ˆ y ) ⊗ φ (ˆ y ))) .
We unrotate z using D ( − θ ) .
Then we reconstruct the image ˆ x = W 28 D ( − θ )
z We expect this to be close to x
so we minimize the reconstruction error | ˆ x − x | 2 with a regularizer to ensure that W T W is identity.
Choosing [10 , 5 , 5 , 5 , 5 , 4 , 4 , 4 , 4 , 4 , 3 , 3 , 3 , 3 , 3 , 2 , 2 , 2 , 2 , 2 ,
1 , 1 , 1 , 1 , 1] as our multiplicities for the a i ’s and [8 , 4 , 4 , 4 , 4 , 3 , 3 , 3 , 3 , 3 , 2 , 2 , 2 , 2 , 2 , 1 , 1 , 1 , 1 , 1] for the b i ’s, we learn a really sharp W 28 as shown in appendix Figure 7. 2.
Coupled Autoencoder(CAE) architecture .
In this setup, Figure 2, we learn W 14 and W 28 in tandem.
We feed both, an image X and a scaled down version of the image x to the network.
The network on top takes x and produces ˆ X a 28x28 image.
The bottom network takes X and produces ˆ x a 14x14 image.
The two networks are connected and we use the same W 28 , and W 14 in the top and bottom layer.
Hyperparameters are multiplicities of SO (2) -irreducible representations in W 28 and W 14 .
We learn the W 14 , W 28 and two SO (2) -equivariant maps φ and ψ .
The bottom network takes X , rotates it by θ and projects the resulting Y on the current W 28 , to get ˆ Y = W T 28 Y .
Applying φ we get φ ( ˆ Y ) .
We unrotate φ ( ˆ Y ) using D 14 ( − θ ) to get z .
We generate a 14x14 image using current W 14 to get ˆ x = W 14 z .
The top network takes x rotates it by the same θ .
This is projected onto the current W 14 to get ˆ y .
We apply ψ to ˆ y ⊕ (ˆ y ⊗ ˆ y ) and
[Figure]
UnrotateApply D ( θ )
[Figure]
Map φ
Map ψ
SO(2)EquivariantMaps ˆ y = W T
y
[Figure] φ (ˆ y ) ⊗ φ (ˆ y ) φ (ˆ y )
z = ψ ( φ (ˆ y ) ⊗ φ (ˆ y )
⊕ φ (ˆ y ))
D ( − θ )
z
ReconstructImage 28x28Image
ˆ x φ (ˆ y )
[Figure]
unrotate the resulting vector using D 28 ( − θ ) to get Z .
We use the current W 28 to get ˆ X
=
W
Z .
We minimize the reconstruction error | X − ˆ X | 2 + | x − ˆ x | 2 .
[Figure]
3 Bootstrapping W 14 to W 28 , W 28 to W 14 .
In this experiment we start with a known W 14 obtained using experiment 1 for 14x14 images, to learn a W 28 .
The setup to go from W 14 to W 28 is exactly the part of the network in Figure 2 going from X to ˆ X along the top.
The hyperparameters are the multiplicities of SO (2) -irreducible representations in W 28 .
We learn the SO (2) -equivariant ψ and W 28 by minimizing | X − ˆ X | 2 .
Similarly starting with a known W 28 we can learn W 14 by following the path in Figure 2 from x to ˆ x along the bottom.
3. C LASSIFICATION USING THE LEARNED CW- BASIS
Figure 3 shows a 2 layer classiﬁcation network for 28 × 28 images - this can be extended to any number of layers and other image sizes.
This is a logical extension of the auto encoder network (Experiment 1).
It is a CNN in Fourier domain using tensor product nonlinearity.
Assume the dimension of W 28 is d W .
We project the image x onto W 28 to get W T 28 x .
Let L 1 be a vector space with an SO (2) -action - hyperparameters are multiplicities l 1 j of irreducibles SO (2) -representations S j in L 1 .
We learn an SO (2) -equivariant map φ 1 from the image space of W T 28 to L 1 .
The SO (2) - decomposition of l 1 = φ 1 ( W T 28 x ) ⊕ ( φ 1 ( W T 28 x ) ⊗ φ 1 ( W T 28 x )) is computed.
This is the input to the next layer.
There we choose a vector space L 2 with a SO (2) -action which splits into irreducibles S j with multiplicities l 2 j (hyperparameters).
We learn a second SO (2) -equivariant map φ 2 from L 1 ⊕ ( L 1 ⊗ L 1 ) to L 2 and compute φ 2 ( l 1 ) .
We compute the tensor product of φ 2 ( l 1 ) with itself and produce l 2 =
φ 2 ( l 1 ) ⊕ ( φ 2 ( l 1 ) ⊗ φ 2 ( l 1 )) .
This is the input to the next layer.
So we learn as many SO (2) -equivariant maps as there are layers.
In the ﬁnal layer, l f is projected onto the invariant space (the multiplicity of S 0 in L f ⊕ ( L f ⊗ L f ) and fed to a soft max classiﬁer.
[Figure]
3. R ESULTS 1. Construction of CW-basis
We construct CW-basis W 14 for 14 × 14 images and CW-basis W 28 for 28 × 28 in the various architectures.
We evaluate these bases in terms of image reconstruction error (MSE).
We also used rotation reconstruction error by comparing with scikits-image rotation.
Figure 4 and Figure 5 shows graphs of the error functions for W 28 constructed in an AE architecture and a CAE as a function of the number of samples given to the AE and CAE architectures to learn the W 28 .
As the ﬁgures show, even when the number of samples is as small as 50, we discover CW-bases which are good at rotation and also reconstruction in both AE and CAE architectures.
Results on Classiﬁcation In Figure 6 we plot the accuracy of these various W 28 obtained above when deployed for classiﬁcation.
Our classiﬁer was trained and tested on MNIST-rot.
We plot the accuracy of the classiﬁer as a function of the number of samples used to construct the CWbases.
When the number of samples is as low as 50 a CAE- W 28 performs better than an AE- W 28 .
Beyond 100 samples the difference is insigniﬁcant.
The plots using a bootstrapping CW-basis are not given as they were very similar to that of the AE.
In Table 1 we give the accuracy of our classiﬁer when trained and tested on combinations using MNIST(NR) and MNIST-rot(R).
The R/NR column for example denotes the accuracy when trained on MNIST-rot and tested on MNIST.
For comparison we use Planar, Spherical CNN accuracies given in Cohen et al.
We also compare with FFS2CNN, (see Kondor et al. (2018c)).
The fourth row shows the accuracy of our classiﬁer that used a CAE- W 28 .
This was obtained using CAE architecture which was given 12000 training samples of MNIST-rot to discover the W 28 .
Our classiﬁer performs better in all the scenarios.
Coupling interchangeability To test how coupled the CAE- W 28 and CAE- W 14 are, we did the following experiment - the classiﬁer trained in row 4 with CAE- W 28 was presented with down sized 14x14 images for classiﬁcation.
No additional training was done.
Instead we use the top half of the coupling network from Figure 2 - given a test image y we compute
ˆ y = W T
y using the CAE- W 14 .
Then ψ ((ˆ y ⊗ ˆ y ) ⊕ ˆ y ) is fed to the trained classiﬁer of row 4.
These results are reported in row 5 as [28/14 Tensor].
For comparison we took the 14x14 images and scaled them up to 28x28. And then fed them to the trained classiﬁer of row 4.
These results are reported in Row 6 as [28/14 Scale]. Using CAE- W 14 is better, indicating that a coupled bases retains scale information well.
We repeated the same experiment when the coupled network is given only 500 samples to learn the CAE- W 28 and CAE- W 14 .
As row 8 shows, performance is till very good. [Table] 3. I MPLEMENTATION
We implement our autoencoders and classiﬁers as neural nets in Python (TensorFlow) and used Adam optimizer. For the classiﬁer we use a 3 layer network with 28K parameters.
In comparison Spherical networks use 58K and FFS2CNN 342K.
C ONCLUSION AND FUTURE WORK
We answer in the afﬁrmative all our motivating questions - we design architectures which learn CW-bases and classify using only tensor product nonlinearity.
We obtain some state of the art results.
It appears that coupling bases do retain scale information and can be interchangeably used for classiﬁcation albeit with a small drop in performance.
There is no appreciable advantage in using coupled bases for classiﬁcation excepting when the bases are learned using small samples of inputs.
Tensor product nonlinearity allows us to decouple the classiﬁcation problem in two steps.
One advantage of decoupling the process is that the basis can be obtained ofﬂine.
Our ideas can be extended to other groups.
We have begun experiments with the action of the S n on images.
We are testing our architectures on CIFAR-10.
One issue we run into when using architectures with a lot of depth, is that accuracy increases and then drops.
A similar issue is resolved in RES-nets
He et al.
by computing residual functions.
In ongoing work we proceed similarly since the residual function is SO (2) -equivariant.
Robustness of classiﬁers to noise is an active area of research, (see Goodfellow et al.
We believe our classiﬁers built from bases learnt in a CAE architecture should be robust to noise - our belief comes from the fact that coupling bases can be used interchangeably.
We plan to investigate this.
A CKNOWLEDGEMENTS
The authors would like to thank Infosys foundation for a generous grant.
Appendices A P ROOFS
Here we prove that the bases obtained in the coupling architecture satisfy the deﬁnition of coupling.
The proof is straightforward assuming that we have zero loss.
Let W 28 and W 14 be the ﬁnal CW-bases learned.
Following the bottom half of the network describing the coupling architecture, we see that under the morphism φ , the image space of W T 28 surjects onto the image space of W T 14 .
Surjection follows since the dotted arrow going from W 14 on the top to W 14 below shows that we can start with any linear combination of the basis elements of Wat the top and obtain it as the image of φ applied to an appropriate linear combination of basis elements of W 28 .
So the image space of W T 14 is isomorphic to a quotient of the image space of W T 28 .
On the other hand following the top half of the network it is clear that ψ maps the image space of
W T 14 ⊕ ( W T 14 ⊗ W T 14 ) surjectively onto the image space of W T 28 .
Surjectivity follows by reasoning as above.
Hence the bases obtained in the coupling architecture satisfy deﬁnition 6.
B A LGORITHM
Algorithm 1: AutoEncoder
Input: X i ∈ R 784 , i ∈ { 1 , 2 , . . . }
Hyper-parameters : a j , b j , β
Let d a = a 0 +
P j =1 2 × a j ; d b = b 0 +
P j =1 2 × b j
Output: W 28 ∈ R 784 × d a
Output: φ ∈ R d b × d a // Equivariant linear map (Block diagonal
Output: ψ ∈ R d a × ( d b + d b 2 ) // matrix) as dictated by Remarkfor i = 1 , 2 , . . .
do θ i ∈
U [0 , 360) //
Choosen uniformly at random
Y i =
Rotate ( X i , θ i ) //
Rotate is the standard image rotation
ˆ Y i
=
{+W T 
+}
Y
i
Z i =
ψ ( φ ( ˆ Y i ) ⊗ φ ( ˆ Y i )
⊕ φ ( ˆ Y i ))
ˆ X i =
D ( − θ i )
Z i //
Unrotate the image end
L =
P i |
X i − ˆ X i |// Loss function
W 28 , φ, ψ = min W 28 ,φ,ψ ( L + β | W T 28 W 28 − Id d a × d a | 1 ) return W 28 , φ, ψ
C V ISUALIZATION OF CW B ASIS
Figure 7 shows a few basis elements learned using AE schema.
D T ABLE WITH STATISTICS
Table 2 gives the mean accuracies in percentage over 5 runs and the standard deviation in paranthesis.
[Table]