E QUIVARIANT NEURAL NETWORKS AND EQUIVARIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Equivariant neural networks are special types of neural networks that preserve some symmetries on the data set.
In this paper, we provide a method to modify a neural network to an equivariant one, which we call equivariﬁcation .
I NTRODUCTION
One key issue in deep neural network training is the difﬁculty of tuning parameters, especially when the network size grows larger and larger Han et al.
In order to reduce the complexity of the network, many techniques have been proposed by analyzing the structural characteristics of data, for example, sparsity Wen et al. (2016), invariance in movement Goodfellow et al.
One of the most important structural characteristics of data is symmetry. By utilizing the translation symmetry of an object in the photo, convolutional neural network (CNN)(Krizhevsky et al. (2012)) uses shared ﬁlters to reduce the number of parameters compared with fully connected networks.
However, to handle the case of rotation and reﬂection, people usually use the data augmentation approach to generate additional input data that has a bunch of training images with different rotation angles of the original images.
In contrast to data augmentation approach, another idea is to design more sophisticated neural networks, such that the input data with certain symmetries can be trained together and applied to reduce the training complexity.
Recent attempts have been made in the equivariant CNN Cohen & Welling (2016a);
Cohen et al. ; Weiler et al.
These existing works target at special cases and cannot be easily generalized to arbitrary networks and arbitrary symmetries.
In this paper, we take advantage of the symmetry of the data set and provide a general method to modify an arbitrary neural network so that it preserves the symmetry.
The process is called equivariﬁcation .
A key feature is that our equivariﬁcation method can be applied without detailed knowledge of a layer in a neural network, and hence, can be generalized to any feedforward neural networks.
Another feature is that the number of parameters in the new neural network that we need to train is the same as the original one, if we equivarify the original network globally (See the second paragraph in Section 4 for details).
In addition, we can also output how each data instance is changed from the canonical form (for example, how many degrees the cat image is rotated from original upside-up image) using the same network.
We rigorously prove that our equivariﬁcation method produces truely equivariant neural networks.
Practically, we equivarify a CNN as an illustration.
We conduct experiments using the MNIST data set where the resulted equivariant neural network predicts the number and also the angle.
If we forget the angle and keep only the number, we get an invariant neural network.
By the symmetry of the data set, we mean a group action on the data set.
Let us consider a simple cat image classiﬁcation example.
Then the data set can be the set of all images of a ﬁxed size. If we consider the rotation symmetry, we can rotate an image and the resulting image still lies in the data set. In other words, the group of rotations acts on the data set.
One can build a cat classiﬁer that assigns a number between 0 and 1 to each image indicating the probability that it is an image of a cat.
If one image is a rotation (say 90 degree counterclockwise) of another one, then it makes sense to require a classiﬁer to assign the same probability to these two images.
A classiﬁer that satisﬁes this property is said to be invariant under 90-degree rotation, which is a special case of being equivariant under 90-degree rotation.
To give an example of an (noninvariant) equivariant neural network, we furthermore require our classiﬁer not only produces the probability of being a cat image, but also outputs an angle, say in { 0 , 90 , 180 , 270 } (more precisely, the probability of each angle).
Suppose that the equivariant classiﬁer predicts that an image is rotated by 90 degrees, then it should predict a 270 degrees rotation for the same image but rotated bydegrees.
Not only does it make sense to require the cat classiﬁer to be equivariant, but also it is more “economical” to be equivariant if it is implemented correctly. Roughly speaking, a regular classiﬁer “treats” an image and its rotated ones as separated things, but ideally an equivariant neural network “sees” their connections and “treats” them together. From another point of view, given a regular neural network, if we apply the data augmentation method carefully, since the training data is symmetric, it is possible that after training, the network is (approximately) equivariant (depending on the initialization). The fact that it is equivariant implies there is now symmetry among the parameters.
For example, some parameters are the same as other parameters, so there is redundancy. While in our equivariant neural network, the equivariance of our neural network is built into the structure of the neural network by sharing parameters, and in particular, it is independent of the loss function, initialization, and the data set. For example, for the training data, it does not make any difference in results whether we prepare it by randomly rotating and recording the angles, or not rotating and labeling everything as degree 0 . In our approach, to make a neural network equivariant, at each layer other than the input layer we add a bunch of neurons (multiplying by the order of the group), but we don’t introduce new parameters.
Instead the added neurons share parameters with the original ones. 1. R ELATED W ORK Invariance in neural networks has attracted attention for decades, aiming at designing neural networks that capture the invariant features of data, for example, face detection system at any degree of rotation in the image plane Rowley et al. (1998), invariance for pattern recognition
Barnard & Casasent (1991), translation, rotation, and scale invariance Perantonis & Lisboa (1992).
Recently, several works have started to look into equivariant CNN, by studying the exact mapping functions of each layer Cohen & Welling (2016a;b); Cohen et al.
; Marcos et al. Lenssen et al. ; Cohen et al. , and some symmetries are studied such as translation symmetry, rotation symmetry.
These methods have been used in different application domains, such as in remote sensing Marcos et al. (2018), digital pathology Veeling et al. (2018), galaxy morphology prediction Dieleman et al.
There are also works that towards the direction of constructing an equivariant neural network without having to specify the symmetry Sabour et al.
As far as we know, our construction provides the ﬁrst truly equivariant neural network (See Section 5.1). 1. A CKNOWLEDGEMENT
We thank all the anonymous referees for helpful and constructive suggestions.
P RELIMINARIES
In this section, we talk about some basics in group theory, like group actions, equivariance, etc.
For those who would like to get a more close look at the group theory and various mathematical tools behind it, please refer to any abstract algebra books, for example, Lang (2002).
Here, we ﬁrst give a couple of deﬁnitions about groups in order to help readers quickly grasp the concepts.
Deﬁnition 2.1.
A group ( G, · ) consists of a set G together with a binary operation “ · ” (which we usually call multiplication without causing confusing with the traditional sense of multiplication for real numbers) that needs to satisfy the four following axioms.
1) Closure: for all a, b ∈ G , the multiplication a · b ∈ G . 2) Associativity: for all a, b, c ∈ G , the multiplication satisﬁes ( a · b ) · c = a · ( b · c ) .
3) Identity element: there exists a unique identity element e ∈ G , such that, for every element a ∈ G , we have e · a = a · e = a . 4) Inverse element: for each element a ∈ G , there exists an element b ∈ G , denoted a − 1 , such that a · b = b · a = e , where e is the identity element.
Note that in general, commutativity does not apply here, namely, for a, b ∈ G , a · b = b · a does not always hold true.
For example, all integers with the operation addition ( Z , +) .
One can easily check that the four axioms are satisﬁed and 0 is the identity element.
As another example, a group consists of a set { 0 , 1 } together with the operation + (mod 2 )
where
+1 = 1+0 = 1 and 1+1 = 0+0 = 0 .
The identity element is 0 .
When applying this to the image processing tasks, this can be interpreted as follows.
1 represents the action of rotating an image by 180 o and 0 represents the action of not rotating an image.
Then 0 + 1 represents the combination of operations that we ﬁrst rotate an image by 180 o and then keep it as it is, so that the ﬁnal effect is to rotate an image by 180 o ; while 1 + 1 represents that we ﬁrst rotate an image by 180 o and then rotate again by 180 o , which is equivalent to that we do not rotate the original image.
Similarly, we can deﬁne the element 1 as the operation of ﬂipping an image vertically or horizontally, which we can give similar explanations for the group.
Therefore, instead of using translations, rotations, ﬂippings, etc., we can use abstract groups to represent operations on images, and hence, we are able to design corresponding equivariant neural networks disregarding the operations of images (symmetries of data) and just following the group representation.
In the following, we give a deﬁnition about group actions.
Let X be a set, and G be a group.
Deﬁnition 2.2.
We deﬁne a G -action on X to be a map
T : G × X
→ X (on the left) such that for any x ∈ X • T ( e, x ) = x , where e ∈ G is the identity element, • for any g 1 , g 2 ∈ G , we have
[Equation]
Frequently, when there is no confusion, instead of saying that T is a G -action on X or equivalently that G acts on X via T , we simply say that G acts on X ; and T is also understood from the notation, i.e., instead of T ( g, x ) we simply write gx , and the above formula becomes g 1 ( g 2 x )
= ( g 1 g 2 ) x .
We say G acts trivially on X if gx = x for all g ∈ G and x ∈ X .
Let X, Y be two sets, and G be a group that acts on both X and Y .
Deﬁnition 2.3.
A map F : X → Y is said to be G -equivariant, if F ( gx ) = gF ( x ) for all x ∈ X and g ∈ G .
Moreover, if G acts trivially on Y then we say F is G -invariant.
Example 2.4 .
Let X be the space of all images of 28 × 28 pixels, which contains the MNIST data set.
Let G be the cyclic group of order 4 .
Pick a generator g of G , and we deﬁne the action of g on
X by setting gx to be the image obtained from rotating x counterclockwise by 90 degrees.
Let Y be the set { 0 , 1 , 2 , ..., 9 } × { 0 , 90 , 180 , 270 } .
For any y = ( num , θ ) ∈ Y we deﬁne gy :=
( num , ( θ + 90) mod 360 ) .
An equivariant neural network that classiﬁes the number and rotation angle can be viewed as a map
F from X to Y .
Equivariance means if F ( x ) =
( num , θ ) then F ( gx ) =
( num , ( θ + 90) mod 360 ) , for all x ∈ X .
Thus, we can see that we can model each layer in the neural network as a group G that acts on a set X , where we can interpret the set X as input to this layer and the group action as the function mapping or operation of this layer.
By abstracting the behaviors on the original input data using groups (for example, using a same group to represent either rotation by 180 o and ﬂipping, or even more abstract operations during intermediate layers), we are able to apply group actions on different sets
X 1 , X 2 , X 3 , . . .
(where each one represents input to different layers) and design similar equivariant network structures based on an original one.
E QUIVARIFICATION
In this section, we talk about the detailed method of performing equivariﬁcation and its theoretical foundation.
This is the key part of the paper to understand our proposed equivariﬁcation method.
Those who would like to avoid mathematical proofs can directly go to the examples we provide to get intuitive ideas of how we construct the equivariant neural networks.
In this section we ﬁx X and Z to be two sets, and G to be a group that acts on X .
Deﬁnition 3.1.
Deﬁne the G -product of Z to be
[Equation]
the set of all maps from G to Z .
We deﬁne a G -action on Z × G by
[Equation]
where gs as a map from G to Z is deﬁned as
[Equation]
for any g ′ ∈ G .
We have the projection map p : Z × G → Z deﬁned by [Equation]
for any s ∈ Z × G where e ∈ G is the identity element.
Then
Lemma 3.2.
For any map F : X → Z , there exists a unique G -equivariant map b F : X → Z × G such that p ( b F ( x ))
=
F ( x ) for all x ∈ X .
Proof.
For any x ∈ X , we deﬁne b F ( x ) as a map from G to Z by
[Equation]
, for any g ∈ G .
To see that b F is G -equivariant, we need to check for any x ∈ X and g ∈ G , b
F ( gx ) =
g ( b F ( x )) .
For any h ∈ G , ( b F ( gx ))
( h ) =
F ( h − 1 gx ) by the deﬁnition of b F , while ( g ( b F ( x )))
( h ) =
( b F ( x ))
( g − 1 h ) =
F ( h − 1 gx ) .
We leave the proof of uniqueness to the readers. Remark 3.3 .
In the Deﬁnition 3.1 and Lemma 3.2, G is an arbitrary group and Z is an arbitrary set. It is easy to see that we can adjust them, if we consider other categories. For example, when G is a compact Lie group and Z is a differentiable manifold, we can re-deﬁne Z ×
G to be the space of differentiable maps from G to Z ; when G is a non-compact Lie group and Z is a differentiable manifold, we can consider the space of compact supported smooth maps.
When we implemented the neural network, if G is inﬁnite, we have to approximate it by a ﬁnite subset of G .
Then in this case, we need to work in the realm of approximately equivariant.
For this implementation reason, we restrict our group G to be a ﬁnite group for the rest of the paper.
This lemma can be summarized as the commutative diagram in Figure 1.
It motivates the following general deﬁnition.
Deﬁnition 3.4.
We say a tuple ( b Z, T, p ) a G -equivariﬁcation of Z if
[Figure]
• b Z is a set with a G -action T ; • p is a map from b Z to Z ; • For any set X with a G -action, and map F : X → Z , there exists a G -equivariant map b F : X → b Z such that p ◦ b F = F .
Here ◦ denotes the composition of maps.
As usual, T will be omitted from notation.
In Section 4 we will see applications of G -equivariﬁcation to neural networks.
From Lemma 3.
we know that the triple of the G -product Z × G , the G -action deﬁned in the Formula 3.1, and the projection p deﬁned in Formula 3.2 is a G -equivariﬁcation.
There are other G -equivariﬁcations.
See Appendix for more discussion.
Example 3.5 .
Let G be the cyclic group of order 4 .
More concretely, we order elements of G by ( e, g, g 2 , g 3 ) .
The set Z × G can be identiﬁed with Z ×
=
Z × Z × Z × Z via the map
[Equation]
[Equation]
Then G acts on Z × 4 by g ( z 0 , z 1 , z 2 , z 3 ) =
( z 3 , z 0 , z 1 , z 2 ) , and the projection map p : Z × 4 → Z is given by ( z 0 , z 1 , z 2 , z 3 ) 7→ z 0 .
Let F : X → Z be an arbitrary map, then after the identiﬁcation b F becomes a map from X to Z × 4 and
[Equation]
One can check that b F is G -equivariant.
The map p is given by [Equation] 0 , z 1 , z 2 , z 3 ) =
z 0 .
It is easy to see that p ◦ b F = F .
A PPLICATION TO NEURAL NETWORKS
In this section, we show through an example how our proposed equivariﬁcation method works.
Let { L i : X i → X i +1 } n i =0 be an n -layer neural network (which can be CNN, multi-layer perceptrons, etc.).
In particular, X 0 is the input data set, and X n +1 is the output data set.
Let G be a ﬁnite group that acts on X 0 .
Let L be the composition of all layers
[Equation]
Then we can equivarify L and get maps b L : X 0 → b X n and p : b X n +1 → X n +1 .
Then b L is an equivariant neural network.
Alternatively, one can construct an equivariant neural network layer by layer.
More precisely, the equivariant neural network is given by { \ L i ◦ p
i : b X i →
b X i +1 } ni =0 , where \ L i ◦ p
i is the equivariﬁcation of L i ◦ p i for i ∈ { 0 , 1 , ..., n } , b X
= X 0 and p
= id is the identity map (See Figure 2).
More precisely, by the commutativity of Figure 2 we know that
[Equation]
Then both \ L n ◦ p n ◦ \ L n − 1 ◦ p n − 1 ◦ · · · ◦ \ L 0 ◦ p 0 and b L are equivariﬁcations of L .
Suppose that for both equivariﬁcations we have chosen b X n +1 to be X n +1 × G .
Then by the statement in Theorem 3.2, we have
[Equation]
· ◦ \
[Equation]
L.
[Figure]
Sometimes, other than equivarifying the map L i ◦ p
i : b X i → X i +1 , it makes sense to construct some other map L ′ i from b X i to some other set X ′
i
+1 , and then we can equivarify L ′ i .
This makes the equivariant neural network more interesting (see the example below).
Example 4.1 .
Let the 0 -th layer L 0 : X 0 → X 1 of a neural network that is deﬁned on the MNIST data set be a convolutional layer, and X 1 = R ℓ 1 , where ℓ 1 = 28 × 28 × c 1 , and c 1 is the number of channels (strides = (1 , 1) , padding = ‘same’).
Let G =
e, g, g 2 , g 3 } be the cyclic group of order 4 such that g acts on X 0 as the 90 -degree counterclockwise rotation.
Then we construct b
L 0 : X
→ R 4 ℓ 1 by
[Equation]
For the next layer, instead of equivarifying L 1 ◦ p 1 : R 4 ℓ 1 → R ℓ 2 , we can construct another convolutional layer directly from R 4 ℓ 1 by concatenating the four copies of R ℓ 1 along the channel axis to obtain R 28 × 28 × 4 c 1 , and build a standard convolution layer on it.
This new construction of course changes the number of variables compared to that of the original network.
From the above analysis and Lemma 3.2, it is not hard to derive the following summary.
Main result Let X =
{ L i : X i
→ X i +1 } 0 ≤ i ≤ n +1 be an original neural network that can process input data { x j 0 } j ⊆ X 0 and labelling data { x jn +1 } j ⊆ X n +1 .
Let G be a ﬁnite group that acts on
X 0 .
The proposed G -equivariﬁcation method is able to generate a G -equivariant neural network b
X = { b L i : b X i →
b X i +1 } 0 ≤ i ≤ n +1 that can process input data { x j 0 } j ⊆ X
=
b X 0 and enhanced labeling data { b x in +1 } j
⊆ b X n +1 .
Furthermore, the number of parameters of b
X is the same as that of X .
E XPERIMENTS
In this section, we show our experiments on the MNIST dataset, which achieve promising performance
Our designed equivariant neural network using the proposed equivariﬁcation method is shown in
Figure 3.
Note that equivariﬁcation process does not increase the number of variables.
In our case, in order to illustrate ﬂexibility we choose not to simply equivarify the original neural network, so the layer conv2 and conv3 have four times the number of variables compared to the corresponding original layers.
Next, we discuss the labeling of the input data.
Since the neural network is G -equivariant, it makes sense to encode the labels G -equivariantly.
Suppose ( x 0 , x n +1 ) ∈
X 0 × X n +1 is one labelled data point.
Then in the ideal case, one hopes to achieve L ( x 0 ) =
x n +1 .
Assuming this, to give a new label for the data point x 0 for our equivariant neural network we need to deﬁne b x n +1 =
b L ( x 0 ) .
For this, it is sufﬁcient to deﬁne b
L ( x 0 )( g ) for all g ∈ G .
By equivariance, b L ( x 0 )( g ) =
L ( g − 1 x 0 ) .
If g = e then b L ( x 0 )( g ) = L ( x 0 ) = x n +1 .
If g 6 = e , it is very likely that the data g − 1 x 0 originally does not have a label, so we do not know ideally what L ( g − 1 x 0 ) should be. In the naive data augmentation approach, L ( g − 1 x 0 ) is labeled the same as L ( x 0 ) hoping to get L as close to an invariant map as possible. In our case, we do not have such restriction, since we do not need L to be invariant. In
X X× X×
X×
Xb Xconv conv1 ◦
g
−conv1 ◦ g −conv1 ◦ g −
X×
X× X×
Xb Xconv conv2 ◦
g −conv2 ◦ g −conv2 ◦ g −
X 3 conv
X [Figure]
X× X×
X×
Xb X 5 = R
[Figure]
We equivarify conv2, stack the four copies of X 2 , and feed it to another convolution layer conv3.
Now instead of equivarifying conv3, we add layer pool and layer dense (logistic layer), and then we equivarify their composition dense ◦ pool ◦ conv3 ◦ g − 1 and get b
X 5 = R 40 .
To get the predicted classes, we can take an argmax afterwards.
practice, X n +1 is a vector space, and we choose to label L ( g − 1 x 0 ) by the origin of X n +1 .
In our
MNIST example, this choice is the same as the following.
For m ∈ { 0 , 1 , 2 , ..., 9 } denote
[Equation]
For an unrotated image x 0 ∈ X 0 that represents the number m , we assign the label e m ⊕ 0 ⊕ 0 ⊕ 0 ∈ R.
Then based on the equivariance, we assign
[Equation]
For each testing image in the MNIST data set, we randomly rotate it by an angle of degree in { 0 , 90 , 180 , 270 } , and we prepare the label as above.
For the training images, we can do the same, but just for convenience, we actually do not rotate them, since it won’t affect the training result at all.
5. E QUIVARIANCE V ERIFICATION
To spot check the equivariance after implementation, we print out probability vectors in R 40 of an image of the number 7 and its rotations.
We see that the probability vectors are identical after a shift by 10 slots.
See Figure 4.
5. A CCURACY
Here we count the prediction as correct if both the number and the angle are predicted correctly.
The accuracy of our neural network on the test data is 96 . 8% .
This is promising when considering the fact that some numbers are quite hard to determine its angles, such as 0 , 1 , and 8 .
C ONCLUSION
In this paper, we proposed an equivariﬁcation method to design equivariant neural networks that are able to efﬁciently process input data with symmetries.
Our proposed method can be generalized to arbitrary networks or functions by leveraging group action, which enables our design to be uniform across layers of feedforward neural networks, such as multi-layer perceptrons, CNNs, without being aware of the knowledge of detailed functions of a layer in a neural network.
As an illustration example, we show how to equivarifying a CNN for image classiﬁcation.
Results show that our proposed method performs as expected, yet with signiﬁcantly reduction in the design and training complexity.
A A PPENDIX - M ORE A BOUT E QUIVARIFICATION
In Section 3 we deﬁne ( Z × G , p ) as an example of G -equivariﬁcation.
In this section, we show that it is “minimal” in the sense of its universal property.
Lemma A.1 (universal property) .
For any G -equivariﬁcation ( b Z ′ , p ′ ) of Z , there exists a Gequivariant map π : b Z ′
→ Z × G such that p ′
=
p ◦ π .
Moreover, for any set X and map F : X → Z , the lifts b F : X → Z × G and b F ′ :
Z → b Z ′ of F satisfy π ◦ b F ′ =
b F. (See Figure 5.)
Proof.
We deﬁne the map π : b Z ′
→ Z × G by [ π ( b z ′ )]
( g ) =
p ′ ( g − 1 b z ′ ) , where b z ′
∈
b Z ′ and g ∈ G .
To show p ′
=
p ◦ π , for any b z ′ ∈
b Z , we check p ◦ π ( b z ′ )
=
p [ π ( b z ′ )]
=
[ π ( b z ′ )]
( e ) = p ′ ( b z ′ ) .
To show π is G -equivariant, for any b z
′ ∈
b Z , and h ∈ G , we compare π ( h b z ′ ) and hπ ( b z ′ ) : for any g ∈ G , [ π ( h b z ′ )]
( g ) =
p ′ ( g − 1 h b z ′ ) and [ hπ ( b z ′ )]
( g ) =
[ π ( b z ′ )]
( h − 1 g ) =
p ′ ( g − 1 h b z ′ ) .
Lastly, we show π ◦ b F ′ =
b F .
Note that π ◦ b F ′ is a G -equivariant map from X to Z × G , and
[Equation]
so by the uniqueness of Lemma 3.2, we get π ◦ b F ′ =
b F.
Now we discuss about ﬁnding a “smaller” equivariﬁcation in another direction, shrinking the group by bring in the information about X .
Let N = { g ∈ G | gx = x for all x ∈ G } , the set of elements in G that acts trivially on X .
It is easy to check that N is a normal subgroup of G .
We say G acts on X effectively if N = { e } .
In the case when G does not act effectively, it makes sense to consider the G/N -product of Z , where G/N is the quotient group.
More precisely, consider
Z × G/N = { s : G/N → Z } , which is smaller in size than Z × G .
For any map F : X → Z , we can get a G/N -equivariant lift b F of F following the same construction as Lemma 3.2 (with G replaced by G/N ).
Since G maps to the quotient G/N , we have that G acts on Z × G/N and b F is also G -equivariant.