R OT DCF: D ECOMPOSITION OF C ONVOLUTIONAL F ILTERS FOR R OTATION -E QUIVARIANT D EEP N ETWORKS
A BSTRACT
Explicit encoding of group actions in deep features makes it possible for convolutional neural networks (CNNs) to handle global deformations of images, which is critical to success in many vision tasks.
This paper proposes to decompose the convolutional ﬁlters over joint steerable bases across the space and the group geometry simultaneously, namely a rotation-equivariant CNN with decomposed convolutional ﬁlters (RotDCF).
This decomposition facilitates computing the joint convolution, which is proved to be necessary for the group equivariance.
It signiﬁcantly reduces the model size and computational complexity while preserving performance, and truncation of the bases expansion serves implicitly to regularize the ﬁlters.
On datasets involving in-plane and out-of-plane object rotations, RotDCF deep features demonstrate greater robustness and interpretability than regular CNNs.
The stability of the equivariant representation to input variations is also proved theoretically.
The RotDCF framework can be extended to groups other than rotations, providing a general approach which achieves both group equivariance and representation stability at a reduced model size.
I NTRODUCTION
While deep convolutional neural networks (CNN) have been widely used in computer vision and image processing applications, they are not designed to handle large group actions like rotations, which degrade the performance of CNN in many tasks (Cheng et al., 2016; Hallman & Fowlkes, 2015; Jaderberg et al., 2015b; Laptev et al., 2016; Maninis et al., 2016).
The regular convolutional layer is equivariant to input translations, but not other group actions.
An indirect way to encode group information into the deep representation is to conduct generalized convolutions across the group as well, as in Cohen & Welling (2016a).
In theory, this approach can guarantee the group equivariance of the learned representations, which provides better interpretability and regularity as well as the capability of estimating the group action in localization, boundary detection, etc.
For the important case of 2D rotations, group-equivariant CNNs have been constructed in several recent works, e.g., (Weiler et al., 2017),
Harmonic Net (Worrall et al., 2017) and Oriented Response Net (Zhou et al., 2017).
In such networks, the layer-wise output has an extra index representing the group element (c.f. Table 1), and consequently, the convolution must be across the space and the group jointly (proved in Section 3.1).
This typically incurs a signiﬁcant increase in the number of parameters and computational load, even with the adoption of steerable ﬁlters (Freeman et al., 1991;
Weiler et al., 2017; Worrall et al., 2017).
In parallel, low-rank factorized ﬁlters have been proposed for sparse coding as well as the compression and regularization of deep networks.
In particular, Qiu et al. (2018) showed that decomposing ﬁlters under non-adaptive bases can be an effective way to reduce the model size of CNNs without sacriﬁcing performance.
However, these approaches do not directly apply to be group-equivariant.
We review these connections in more detail in Section 1.1.
This paper proposes a truncated bases decomposition of the ﬁlters in group-equivariant CNNs, which we call the rotation-equivariant CNN with decomposed convolutional ﬁlters (RotDCF).
Since we need a joint convolution over R 2 and SO (2) , the bases are also joint across the two geometrical domains, c.f. Figure 1.
The beneﬁts of bases decomposition are three-fold: (1) Reduction of the number (2) Implicit regularization of the convolutional ﬁlters, leading to improved robustness of the learned deep representation shown experimentally in Section 4; (3) Theoretical guarantees on stability of the equivariant representation to input deformations, which follow from a generic condition on the ﬁlters in the decomposed form, c.f. Section 3.2.
1. R ELATED W ORK
Learning with factorized ﬁlters.
In the sparse coding literature, Rubinstein et al. (2010) proposed the factorization of learned dictionaries under another preﬁxed dictionary.
Separable ﬁlters were used in Rigamonti et al. (2013) to learn the coding of images.
Papyan et al. (2017) interpreted
CNN as an iterated convolutional sparse coding machine, and in this view, the factorized ﬁlters should correspond to a “dictionary of the dictionary” as in Rubinstein et al.
In the deep learning literature, low-rank factorization of convolutional ﬁlters has been previously used to remove redundancy in trained CNNs (Denton et al., 2014; Jaderberg et al., 2014).
The compression of deep networks has also been studied in Chen et al. ; Han et al. (2016; 2015), SqueezeNet (Iandola et al., 2016), etc., where the low-rank factorization of ﬁlters can be utilized.
MobileNets (Howard et al., 2017) used depth-wise separable convolutions to obtain signiﬁcant compression.
Tensor decomposition of convolutional layers was used in Lebedev et al. (2014) for CPU speedup.
Tai et al. (2015) proposed low-rank-regularized ﬁlters and obtained improved classiﬁcation accuracy with reduced computation.
Qiu et al.
(2018) studied decomposed-ﬁlter CNN with preﬁxed bases and trainable expansion coefﬁcients, showing that the truncated bases decomposition incurs almost no decrease in classiﬁcation accuracy while signiﬁcantly reducing the model size and improving the robustness of the deep features.
None of the above networks are group equivariant.
Group-equivariant deep networks.
The encoding of group information into network representations has been studied extensively.
Among earlier works, transforming auto-encoders (Hinton et al., 2011) used a non-convolutional network to learn group-invariant features and compared with handcrafted ones.
Rotation-invariant descriptors were studied in Schmidt & Roth (2012b) with product models, and in Jaderberg et al. (2015a); Kivinen & Williams (2011); Schmidt & Roth (2012a) by estimating the speciﬁc image transformation.
Gonzalez et al. ; Wu et al. (2015) proposed rotating conventional ﬁlters to perform rotation-invariant texture and image classiﬁcation.
The joint convolution across space and rotation has been studied in the scattering transform (Oyallon & Mallat, 2015; Sifre & Mallat, 2013).
Group-equivariant CNN was considered by Cohen & Welling (2016a), which handled several ﬁnite small-order discrete groups on the input image.
Rotation-equivariant
CNN was later developed in Weiler et al.
; Worrall et al.
; Zhou et al. (2017) and elsewhere.
In particular, steerable ﬁlters were used in Cohen & Welling (2016b); Weiler et al.
Worrall et al.
SO (3) -equivariant CNN for signals on spheres was studied in Cohen et al. (2018) in a different setting.
Overall, the efﬁciency of equivariant CNNs remains to be improved since the model is typically several times larger than that of a regular CNN.
The current paper adopts bases-decomposed ﬁlters previously studied in the non-equivariant setting (Qiu et al., 2018), however, the joint convolution scheme in equivariant CNNs is over the two geometries of space and orientation simultaneously and neither the approach nor the analysis there can be directly applied.
2. R OTATION - EQUIVARIANT CNN
A rotation-equivariant CNN indexes the channels by the SO (2) group (Weiler et al., 2017;
Zhou et al., 2017):
The l -th layer output is written as x ( l ) ( u, α, λ ) , the position u ∈ R 2 , the rotation α ∈ S 1 , and λ ∈ [ M l ] , M l being the number of unstructured channel indices.
Throughout the paper, [ m ] stands for the set { 1 , · · · , m } .
We denote the group SO (2) also by the circle S 1 since the former is parametrized by the rotation angle.
The convolutional ﬁlter at the l -th layer is represented as W ( l ) λ (cid:48) ,λ ( v, α ) , λ (cid:48) ∈
[ M l − 1 ] , λ ∈ [ M l ] , v ∈ R 2 , α ∈ S 1 , except for the 1st layer where there is no indexing of α .
In practice, S 1 is discretized into N θ points on (0 , 2 π ) .
We denote the summation over u and α by continuous integration, and the notation (cid:82) S 1 ( · · · ) dα means 12 π (cid:82) 2 π 0 ( · · · ) dα .
Let the 2D rotation by angle t be denoted by Θ t , in the 1st layer of the group-invariant CNN, x
[Equation]
Note that the 1st layer output has N θ orientations indexed by α , forming a “channel geometry” [Equation] Table 1).
For l > 1 , the convolution is jointly over R 2 and SO (2) , which takes the form as x ( l ) ( u, α, λ ) = σ
[Equation]
The joint convolution over R 2 and SO (2) is both sufﬁcient and necessary to guarantee groupequivariance (Theorem 3.1).
While group equivariance is a desirable property, the model size and computation can be increased signiﬁcantly due to the extra index α ∈ [ N θ ] .
2. D ECOMPOSED F ILTERS U NDER S TEERABLE B ASES
We decompose the ﬁlters with respect to u and α simultaneously: Let { ψ k } k be a set of bases on the unit 2D disk, and {
ϕ m } m be bases on S 1 .
At the l -th layer, let j l be the scale of the ﬁlter in u , and ψ j,k = 2 − 2 j ψ k (2 − j u ) (the ﬁlter is supported on the disk of radius 2 j l ).
Since we use continuous convolutions, the down-sampling by “pooling” is modeled by the rescaling of the ﬁlters in space.
The decomposed ﬁlters are of the form
[Equation]
which is illustrated in Figure 1 (for l > 1 ).
We use Fourier-Bessel (FB) bases for { ψ k } k and Fourier bases for { ϕ m } m .
Speciﬁcally, ϕ m ( α ) =
e imα , and FB basis has the expression
[Equation]
where m is the angular frequency, q the radial frequency, J m the Bessel function ( R m,q the q -th root of J m ) and c m,q a normalizing constant (Abramowitz & Stegun, 1964).
Both bases are “steerable”, i.e. the operation of rotation is a diagonalized linear transform under both bases.
In the complexvalued version, ψ k (Θ t v ) =
e − im ( k ) t ψ k ( v ) , ϕ m ( α − t ) =
e − imt ϕ m ( α ) .
This means that after the convolutions on R 2 × S 1 with the bases ψ k ( v ) ϕ
l ( α ) are computed for all k and l , both up to certain truncation, the joint convolution (1), (2) with all rotated ﬁlters can be calculated by the algebraic manipulation of the expansion coefﬁcients a ( l ) λ (cid:48) ,λ ( k, m ) , and without any re-computation of the spatial-rotation joint convolution.
Standard real-valued versions of the bases ψ k and ϕ m in sin ’s
and cos ’s are used in practice.
During training, only the expansion coefﬁcients a ’s are updated, and the bases are ﬁxed. [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] Apart from the saving of parameters and computation, the bases truncation also regularizes the convolutional ﬁlters by discarding the high frequency components.
As a result, DCF Net reduces response to those components in the input at all layers, which improves the robustness of the learned feature without affecting recognition performance.
The visibly smoother trained ﬁlters in RotDCF are shown in Figure A.1, demonstrating the same regularization effects as in Qiu et al. (2018), the latter being without the rotation-equivariant setting.
The theoretical properties of RotDCF Net, particularly the representation stability, will be analyzed in Section 3.
2. N UMBERS OF P ARAMETERS AND C OMPUTATION F LOPS
Number of trainable parameters :
In a regular CNN, a convolutional layer of size L × L × M (cid:48)
× M( L × L being the patch size) has L 2 M (cid:48)
M 0 parameters.
In an equivariant CNN, a joint convolutional ﬁlter is of size L × L × N θ × M (cid:48)
× M , so that the number of parameters is L 2 N θ M (cid:48) M .
In a RotDCF
Net, K bases are used in space and K α bases across the angle α , so that the number of parameters is
KK α M (cid:48) M .
This gives a reduction of KL 2 · K α N θ compared to non-bases equivariant CNN.
In practice, after switching from a regular CNN to a RotDCF Net, typically M ≤ 12 M 0 or more due to the adoption of ﬁlters in all orientations.
The factor KL 2 is usually between 18 and 13 depending on the network and the problem (Qiu et al., 2018).
In all the experiments in Section 4, K α is typically 5, and N θ = 8 or 16.
This means that RotDCF Net achieves a signiﬁcant parameter reduction from the non-bases equivariant CNN (the factor KK α L 2 N θ is 18 or more), and even reduces parameters from a regular CNN by a factor of 1 2 or more.
Computation in a forward pass : When the input and output are both W × W in space, the regular
CNN layer needs 2 M (cid:48)
M 0 W 2 L 2 many ﬂops, and a non-bases equivariant convolutional layer needs about 2 M (cid:48)
MW 2 L 2 N 2 θ .
In contrast, the computation in a RotDCF layer is dominated by a term of 2 M (cid:48)
MW 2 K α KN θ , which is reduced from the non-bases equivariant network by a factor of
KL 2 · K α N θ .
Detailed calculations in Appendix A.
In summary, RotDCF Net achieves a reduction of KL 2 · K α N θ from non-bases equivariant CNNs, in terms of both model size and computation.
With typical network architectures, RotDCF Net may be of a smaller model size than regular CNNs.
Numbers for speciﬁc networks are shown in Section 4.
T HEORETICAL A NALYSIS OF D EEP F EATURES
This section presents two analytical results: (1) Joint convolution (1), (2) is sufﬁcient and actually necessary to obtain rotation equivariance; (2) Stability of the equivariant representation with respect to input variations is proved under generic conditions, which is important in practice since rotations are never perfect.
3. G ROUP - EQUIVARIANT P ROPERTY
We consider the change of the l -th layer output when the input image undergoes some arbitrary rotation.
Let rotation around point u 0 by angle t be denoted by ρ
= ρ u 0 ,t , i.e. ρ u 0 ,t u = u 0 + Θ t ( u − u 0 ) , for any u ∈ R 2 , and the transformed image by D ρ x (0) ( u, λ ) =
x (0) ( ρ u 0 ,t u, λ ) , for any λ ∈ [ M 0 ] .
We also deﬁne the action T ρ on the l -th layer output x ( l ) , l > 0 , as
[Equation]
The following theorem, proved in Appendix B, shows that the joint convolution scheme (1), (2) not only produces group-equivariant features at all layers in the sense of
[Equation]
but is necessary for a CNN with SO (2) -indexed channels to achieve (6).
The sufﬁciency part is previously shown in Weiler et al.
The necessity of the joint convolution motivates the design and the efforts of reducing the complexity of such models.
Note that RotDCF is a type of the channel-indexed CNNs considered in the theorem, so it follows that RotDCF is group-equivariant.
Theorem 3.1.
In a CNN with SO (2) -indexed channels, let x ( l ) [ x (0) ] be the output at the l -th layer from input x (0) ( u, λ ) .
The relation (6) holds for all l if and only if the convolutional layers are given by (1) , (2) .
Assumptions on the RotDCF layers.
Following Qiu et al. (2018), we make the following generic assumptions on the convolutional layers: First,
(A1) Non-expansive sigmoid: [Equation] expansive [Equation]
Second, we also need a boundedness assumption on the convolutional ﬁlters W ( l ) for all l : (A2) Boundedness of ﬁlters:
In all layers, A l ≤ 1 , where A l is deﬁned as
[Equation]
µ k being the Dirichlet Laplacian eigenvalues of the unit disk in R 2 .
Note that (A2) bounds the expansion coefﬁcients, which implies a sequence of boundedness conditions on the convolutional ﬁlters in all layers (Proposition B.1), based upon which the stability results below are derived.
Since µ k typically increases in order, (A2) suggests truncating the series to only include low-frequency k and m ’s, which is implemented in Section 4.
The validity of the boundedness assumption can be qualitatively fulﬁlled by normalization layers which is standard in practice.
Non-expansiveness of the network mapping.
Let the L 2 norm of x ( l ) be deﬁned as
[Equation]
and (cid:107) x (0) (cid:107) 2 = 1 M 0 (cid:80) λ 1 | Ω | (cid:82) R 2 x (0) ( u, λ ) 2 du .
Ω is the domain on which x (0) is supported, usually Ω =
[ − 1 , 1] × [ − 1 , 1] ⊂ R 2 .
The following result is proved in Appendix B:
Proposition 3.2.
In a RotDCF Net, under (A1), (A2), for all l ,
(a) The mapping of the l -th convolutional layer (including σ ), denoted as x ( l ) [ x ( l − 1) ] , is nonexpansive,
i
e., (cid:107) x
( l ) [ x 1 ] − x ( l ) [ x 2 ] (cid:107) ≤ (cid:107) x 1 − x 2 (cid:107) for arbitrary x 1 and x 2 .
(b) (cid:107) x
( l ) c (cid:107) ≤ (cid:107) x
( l − 1) c (cid:107) for all l where x ( l ) c ( u,
α
λ
) = x ( l ) ( u,
α
λ
−
x (
l λ without index α when l =1) is the centered version of x ( l ) by removing x ( l ) 0 , deﬁned to be the output at the l -th layer from a zero bottom-layer input.
As a result, (cid:107) x l ) c (cid:107) ≤ (cid:107) x (
) c (cid:107) = (cid:107) x (0) (cid:107) .
Insensitivity to input deformation.
We consider the deformation of the input “module” to a global rotation.
Speciﬁcally, let the deformed input be of the form D ρ ◦ D τ x (0) , where D ρ is as in Section 3.1, ρ = ρ u 0 ,t being a rigid 2D rotation, and D τ is a small deformation in space deﬁned by
[Equation]
with τ : R 2 → R 2 is C 2 .
Following Qiu et al. (2018), we assume the small distortion condition:
[Equation]
with
[Equation]
being the operator norm.
The mapping u (cid:55)→ u − τ ( u ) is locally invertible, and the constant 15 is chosen for convenience.
With
T ρ as deﬁned in (5), the stability result is summarized as
Theorem 3.3.
Let ρ = ρ u 0 ,t be an arbitrary rotation in R 2 , around u 0 by angle t , and let D τ be a small deformation.
In a RotDCF Net, under (A1), (A2), (A3), c 1 = 4 , c 2 = 2 , for any L ,
[Equation]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] To prove Theorem 3.3, we ﬁrstly establish an approximate equivariant relation for all layers l (Proposition B.2), which can be of independent interest for estimating the image transformations.
All the proofs are left to Appendix.
Unlike previous stability results for regular CNNs, the above result allows an arbitrary global rotation ρ with respect to which the RotDCF representation is equivariant, apart from a small “residual” distortion τ whose inﬂuence can be bounded.
This is also an important result in practice, because most often in recognition tasks the image rotation is not a rigid in-plane one, but is induced by the rotation of the object in 3D space.
Thus the actual transformation of the image may be close to a 2D rotation but is not exact.
The above result guarantees that in such cases the RotDCF representation undergoes approximately an equivariant action of T ρ , which implies consistency of the learned deep features up to a rotation.
The improved stability of RotDCF
Net over regular CNNs in this situation is observed experimentally in Section 4.
E XPERIMENTAL R ESULTS
In this section, we experimentally test the performance of RotDCF Nets on object classiﬁcation and face recognition tasks.
The advantage of RotDCF Net is demonstrated via improved recognition accuracy and robustness to rotations of the object, not only with in-plain rotations but with 3D rotations as well.
To illustrate the rotation equivariance of the RotDCF deep features, we show that a trained auto-encoder with RotDCF encoder layers is able to reconstruct rotated digit images from “circulated” codes.
All codes will be publicly available.
4. O BJECT C LASSIFICATION
Non-transfer learning setting .
The rotMNIST dataset contains 28 × 28 grayscale images of digits from 0 to 9, randomly rotated by an angle uniformly distributed from 0 to 2 π (Cohen & Welling, 2016a).
We use 12,000 and 6,000 training samples, and 50,000 testing samples.
The baseline net is
[Table]
Notice that the reduction from non-bases rotation-equivariant CNNs (the fair comparison case) can be even smaller, which is the factor of KK
α L 2 N θ , c.f. Section 2.3. The case of maximum K and K α ( K =14, K α =8 in rotMNIST) is mathematically equivalent to using “full” ﬁlters without bases truncation. a CNN with 6 convolutional layers (Table A.2), and we also compare with DCF (Qiu et al., 2018), which is non-rotation-equivariant but uses bases truncation, and SFCNN (Weiler et al., 2017), which is rotation-equivariant but does not use bases decompostion.
The RotDCF net is made by replacing the regular convolutional layers with RotDCF layers, with N θ many rotation-indexed channels and a reduced number of (unstructured) channels M .
K many ψ k bases and K α many ϕ m are used.
The classiﬁcation accuracy is shown in Table 3.
We see that RotDCF net obtains improved classiﬁcation accuracy from CNN with signiﬁcantly reduced number of parameters, e.g., with 12K training, RotDCF Net with M =8, K =3, K α =5 improves the test accuracy from 96 .
54 to 98 .
40 with less thanmany parameters of the CNN model and 13 of the DCF model.
The performance is also comparable to SFCNN which has signiﬁcantly larger model complexity.
The results are similar with reduced training size (5K) and on a shallower net with 3 conv layers, c.f.
Table A.3.
The CIFAR10 dataset consists of 32 × 32 colored images from 10 object classes (Krizhevsky, 2009), and we use 10,000 training and 50,000 testing samples.
The network architecture is modiﬁed from VGG-16 net (Simonyan & Zisserman, 2014)
(Table A.4).
As shown in Table 3, RotDCF
Net obtains better testing accuracy with reduced model size from the regular CNN baseline model.
Throughout these experiments, the higher accuracy is due to the group equivariance and the lower model complexity is due to the bases decomposition.
Transfer learning setting.
We train a regular CNN and a RotDCF Net on 10,000 up-right MNIST data samples, and directly test on 50,000 randomly rotated MNIST samples where the maximum rotation angle MaxRot=30 or 60 degrees (the “no-retrain” case).
We also test after retraining the last two non-convolutional layers (the “ fc-retrain” case).
To visualize the importance of image regions which contribute to the classiﬁcation accuracy, we adopt Class Activation Maps (CAM)
(Zhou et al., 2016), and the network is modiﬁed accordingly by removing the last pooling layer in the net in Table
A.1.
and inserting a “gap” global averaging layer.
The test accuracy are listed in Table 2, where the superiority of RotDCF Net is clearly shown in both the “no-retrain” and “ fc-retrain” cases .
The improved robustness of RotDCF Net is furtherly revealed by the CAM maps (Figure 2): the redcolored region is more stable for RotDCF Net even in the case with retraining.
4. I MAGE R ECONSTRUCTION
To illustrate the explicit encoding of group actions in the RotDCF Net features, we train a convolutional auto-encoder on the rotated MNIST dataset, where encoder consists of stacked RotDCF layers, and the decoder consists of stacked transposed-convolutional layers (Table A.5).
The encoder maps a 28 × 28 image into an array of 16 × 32 , where the ﬁrst dimension is the discretization of the rotation angles in [0 , 2 π ] , and the second dimension is the unstructured channels.
Due to the rotation equivariant relation, the “circulation” of the rows of the code array should correspond to the rotation of the image.
This is veriﬁed in Figure 3: The top panel shows the code array produced [Figure] consistent face regions across the same-person images to determine the identity.
Across different in-plane rotated copies, RotDCF chooses signiﬁcantly more consistent discriminative regions than
CNN, indicating more stable representations.
In this experiment, we obtain 0.54% recognition accuracy using CNN (nearly random guess), and 97.04% accuracy using RotDCF with feature alignment, on known subjects.
[Figure] from a testing image, and the 16 row-circulated copies of it. The bottom panel shows the output of the decoder fed with the codes in the top panel.
4. F ACE R ECOGNITION
As a real-world example, we test RotDCF on the Facescrub dataset (Ng & Winkler, 2014) containing over 100,000 face images of 530 people.
A CNN and a RotDCF Net (Table A.6) are trained respectively using the gallery images from the 500 known subjects, which are preprocessed to be near-frontal and upright-only by aligning facial landmarks (Kazemi & Sullivan, 2014).
See Appendix C for data preparation and training details.
For the trained deep networks, we remove the last softmax layer, and then use the network outputs as deep features for faces, which is the typical way of using deep models for face veriﬁcation and recognition to support both seen and unseen subjects (Parkhi et al., 2015).
Using deep features generated by the trained networks, a probe image is then compared with the gallery faces whose identities are known and classiﬁed as that of the top match.
Under this gallery-probe face recognition setup, we obtain 94.10% and 96.92% accuracy for known and unknown subjects respectively using the CNN model; using RotDCF, the accuracies are 93.42% and 96.92%.
Testing on unknown subjects are critical for validating the model representation power over unseen identities, and the reason for higher accuracy is simply due to the smaller number of classes.
For both cases, RotDCF reports comparable performance as CNN, while the number of parameters in the RotDCF model is about one-fourth of the CNN model (see Appendix).
In-plane rotation.
This experiment demonstrates the rotation-equivariance of the RotDCF features.
We apply in-plane rotations at intervals of π 4 to the probe images (Figure 4), and let the original probe set be the new gallery, the rotated copies be the new probe set.
In this setting, using the RotDCF model we obtain 97.04% and 97.58% recognition accuracy for known and unknown subjects respectively, after aligning the deep features by circular shifts (using the largest-magnitude α channel as reference).
Notice that the model only sees upright faces.
This is due to the rotation-equivariant property of the RotDCF Net, which means that the face representation is consistent regardless of its orientations after the group alignment.
Lacking such properties, CNN obtains 0.54% and 5.05% recognition accuracies, which is close to random guess.
We further compare CNN and RotDCF models via the CAM maps, which indicate the image regions relied by the CNN to make a classiﬁcation prediction.
In Figure 4, with regular CNN face regions used for prediction varies dramatically from image to image, and with RotDCF a signiﬁcantly more consistent region mostly covering eyes and nose are selected across images, which explains its superior accuracy.
Out-of-plane rotation.
To validate our theoretical result on representation stability under input deformations, we introduce out-of-plane rotations to the probe.
Each probe image is ﬁtted to a 3D face mesh, and rotated copies are rendered at the 10 o intervals with -40 o to 40 o yaw, and -20 o to 20 o pitch, generating 45 synthesized faces in total (Figure 5).
The synthesis faces at two poses (highlighted in red) are used as the new gallery, and all remaining synthesis faces form the new probe.
The out-of-plane rotations here can be viewed as mild in-plane rotations plus additional variations, a situation frequently encountered in the real world.
With this gallery-probe setup, the
RotDCF model obtains 89.66% and 97.01% recognition accuracy for known and unknown subjects, and the accuracies are 80.79% and 89.97% with CNN.
The CAM plots in Figure 6 also indicate that RotDCF Net chooses more consistent regions over CNN in describing a subject across different poses.
Since the out-of-plane rotations as in Figure 5 can be considered as in-plane rotations with additional variations, the superior performance of RotDCF is consistent with the theory in Section
C ONCLUSION AND D ISCUSSION
This work introduces a decomposition of the ﬁlters in rotation-equivariant CNNs under joint steerable bases over space and rotations simultaneously, obtaining equivariant deep representations with signiﬁcantly reduced model size and an implicit ﬁlter regularization.
The group equivariant property and representation stability are proved theoretically.
In experiments, RotDCF demonstrates improved recognition accuracy and better feature interpretability and stability on synthetic and realworld datasets involving object rotations, particularly in the transfer learning setting.
To extend the work, implementation issues like parallelism efﬁciency and memory usage should be considered before the computational savings can be fully achieved.
The framework should also extend to other groups and joint geometrical domains
A CKNOWLEDGEMENTS
Work partially supported by NSF, DoD, NIH and AFOSR.
A F LOPS C ALCULATION IN S ECTION 2 AND C HOICE OF H YPERPARAMETERS
When the input and output are both W × W in space, the forward pass in a regular convolutional layer needs M (cid:48)
M 0 W 2 (1 + 2 L 2 ) ∼ 2 L 2 M (cid:48)
M 0 W 2 ﬂops.
(Each convolution with a L × L ﬁlter takes 2 L 2 W 2 , and there are M (cid:48)
M 0 convolution operations, plus that the summation over λ (cid:48) takes
W 2 M (cid:48)
M 0 ﬂops.)
In a rotation equivariant CNN without using bases, an convolutional layer would take ∼ 2 M (cid:48) MW 2 L 2 N 2 θ ﬂops.
In a RotDCF layer, the computation consists of three parts: (1) The inner-product with ϕ m bases takes W 2 M (cid:48) · 2 N θ K α ﬂops.
The spatial convolution with the ψ k bases takes K α M (cid:48) K · 2 L 2 Wﬂops.
The multiplication with a λ (cid:48) ,λ ( k, m ) e − im ( k ) α − imα and summation over λ (cid:48) , k, m takes
MN θ (4 KK α M (cid:48)
+ 2 W 2 KK α M (cid:48) )
ﬂops (real-valued version).
Putting together, the total is 2 M (cid:48)
W 2 K α ( N θ + L 2 K + MN θ K ) , and when M is large, the third term dominates and it gives 2 M (cid:48)
MW 2 K α KN θ .
Thus the reduction by using bases-decomposed ﬁlters is again a factor of
KL 2 · K α N θ , and the relative ratio with a regular CNN is about M (cid:48)
M M
[Equation]
The choice of hyperparameters K and K α can be interpreted as the “frequency” truncation both in R 2 and in S 1 , and it trades-off between ﬁlter regularization and ﬁlter expressiveness: The more severe the truncation, the less many trainable parameters, the smoother the ﬁlter (Figure A.1) and less overﬁtting, while the potential risk of underﬁtting is larger.
This is revealed in Table 3: as K and
K α are reduced, the accuracy decreases monotonically in the rotMNIST experiment.
When K α is reduced to 3 then underﬁtting becomes noticeable and accuracy drops even more (not reported).
The parameter N θ corresponds to spatial discretization along S 1 and, since we adopt a spectral approach, it does not affect the number of trainable parameters, which is determined by K α .
In other words, the choice of N θ is only constrained by memory storage.
B P ROOFS IN S ECTIONB. T HEOREM 3.
Proof of Theorem 3.1.
Note: The bases expansion under ψ j l ,k and ϕ m does not affect the form of convolutional layers, but only impose regularity of the ﬁlters, thus the group-equivariant property of
RotDCF follows from the theorem.
Observe that the equivariant relation (6) is equivalent to that
[Equation]
for all l , where T ρ x (0) means D ρ x (0) .
The sufﬁciency part: When l = 1 , by (1),
[Equation]
Since ρ u 0 ,t ( u + v ) = ρ u 0 ,t +Θ t v , we have that T ρ x (1)
[ x (0) ]
= x (1) [ D ρ x (0) ] by a change of variable of Θ t v (cid:55)→ v .
When l > 1 , by (2),
[Equation]
Again, inserting ρ u 0 ,t ( u + v ) =
ρ u 0 ,t + Θ t v , the claim follows by changing variables Θ t v (cid:55)→ v and α (cid:48) − t (cid:55)→ α
[Equation]
The necessity part: When l = 1 , denote the general convolutional ﬁlter as w (1) ( v ; λ (cid:48) , λ, α ) , and x (1)
[Equation]
u
[Equation]
=
[Equation]
Recall that
[Equation]
and then (A.1) with l = 1 holding for any x (0) implies that (cid:88)
[Equation]
By that ρ u 0 ,t ( u + v ) = ρ u 0 ,t + Θ t v , the above equality gives that
[Equation]
Let t = α , and F λ (cid:48) ,λ ( v ) =
w (1) ( v ; λ (cid:48) , λ, 0) , we have that
[Equation]
and this gives that w (1) ( v ; λ (cid:48) , λ, α ) =
F λ (cid:48) ,λ (Θ α v ) .
This proves that (1) is necessary.
When l > 1 , consider the general convolutional ﬁlter as w ( l ) ( v ; λ (cid:48) , λ, α (cid:48) , α ) .
Using a similar argument, (A.1) implies that
[Equation]
Let t = α , and F λ (cid:48) ,λ ( v, α (cid:48) )
=
w ( l ) ( v ; λ (cid:48) , λ, α (cid:48) , 0) , then
[Equation]
which gives that w ( l ) ( v ; λ (cid:48) , λ, α (cid:48) ,
[Equation]
α ) , which proves that (2) is necessary.
B. P ROPOSITION 3.
Proposition B.1.
For all l ,
[Equation]
where
[Equation]
As a result
[Equation]
where
[Equation]
and thus (A2) implies that B l , C l , 2 j l D l ≤ 1 for all l .
Proof of Proposition B.1.
The proof for the case of l = 1 is the same as Lemma 3.5 and Proposition 3.6 of Qiu et al.
We reproduce it for completeness.
When l = 1 , it sufﬁces to show that for
F ( v ) =
(cid:80)
k
a k ψ k ( v ) ,
[Equation]
Rescaling to ψ j l ,k in v leads to the desired inequality with the factor of 2 j l for D ( l ) λ (cid:48) ,λ .
To prove (A.4), observe that F is supported on the unit disk, and then (cid:107) F (cid:107) 1 , (cid:82) | v ||∇ F ( v ) | dv ≤ (cid:107)∇ F (cid:107) 1 ≤ √ π (cid:107)∇ F (cid:107) 2 , where (cid:107)∇ F (cid:107)
= π (cid:80) k
µ k a 2 k due to the orthogonality of ψ k .
For l > 1 , similarly, we only consider the rescaled ﬁlters supported on the unit disk in v .
Let
F ( v, β ) =
(cid:80) k,m a k,m ψ k ( v ) ϕ m ( β ) , β ∈ S 1 , similarly as above, we have that
[Equation]
recalling that (cid:82) dβ on S 1 has the normalization of 12 π .
Again, (cid:82) (cid:82) |∇ v F ( v, β [Equation] | 2 dvdβ =
π (cid:80) k,m µ k a 2 k,m due to the orthogonality of ψ k and ϕ m .
This proves that
[Equation]
which leads to the claim after a rescaling of v .
Remark 1 (Remark to Proposition 3.2) .
The proposition only needs B l , deﬁned in (A.3) , to be less than 1 for all l , in a rotation-equivariant CNN, which is implied by (A2) by Proposition B.1.
Proof of Proposition 3.2.
The proof is similar to that of Proposition 3.1(a) of Qiu et al.
Speciﬁcally, in (a), the argument is the same for l = 1 , making use of the fact that
[Equation]
and (cid:82) S 1 dα = 1 due to the normalization of 12 π .
For l > 1 , the same technique proceeds with the new deﬁnition of B ( l ) λ (cid:48) ,λ as in (A.2) which involves the integration of (cid:82) S 1 ( · · · ) dβ .
The detail is omitted.
To prove (b), we ﬁrstly verify that x ( l ) 0 only depends on λ .
When l = 1 , x (1)0 ( u, α, λ ) =
σ ( b (1) ( λ )) .
Suppose that it holds for ( l − 1) , consider l > 1 ,
[Equation]
Thus x ( l ) 0 ( u, α, λ ) = x ( l ) 0 ( λ ) for all l (without index α for l = 1 ).
The rest of the argument follows from that (cid:107) x ( l ) c (cid:107) = (cid:107) x
( l ) − x ( l ) 0 (cid:107) = (cid:107) x
( l ) [ x ( l − 1) ]
− x ( l ) [ x ( l − 1)] (cid:107) ≤ (cid:107) x
[Equation]
l
[Equation]
) − x ( l − 1)(cid:107) = (cid:107)
[Equation]
where the inequality is by (a).
B. T HEOREM 3.
Proposition B.2.
In a RotDCF Net, under (A1), (A2), (A3), c 1 = 4 , for any l ,
[Equation]
where D τ only acts on the space variable u of x ( l ) similar to (9) .
Proof of Proposition B.2.
We ﬁrstly establish that for all l ,
[Equation]
where T ρ is replaced by D ρ if applies to x (0) which does not have index α .
This is because that
[Equation]
by Theorem 3.1, and that
[Equation]
by the deﬁnition of T ρ (a rigid rotation in u , and a translation in α ).
This term can be upper bounded by c 1 ( B l + C l ) |∇ τ | ∞ (cid:107) x
( l − 1) c (cid:107) (Lemma B.3), which leads to the desired bound under (A2) by
Proposition B.1.
The rest of the proof is similar to that of Proposition 3.3 of Qiu et al.
Write x ( l ) [ D ρ ◦
D τ x (0) ] − T ρ ◦ D τ x ( l ) [ x (0) ] as the sum of the differences x ( l ) [ x ( j ) [ D ρ ◦ D τ x ( j − 1) ]] − x ( l ) [ T ρ ◦
D τ x ( j ) [ x ( j − 1) ]] for j = 1 , · · · , l .
The norm of the j -th term is bounded by (cid:107) x ( j )
[ D ρ ◦ D τ x ( j − 1) ] −
T ρ ◦ D τ x ( j ) [ x ( j − 1) ] (cid:107) due to Proposition 3.2 (a), which, by applying (A.5) together with Proposition 3.2 (b), can be bounded by 2 c 1 |∇ τ | ∞ (cid:107) x (0) (cid:107) .
Summing over j gives the claim.
Proof of Theorem 3.3.
The proof is similar to that of Theorem 3.8 of Qiu et al.
With the bound in Proposition B.2, it sufﬁces to show that
[Equation]
By the deﬁnition of T ρ , the l.h.s.
equals (cid:107) D τ x ( L )
[ x (0) ] − x ( L )
[ x (0) ] (cid:107) , which can be shown to be less than c 2 | τ | ∞ D L (cid:107) x ( l − 1) c
(cid:107) by extending the proof of Proposition 3.4 of Qiu et al. (2018), similar to the argument in proving Lemma B.3.
The desired bound then follows by that D L ≤ 2 − j L A L ≤ 2 − j L (Proposition B.1 and (A2)) and that (cid:107) x ( l − 1) c (cid:107) [Equation] (cid:107) x (0) (cid:107) (Proposition 3.2 (b)).
Lemma B.3.
In a rotation-equivariant CNN, B l , C l deﬁned as in (A.3) , under (A1), (A3), for all l > 0 , with c 1 = 4 , x ( l ) c as in Proposition 3.2,
[Equation]
[Table] [Table]
Proof of Lemma B.3.
The proof is similar to that of Lemma 3.2 of Qiu et al.
Specifically, when l = 1 , the argument is the same, making use of the fact that (cid:82) | w (Θ α v ) | dv = (cid:82) | w ( v ) | dv, ∀ α ∈ S 1 and (cid:82) S 1 dα
= 1 due to the normalization of 12 π .
When l > 1 , the same technique applies by considering the joint integration of (cid:82) R 2 (cid:82) S 1 ( · · · ) dvdβ instead of just dv .
The only difference is in using the new deﬁnitions of B ( l ) λ (cid:48) ,λ and C ( l ) λ (cid:48) ,λ for l > 1 as in (A.2), both of which involve the integration of (cid:82) S 1 ( · · · ) dβ .
The detail is omitted.
C E XPERIMENTAL D ETAILS IN S ECTIONC. O BJECT RECOGNITION WITH ROT MNIST AND CIFAR
In the experiments on rotMNIST dataset, the network architecture for Conv-3 and Conv-6 nets are shown in Table A.1 and Table A.2.
Stochastic gradient descent (SGD) with momentum is used to train 100 epochs with decreasing learning rate from 10 − 2 to 10 − 4 .
In the experiments on CIFAR10 dataset, the VGG16-like network architecture is shown in Table
A.4.
SGD with momentum is used to train 100 epochs with decreasing learning rate from 10 − 2 to 10 − 4 .
[Table] [Table]
[Table]
C. C ONVOLUTIONAL A UTO - ENCODER FOR IMAGE RECONSTRUCTION
The network architecture is shown in Table A.5.
The network is trained on 50,000 training samples, the training set is augmented by rotating each sample at 8 random angles, producing 400k training set.
The network is trained for 10 epochs, where the learning rate decreases from 10 − 3 to 10 − 6 .
C. F ACE RECOGNITION ON F ACESCRUB
To facilitate the evaluation on both known and unknown subjects, we select the ﬁrst 500 of theidentities as our training subjects.
The remaining 30 subjects are used for validating out of sample performance, namely the unknown subjects.
The experiment on unknown subjects is critical for face models to generate over unseen people.
For both known and unknown subjects, we hold 10 images from each person as the probe images, and the remaining as the gallery images.
The images are preprocessed by aligning facial landmarks using Kazemi & Sullivan (2014).
and crop the aligned face images to 112 × 112 with color.
Thus, both our CNN and RotDCF models are trained with near-frontal and upright-only face images.
The network architecture is shown in Table A.6.
According to the formula in Section 2, the number of trainable parameters in the RotDCF Net is about ( 12 ) 2 · KL [Equation] K [Equation]
= 14 of that of the CNN.
C. E FFECTS OF R EGULARIZATION BY F ILTER D ECOMPOSITION
Sample trained ﬁlters in the reconstruction experiment are shown in Figure A.1, which shows that
RotDCF obtains smoother trained ﬁlters as a result of the regularization applied by the bases truncation.
In the reconstruction experiment, increasing the number of bases degrades the synthesis (Right
[Figure]
[Table]
in Figure A.1).
Such improved stability due to the truncated decomposition supports the analysis in
Section 3.