E MPOWERING N ETWORKS W ITH S CALE AND R OTATION E QUIVARIANCE U SING A S IMILARITY C ONVOLUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
The translational equivariant nature of CNN is a reason for its great success in the ﬁeld of computer vision.
However, networks do not enjoy more general equivariance properties such as rotation or scaling.
This limits the generalization performance of the network.
In this paper, we devise a method that provides networks with equivariance with respect to translation, rotation, and scaling simultaneously.
We deﬁne a convolution-like operation and ensure equivariance based on our proposed scalable Fourier-Argand representation.
The method has similar efﬁciency as a traditional network and hardly introduces any additional learnable parameters, since it does not face the computational issue often occurs in group-convolution operator.
We veriﬁed the quality of our approach in the image classiﬁcation task, demonstrating the robustness and the generalization ability to both scaled and rotated inputs.
I NTRODUCTION
The great success of network architectures can be attributed to large datasets and large number of parameters, making it possible to “remember” more information.
On the contrary, humans can learn new concepts with very little data, and are able to generalize this knowledge.
Due to a lack of modelization of geometric deformations, networks prefer to “remember” all the data through ﬁlter parameters instead of “learning” a full generalization ability.
For example, in the classiﬁcation task, networks trained with datasets with speciﬁc object size will make the test fails when using the same object but with a size that does not appear in the training set.
The ability to factor out transformations, such as rotation or scaling, in the learning process remains to be addressed.
It is indeed quite frequent to deal with images in which objects have a different orientation and scale than in the training set, for instance, as a result of distance and orientation change of the camera.
To mitigate this issue, data augmentation before training (Krizhevsky et al., 2012) is quite common.
However, this leads to a substantially larger dataset and makes training more complicated.
Moreover, this strategy tends to learn a group of duplicates of almost the same ﬁlters, which usually requires more learnable parameters to achieve competitive performance.
A visualization of the weights of the ﬁrst layer (Zeiler & Fergus, 2014) points out that many ﬁlters are similar but rotated and scaled versions of a similar prototype, which results in signiﬁcantly more redundancy.
The concept of equivariance was brought to solve this issue, which can be roughly described as follows:
If the input undergoes a particular geometric transformation, the output feature from the network (with randomly initialized weights) should exhibit a similarly predictable geometric transformation.
Should a network satisfy equivariance to scalings and rotations, training it with only one size and orientation would naturally generalize its performance to all sizes and orientations.
To achieve this property, methods that use group convolution have dominated this ﬁeld.
An oversimpliﬁed interpretation of a typical group convolution method is as follow: Features are convolved with dilated ﬁlters of the same template to obtain multi-channel features.
And the distortion of the input features then corresponds to the cycle shift between channels.
For example, equivariant CNNs on truncated directions (Cohen & Welling, 2016; Zhou et al., 2017) use several directional ﬁlters to obtain equivariance within a discrete group.
Further works generalized the rotation equivariance to continuous group, with the steerable ﬁlters (Weiler et al., 2018; Cohen et al., 2019), B-spline interpolation (Bekkers, 2020) or Lie Group Theory (Bekkers, 2020; Finzi et al., 2020).
A similar path to scaling equivariance is explored, though scaling is no longer intrinsically periodic.
Deep scale space (Worrall & Welling, 2019) deﬁned a semi-symmetry group to roughly obtain the scale equivariance, while Sosnovik et al.
(2020) applied steerable CNNs on scaling.
However, integrating equivariance to rotations and scalings at the same time results in a bigger group (e.g., a rotation group with M points and a scaling group with N points results in M × N points for the joint rotation and scaling group), making the task more difﬁcult.
Furthermore, certain ”weight-sharing” techniques based on group convolution are computationally and memory storage intensive.
SREN
SREN
Therefore, for the training of an object with a certain shape, our method can handle all the distorted(rotate, scaling, and translation) objects without further training.
Despite the difﬁculties, empowering the model with equivariance of rotation and scaling together can be very beneﬁcial.
For example, in object detection, the distance changes between the camera and the object, or the random rotations of the object, can largely inﬂuence the method’s accuracy.
In this paper, we aim to propose a CNNs that is continuously equivariant with rotation and scaling.
Such work will ﬁll a void in the area of equivariant.
To achieve this, we provide a theory and analysis to guarantee that the network preserves the inherent equivariance property.
Based on this, we propose a Scale and Rotation Equivariant Network (SREN) architecture.
The method avoids the abovementioned limitations and does not sizably increase computational complexity.
Speciﬁcally, We ﬁrst designed a scalable Fourier-Argand representation.
The expression of the basis makes it possible to operate the angle and scale in one shot.
Based on this, we propose a new convolution-like operator that is slightly different while functionally similar to the traditional convolution.
Furthermore, we show that the computational complexity is similar to convolution, so this method can easily replace the typical network structure.
Finally, the method we designed can model rotation and scale scaling.
Our model can achieve consistent results when tested with datasets with different transformations (such as rotation and scaling).
The main contributions in this paper are summarized as follows:
• We propose the scalable Fourier-Argand representation.
This representation allows for achieving the similarity equivariance property.
• We propose the SimConv operator, together with the scalable Fourier-Argand ﬁlter, formed as the Scale Rotation Equivariant Network (SREN) architecture, which is an equivariant network for rotation and scaling.
• The method is very different from the group-convolutional neural networks.
This provides a new possible path to solve the problem for the community and is not limited to the idea of group convolution.
R ELATED W ORK
Group convolution Unlike the previous methods, which can achieve equivariance only in one aspect, our goal is to ensure rotation and scaling equivariant in a unique network.
A possible direction is the application of group theory to achieve equivariance.
Cohen & Welling (2016) introduced group convolution and enforced equivariance to small and discrete groups of transformation, i.e., rotations by multiples of 90 degrees.
Subsequent works aim at generalizing the equivariantZhou et al.
(2017) and also focus on continuous group coupled with the idea of steerable ﬁlters(Cohen & Welling, 2017).
To achieve this purpose, Lie group theory, such as LieConv citepﬁnzi2020generalizing, is also presented.
These studies allow for precise equivariance, but only for compact groups.
In contrast to the rotating group, the scaling group is non-compact.
And methods usually treat it as a semi-group and approximately achieve the truncated scaling equivariance.
TridentNet (Li et al., 2019) gets scale invariance by sharing weights among kernels with different dilation rates.
Scale-space theory (Lindeberg, 2013) is also bring out by Worrall & Welling (2019) consider moving band-limits caused by rescaling in achieving semi-group equivariance.
Bekkers (2020) generate the G-CNNs for arbitrary Lie groups by B-spline basis functions.
With different settings, it can achieve scale or rotation equivariance.
SiamSE(Sosnovik et al., 2021c) equip the Siamese network with additional built-in scale equivariance. Although Sosnovik et al.
(2021b;a) replacing weight sharing scheme with dilated ﬁlters can parallel the process, and thus O(1) in terms of time, the overall computational load is still increased concerning the group size.
Beyond all these methods, expanding the group to a larger space can increase the overhead computation.
Steerable ﬁlters Works with steerable ﬁlters are also used to achieve equivariance.
The underlying insight is to use different linear combinations of ﬁxed basis to represent ﬁlters of different sizes or rotation angles.
H-Net
(Worrall et al., 2017) uses complex circular harmonics while SESN (Sosnovik et al., 2020) uses the Hermite polynomials as ﬁlter bases to achieve equivariance.
This is a bit related to our method.
However, due to the lack of a proper existing basis in the image processing ﬁeld, this path to achieving rotation-scaling equivariant is also difﬁcult.
Other works improved the equivariant on different aspects.
Polar transformer networks (Esteves et al., 2018) generalizes groupequivariance to rotation and dilation.
Attentive group convolutions (Romero et al., 2020)
Use the attention mechanism to generalize the group convolution.
Some other techniques, for example, Shen et al.
(2020) propose using the partial differential operator to maintain the equivariance.
Jenner &
Weiler (2022) use the partial differential operators (PDOs) to design the equivariant CNNs effectively.
Gao et al.
(2022) presents a roto-scale-translation equivariant CNN, but it expands ﬁlters of the
G-CNN in the scale dimension with a truncated interval.
Differences between related works and ours.
Unlike the previous methods our goal is to ensure continuous rotation and scaling equivariant in a unique network.
However, methods that are based on modifying the ﬁlters used in the neural network to achieve some form of equivariance, require more parameters to learn and are computationally more expensive.
Instead, we propose using a scalable and steerable ﬁlter representation (scalable Fourier-Argand) to modify the convolution operator (SimConv) so that it embodies scale, rotation, and shift equivariance.
This transformation does not introduce new learnable parameters.
Thus, we can then achieve the rotation and scale equivariance efﬁciently that none of the other methods enjoy.
P RELIMINARIES AND NOTATION
This section clariﬁes the notations about the sim(2) transformation and the convolution that is often mentioned later.
The property of equivariant and invariant are also formulated explicitly, which is derived from our proposed method.
3. S IM (2) TRANSFORMATION
In Euclidean geometry, two objects can be transformed into each other by the similarity transformation if they share the same shape.
This similarity is critical in instance-level computer vision, as objects do not change with scale, translation, and rotation transformations.
This motivates us to study the similarity equivariant.
Let’s consider the Sim(2) group that the invertible translation matrix is as follows,
[Equation]
where R = (cid:20) cos θ sin θ − sin θ cos θ (cid:21) ∈ R 2 × 2 denotes the rotation matrix, t ∈ R 2 is a translation vector and s ∈ R is the scale factor.
In this way, rotation (by θ angles) , scaling (by s times) and translation (by t pixels) are integrated in one matrix and can written as the matrix multiplication of three shape-preserving matrix, i.e., T =
A s Y t R θ .
Now let’s consider a spatial index on 2D image plane ˜ x
= [ x 1 , x 2 ] (cid:62) ∈ R 2 , we extend its domain to triples as x =
[˜ x, 1] (cid:62) ∈ R 3 .
Then for an input signal f , we deﬁne a linear transformation L T : L 2 ( X ) →
L 2 ( X ) that transform feature maps f ∈ L 2 ( X ) on some space X , written as,
[Equation]
The left-hand side can be regarded as a linear transformation L T acting on a set of feature maps f , and the right-hand side can be interpreted as ﬁnding the value of the feature map f at the point
T − 1 x .
Additionally, we have L T [ · ] =
L ( A s Y t R θ )
[ · ] =
L A s [ L Y t [ L R θ [ · ]]] .
When considering the transformation we have,
[Equation]
This corresponds to a rotation transformation followed by a translation and scaling.
3. F ORMULATION OF C ONVOLUTION
Let’s consider a stack of two-dimensional features as a function f ( · ) = { f c ( · ) } Nc =1 : R 2 × 1 → R N , where N is the number of the channel.
Similarly, a ﬁlter bank contains M elements in convolutional layer can be formalized as ϕ
= { ψ k } Mk =1 , where each ﬁlter with N channels can be written as ψ k = { ψ k,c } Nc =1 , and ψ k : R 2 × 1 → R N is a vector valued output ﬁlter.
Then, for a continuous input with N channels, we regard the spatial cross-correlation between the input and the continuous ﬁlter bank Ψ as an operator Φ[ · ] =
[ · (cid:63) ϕ ] : R N → R M , written as follow,
[Equation]
Without loss of generality, we can set N = M = 1 and simplify the above formula with the following formula,
[Equation]
To clarify, although t is three dimension vector, this integral is still a double integral (along the ﬁrst and second dimensions), the same as the regular convolution. Since the last dimension of t is only a placeholder.
We use this simpliﬁed formula in the remaining paper to make the deduction more readable.
3. E QUIVARIANCE AND I NVARIANCE P ROPERTY
Deﬁnition 3.1 (Equivariance) .
Given an operator Φ : L 2 ( N ) → L 2 ( M ) , the operator Φ is equivariant to the transform L T if for any x ∈ R d +1 , we can ﬁnd a predictable transform ˜ L T , such that the equation below holds.
[Equation]
If L T = ˜
L T , we can also say that these two operators are commutable.
This equivariance property provides structure-preserving properties for the network.
Deﬁnition 3.2 (Invariance) .
Given an operator Φ : L 2 ( N )
→ L 2 ( M ) , the operator Φ is invariant to the transform L T if for any x ∈ R d +1 , the equation below holds.
[Equation]
This paper aims to design a convolution-like operation that is similar in function to convolution without lifting the intermediate variable size of the network (This is common in group methods.), as well as satisﬁes the equivariance or invariance property for any similar transform.
M ETHOD
This section discussed the possibility of making the traditional convolutional neural network to be equivariance in rotation, scale, and translation simultaneously.
The underlying intuition behind the method is: we ﬁrst obtain the local scale and orientation of the image.
Then, this local information is used to adapt the scale and direction of the ﬁlter used for the convolution.
Moreover, this (imagedependent) spatially-varying convolution has an efﬁcient implementation.
4. S CALABLE F OURIER -A RGAND R EPRESENTATION
We ﬁrst proposed a representation that we called the scalable Fourier-Argand Representation.
The insight of designing this exact and steerable representation is that we can use this representation to retrieve the local geometric information and use it as a covariance indicator in Equation (12) for further usage.
We ﬁrst deﬁne scalable Fourier-Argand Representation as follows,
Deﬁnition 4.1 (The scalable Fourier-Argand representation) .
Consider a square-integrable function h , and express it as h ( r, θ ) in polar coordinates.
We propose its scalable Fourier Argand representation as the following series form,
[Equation]
Where Z denotes the set of all integers.
One has ﬂexibility in choosing any m k as a constant value.
The function has limited support that ( r, θ ) ∈ [0 , 2 π [ × [ a, b [ .
In reality, k 1 , k 2 can be truncated for a subset of integers and approximate h .
Proposition 4.1.
Let h k 1 ,k 2 : Z 2 → C as the coefﬁcient of each item, and we can compute it as follows,
[Equation]
The proof of this can be found in Appendix A.
This formula can represent the ﬁlter as a composition of elementary feature basis.
The part about θ can be seen as the Fourier series of ﬁlters on the Argand plane, which is inspired by Zhao & Blu (2020).
The ﬁlter itself is in relation to have a connection with the harmonic ﬁlterWorrall et al.
For the scaling part, We use the logarithmic method to make the scaling of the size a linear shift.
We also restrict the support of the function as a plane that radius ranges in [ a, b ] , where a → 0 + .
In practice, we set m k = − 1 that acts like a window function, which ensures that the function can vanish at inﬁnity.
The above explains why we call it the scalable Fourier-Argand representation.
One beneﬁt of the expression we propose is that its basis functions are steerable for rotation and scalable for scaling.
Speciﬁcally, consider the transformation matrix T
=
A s Y 0 R α that contains only rotation and scaling, we have,
[Equation]
This equation holds for any α ∈ [0 , 2 π ) and s ∈ ( a, b ) .
We can easily verify the following proposition,
Proposition 4.2.
For a continuous ﬁlter h ∈ R 2 that can be decomposed by a set of basis, let
T =
A s Y
R α as a transformation matrix.
Then the transformed ﬁlter of h , noted as L T [ h ] , can still be represented by the same basis, with a steerable linear combination:
[Equation]
With this property, we can ﬁx the basis H k 1 ,k 2 and estimate the ﬁlter h for rotation and scaling by using different linear combinations.
Furthermore, if we want to convolve the input signal f with the ﬁlter h and its various L T [ h ] , we can pre-convolve the image with the basis of the original ﬁlter H k 1 ,k 2 .
Besides, the normalized cross-correlation is a more robust substitution of traditional convolution, denoted as (cid:63) .
This allows us to obtain the intermediate variable f k 1 ,k 2 by the following formula,
[Equation]
Here ∗ is the traditional convolution, µ , and σ are the mean and variance of the basis ﬁlters or signal.
With these signal basis, we can then obtain the optimal orientation and scale by calculating the argmax of the combination of basis, written as follows,
[Equation]
Where c k 1 ,k 2 ( λ, γ ) =
exp( − ik 1 γ − ik 2
π ln λ ln b/a ) λ − m is a coefﬁcient that only relys on λ and γ .
c γ,λ and F are vectors of all possible f k 1 ,k 2 and c k 1 ,k 2 Λ f ( x ) , Γ f ( x ) can be understood as the projection of signal f for the orientation and scale aspect.
These two indicators meet the following properties,
Lemma 4.1.
Let T =
A s Y t R α as a similarity transformation.
Then for a input image f
and its distorted version L T [ f ]( x ) , for any position x , we can have a relationship of this pair of images by following property,
[Equation]
The proof can be found in Appendix B.
This is the condition that we achieved and applied later in
Section 4.2.
4. S IMILARITY C ONVOLUTION
We propose the similarity convolution (SimConv) as an alternative to traditional convolution.
We ﬁrst list the feature of SimConv that we expect as follows: 1).
Theoretically, it should have equivariance properties of rotation, scale, and translation. 2).
It should be a convolution-like operation, containing some learnable parameters and extracting image features by ”blending” one function with another.
Its computational complexity should be close to traditional convolution.
Therefore, unlike group convolution, it does not face the problem of computational disaster when extending the group size.
These criteria drive us design the similarity convolution as follows,
Deﬁnition 4.2 (Similarity convolution) .
The similarity convolution between the input signal f and the ﬁlter ϕ is deﬁned as
[Equation]
15) where M f ( x ) is deﬁned as following pixel-wise matrix,
[Equation]
This SimConv has a convolution-like structure between feature f and the learnable ﬁlter ϕ .
This is clear when we do variable substitude with t =
M − 1 f ( x ) ˜t .
And it can degenerate to traditional convolution when setting M f ( x ) to Identity matrix for all x .
We note this SimConv as an operator Φ[ · ] = [ · (cid:126) ϕ ] : R N → R M , where we set M = N = 1 for easier read.
This newly deﬁned convolution is, in fact, equivariant for rotation, scaling, and translation.
We verify this property below.
Considering M f ( x ) in Equation (16) and combined with the condition of Equation (14), we can easily prove that
[Equation]
This leads to the following summarized condition: T −
[Equation]
If we apply a similarity transformation L T that rotates the signal by α degree centered at t , as well as scaling by s , to the signal f following Equation (2) [Equation] then for signals after convolution
[Equation]
we have
[Equation]
Besides
[Equation]
the commutator operator of the above can similarly deduct as,
[Equation]
Replace the second T in Equation (19) [Equation] and compare with Equation (18), we can conclude that
[Equation]
This shows that for the input signal f , if we apply the similarity transform and then apply the SimConv, it is equivalent to ﬁrst applying the SimConv and then the transform.
So with scalable Fourier Argand representation, the proposed Similarity Convolution satisﬁes the equivalence property.
Moreover, if each layer in the network satisﬁes this property, then the transformation can be passed from the ﬁrst layer to the last layer, which can be expressed as the following formula,
[Equation]
Moreover, there are few choices to convert the equivariant features into the invariant features, formulated as P ◦ L T = P . One option is to add an adaptive max pooling layer at the end of the network.
Let P [ · ] = torch.nn.
AdaptiveMaxPool2d(1) as a function that takes the maximum response over the entire spatial domain.
Since the maximum value won’t be affected by the position distortion of the feature, the output is invariant.
And from Equation (20) we have,
[Equation]
This means the transformation matrix T will not inﬂuence the output, which can be beneﬁcial in tasks such as classiﬁcation.
4. D ISCRETIZATION METHOD
Although the continuous formulation in the previous section is necessary to go, from the intuitive approach (ﬁnd scale + orientation, then ﬁlter accordingly), to the efﬁciently implementable formulation (Equation (14)) through a change of variables in an integral, the digital images or feature maps are usually discrete data aligned on the mesh grid.
Therefore, we detailed the discretization of the integral and approximation implementation.
We rewrite Equation (15) in discrete form as follows,
[Equation]
Where R is the support of ϕ .
Without loss of generality, take 3 × 3 convolution as an example, then R = { ( t x , t y , 1) T | t x , t y ∈ {− 1 , 0 , 1 }} .
n is the number of the elements of the set R .
Let y t
= x + M f ( x ) t .
Generally, this is a fractional location index.
So estimating this value and its gradient becomes a problem.
Here we approximate f ( y t ) using the bilinear interpolation technique, written as, f ( y t ) =
(cid:80) m G ( y t , m ) f
( m ) Where m = ( m 1 , m 2 , 1) T ∈
Z 3 .
G is bilinear interpolation kernel and its formula is G ( m, n )
=
g ( m 1 , n 1 ) g ( m 2 , n 2 ) , where g ( a, b ) = max (0 , 1 − | a − b | ) .
So the similarity convolution becomes
[Equation]
[Equation]
With this implementation, we also make the similarity convolution differentiable.
The gradient of the input is written as,
[Equation]
Here G is a differentiable function, and since it is non-zero only when m aligns on the grid of y t , it does not require too much computation.
E XPERIMENTS 5. C HARACTER RECOGNITION TASK Dataset.
MNIST-
ROT -12 K DATASET (Larochelle et al., 2007) is usually used to validate the rotation equivariant algorithms.
Yet this dataset is insufﬁcient to verify equivariant on both scaling and rotation.
Thus, to better facilitate model comparison, we modify the original MNIST dataset following a similar path and construct the SRT-MNIST DATASET .
Speciﬁcally, we ﬁrst zero-padded the original dataset to 56 × 56 pixel size.
Then we keep training set still and randomly rotate, scaling, and translate each image by a range of θ =
[0 , 2 π ) , s =
[1 , 2[ and t = ± 10 respectively.
This out-of-distribution setting of the test image can sufﬁciently evaluate the model’s generalization ability.
[Table]
Experiments setup.
We use a
ResNet-18 He et al.
(2016) as the architecture.
For every convolution layer, we replace it with our SimConv while maintaining the same trainable parameters.
All layers share the same scalable Fourier-Argand ﬁlters.
We use Adam optimizer (Kingma & Ba, 2015) with a weight decay of 0.01.
Weights are initialized by
Xavier (Glorot & Bengio, 2010).
The learning rate is set to 0.01 and is decayed by a factor of 0.1 every 50 epochs.
We set the batch size as 128, and stopped the training after 200 epochs.
All model parameters are 11.68M, and the FLOPs is 0.12G for an input image with 56 × 56 .
Generalization ability study.
We conduct the ablation study to verify our method’s equivariance property on rotation and scaling independently and concurrently.
In Table 1, R-MNIST, S-MNIST means we only apply rotation or scaling to the test dataset.
SO(2)-Conv and R ∗ -Conv methods have the same structure as our proposed SREN but set all the Γ f or λ f to unitary, which is a convenient way to disable the equivariance property partially.
SREN+ indicates our method with a data augmentation technique randomly rotates for ± 30 degrees, scaling for [0 . 8 , 1 . 2] times.
We can see that SO(2)-Conv and R ∗ -Conv achieved equivariance property on rotation or scaling.
With the equivariant mechanism, our SREN algorithm can reach an accuracy rate of over 95% on every dataset, whereas the CNNs can only overﬁt the original dataset with limited generalization ability.
Equivariance error analysis.
We numerically validate the quality of equivariance by the equivariant error to reveal how stable the equivariance property of the method is and the main factor that affects the stability.
We deﬁne the equivariant error by measuring the normalized L − 2 distance as follows,
[Equation]
26) where || · || F is the Frobenius norm.
The formula is the relative percentage error of the two obtained features after the input is ﬁrst convolved, then transformed, and after the input is ﬁrst distorted and then convolved.
We compare the k -th layer feature with the convolutional network.
The result is shown in Figure 3.
It can be seen that the average error is lower than 0.01.
This shows that our network can be equivariant in good quality.
Besides, we test a speciﬁc case that image is rotated by 90 degrees.
In this case, there will not be a discretization problem for rotation, and the equivariance error can be as small as 1 .
73 × 10 − 6 and almost negligible.
This shows that our method can be extremely accurate for equivariant without discretization.
5. N ATURAL I MAGE C LASSIFICATION
[Table]
Experimentssetup.
To evaluate the generalizationabilityofourmethod ,wedo experimentson the challenge oftheSTLdatasetCoates et al.
Thelabeledsubset is an excellent choice to evaluate how efﬁcient the network can use these limited training samples and how much the network’s generalization ability is when the training set is small.
We evaluate methods by in-distribution testing (ID) that unchange the test dataset
[Table]
and out-of-distribution testing that randomly rotates and scales the dataset.
The OOD test measures the ability of a method to handle the never-seen inputs.
We use the ResNet (He et al., 2016) with 16 layers as our backbone, and use our SimConv layers to replace all convolutional layers.
We trained the network for 1000 epochs with a batch size of 128.
And we use Adam as the optimizer.
The initial learning rate is set to 0.1 and adjusted with a cosine annealing schedule during training.
Following Zhu et al.
, data augmentation without scaling and rotation is also applied.
We compare our method with Rotation Decomposed
Convolutional Filters network (RDCF)
(Cheng et al., 2019), Scale-Equivariant Steerable Networks (SESN)
(Sosnovik et al., 2020), Scale Decomposed Convolutional Filters network (SDCF) (Zhu et al., 2019), Roto-Scale-Translation Equivariant CNNs (Gao et al., 2022).
All methods’ backbone is set to ResNet (He et al., 2016) with 16 layers to make the model parameters comparable and make the comparison fair enough.
Results & Discussion.
Table 2 demonstrates our main results compared to recent baselines.
R -Equi and S -Equi indicate whether method achieves equivariance property in rotation or scaling.
Conti means whether the method achieves equivariance at a continuous scale or rotation.
As we can see, our method achieved the highest accuracy among all other approaches, especially for the out-of-distribution test.
This shows that our method has a good generalization ability.
Besides, to be noticed that we make a comparison with methods that “partially” achieve equivariant since there are no like-for-like methods (achieving rotation and rotation scaling equivariant in the continuous group) to compare.
MacDonald et al.
(2022) is another paper that guarantee the equivariance to any ﬁnite-dimensional Lie group, but the memory efﬁciency limits it to scale up to a large network and make an affair comparation.
It is not easy to achieve this property efﬁciently, and this paper is one of the few ways to achieve this property.
Being able to achieve this property itself is already another state-of-the-art.
C ONCLUSION
Although there have been numerous studies on how to achieve rotation and scale equivariant, achieving continuous equivariant in rotation and scaling is novel, to our knowledge.
In this paper, we propose to develop scalable steerable ﬁlters based on the Fourier-Argand representation and to use the local scale and orientation provided by these ﬁlters to empower the convolution operator with local scale and rotation equivariant: SimConv.
Mathematical and experimental analyses are detailed to explain why it works and to what extent it can achieve the desired property.
A P ROOF OF S CALABLE F OURIER A RGAND R EPRESENTATION
Deﬁnition A.1.
Consider a square integrable function h , and express it as h ( r, θ ) in polar coordinates.
We propose its Scalable Fourier Argand representation as the following series form,
[Equation]
Where h k 1 ,k 2 : Z 2 → C is the coefﬁcient of each item
[Equation]
which can be computed as follows,
[Equation]
Proof.
We start from Equation (27) and replace the polar coordinate r by an exponential term r = e ρ
[Equation]
then ρ = ln r and we have,
[Equation]
Let’s deﬁne a quantity g k 1 ,k 2 as follows,
[Equation]
Replace h ( e ρ , θ ) in Equation (30) with the scalable Fourier-Argand representation in Equation (29), we have,
[Equation]
[Equation]
[Equation]
The last equation holds since, when t 2 = k 2 , then,
[Equation]
And when t 2 (cid:54) =
k 2 , then
[Equation]
The holds of Equation (31) proves the proposition [Equation]
B P ROOF OF D ISTORTION P ROPERTY
Lemma B.1.
Let T
= S s Y t
R α , then for a transformed input L T [ f ]( x ) , we can get the following property,
[Equation]
Proof.
Let’s consider f ( x ) and L T [ f ]( x ) as the original and distroted image.
Using the scalable
Fourier Argand basis [Equation] we have,
[Equation]
Substitude x by T − 1 x , we have,
[Equation]
Besides
[Equation]
from Equation (10) we have, [Equation]
[Equation]
We have,
[Equation]
Finally
[Equation]
we can conclude that,
[Equation]
C D ETAILS , A NALYSIS , AND V ISUALIZATION C. A DDITIONAL S TUDIES
Stability We ﬁrst quantify the deformation stability, and seek to answer in what degree of distortion can the method persist the equivariant (or the generalization ability) compared with original convolution.
To achieve this, we evalue our method on the test dataset that rotated and scaled entirely by a certain amount.
We do this experiment with the MNIST dataset.
Figure 4 displays the accuracy decay with different settings.
It can be observed that when rotate the image that largely differ from the training dataset, the CNNs will drop accuracy largely, while our method maintains a consistently excellent performance.
For scale changing test, our method maintain a relatively good capability on a wide range of scale change.
For extremely large scale change, our method have performance drop due to limited ﬁlter spot and sampling approximation.
C. F EATURE VISUALIZATION
Besides numerical experiments, we also visualize the features of the network to verify the equivariance achieved by our network more intuitively.
The visualization result is shown [-in Figure 
-]inFigure 5.
This visualization answer the question whether the hidden feature visually looks equivariant with respect to rotation and scaling. The parameters are randomly initialized for both CNN and SREN model.
With the compensation view, we can clearly see that our network indeed achieve the equivariant in rotation and scaling.
C. A RCHITECTURE AND PIPELINE
The main ﬂow of our algorithm is shown in Figure 2.
This method can be applied to a multi-layer network structure.
Assuming that the network contains K blocks (which we named as SimBlock), where each block contains multiple convolutional layers. And the input signal of the i -th block is denoted as f i .
We ﬁrst calculate the scalable Fourier Argand feature proposed in Section 4.1.
This allows us to get a spatial-wised matrix M f i ( x ) .
Then all the SimConv layer (proposed in Section 4.2) within the SimBlock share this same scalable Fourier Argand feature. Let’s assume there are N SimConv layers in each SimBlock.
Based on the property of Equation (21), the output feature of this block, denoted as f i +1 , is guaranteed to be an equivariant feature corresponding to the input image.
For the classiﬁcation task, the invariance property is desired.
We pooled the last layer for each channel, followed by the MLP layer.
The ﬁnal output is then obtained and guaranteed invariant for any similar transform.
The whole process is shown in Algorithm 1.
C. STL-10 DATASET COMPARATION
This section compare our method with other related methods with the same backbone. All method use WideResNet as the backbone with 16 layers and a widen factor as 8. We trained our network for 1000 epochs with a batch size of 128, we use SGD as the optimizer. The initial learning rate is set to 0.1 and decreased by a factor of 0.2 after 300 epoch, and the drop rate probability as 0.3.
We ues the data augmentation technique following Sosnovik et al.
(2021a).
Table 3 demostrates our main results compared to recent baselines, such as SiCNN (Kanazawa et al., 2014), DSS (Worrall & Welling, 2019), SESN (Sosnovik et al., 2020), DISCO (Sosnovik et al., 2021a).
Our method obtained a competitive result that is close to the state-of-the-art paper.
Besides, we want to highlight that our method’s computation cost and parameters are similar to the classic convolutional neural network with the same backbone.
This is because the two quantities Λ and Γ can be shared within different ﬁlters.
Thus the cost is small compared with the abundance of the convolutional (or our
SimConv) layer.
And the complexity of our SimConv operator is similar to the convolutional layer.
Input: Batch of data { f i } Ti =
Output: predicted output value v for i = 1 : K levels do
Initialize ﬁlter h , calculate its basis H k 1 ,k 2 ; Take f i as input;
Get f k 1 ,k 2 ← f (cid:63) H k 1 ,k 2 ; Get Γ f i , λ f
i ← arg max γ,λ c γ,λ F ; Calculate M f ← Γ f i , λ f i ; for m = 1 : M layers in speciﬁc level n do
Apply M f to SimConv; f i,m +1 ← f i (cid:126) ϕ i,m ; f i +1 ← f i,M
Get the output with a head layer: v ← MLP ( f ) [Table]
For comparison, the computational cost of DISCO method takes more than 5 times longer and SESN takes more than 16.5 times longer than classic CNNs, reported in Sosnovik et al.
(2021a).