E XPLOITING R EDUNDANCY : S EPARABLE G ROUP C ONVOLUTIONAL N ETWORKS ON L IE G ROUPS
Anonymous authors
Paper under double-blind review
A BSTRACT
Group convolutional neural networks (G-CNNs) have been shown to increase parameter efﬁciency and model accuracy by incorporating geometric inductive biases.
In this work, we investigate the properties of representations learned by regular G-CNNs, and show considerable parameter redundancy in group convolution kernels.
This ﬁnding motivates further weight-tying by sharing convolution kernels over subgroups.
To this end, we introduce convolution kernels that are separable over the subgroup and channel dimensions.
In order to obtain equivariance to arbitrary afﬁne Lie groups we provide a continuous parameterisation of separable convolution kernels.
We evaluate our approach across several vision datasets, and show that our weight sharing leads to improved performance and computational efﬁciency.
In many settings, separable G-CNNs outperform their non-separable counterpart, while only using a fraction of their training time.
In addition, thanks to the increase in computational efﬁciency, we are able to implement G-CNNs equivariant to the Sim(2) group; the group of dilations, rotations and translations of the plane.
Sim(2) -equivariance further improves performance on all tasks considered, and achieves state-of-the-art performance on rotated MNIST.
I NTRODUCTION
Minsky & Papert (1988) suggest that the power of the perceptron comes from its ability to learn to discard information irrelevant to the problem it is tasked with solving.
In other words; information that is not relevant to the task does not inﬂuence the representation built by the network.
According to Minsky & Papert (1988) this leads to a deﬁnition of perceptrons in terms of the symmetry groups their learned representations are invariant to.
Progress in geometric deep learning has shown the power of pro-actively equipping models with such geometric structure as inductive bias, reducing model complexity and improving generalisation and performance (Bronstein et al., 2017).
An early example of such geometric inductive bias at work can be seen in the convolutional layer in a CNN (LeCun et al., 1998).
CNNs have been instrumental in conquering computer vision tasks, and much of their success has been attributed to their use of the convolution operator, which commutes with the action of the translation group.
This property, known as equivariance to translation, comes about as a result of the application of the same convolution kernel throughout an input signal, enabling the CNN to learn to detect the same features at any location in the input signal, directly exploiting translational symmetries that naturally occur in many tasks.
Although invariance to object-identity preserving transformations has long been recognised as a desirable model characteristic in machine learning literature (Kondor, 2008; Cohen, 2013; Sifre & Mallat, 2014), only recently Cohen & Welling (2016a) introduced the Group Equivariant CNN (G-CNN) as a natural extension of the CNN (LeCun et al., 1998), generalising its equivariance properties to group actions beyond translation.
The layers of a G-CNN are explicitly designed to be equivariant to such transformations, hence the model is no longer burdened with learning invariance to transformations that leave object identity intact.
It has since been shown that equivariant deep learning approaches may serve as a solution in ﬁelds that as of yet remain inaccessible to machine learning due to scarce availability of labelled data, or when compact model design due to limited computational power is required (Winkels & Cohen, 2018; Linmans et al., 2018; Bekkers, 2019).
Complexity and redundancy issues impeding regular group convolutions. A growing body of work shows applications of G-CNNs consistently and decisively outperforming classical CNNs et al., 2017; Weiler et al., 2018a;
Bekkers et al., 2018; Esteves et al., 2018; Bekkers, 2019; Worrall & Welling, 2019; Sosnovik et al., 2021b)
However, a practical challenge impeding application to larger groups is the computational complexity of regular group convolutions, which scales exponentially with the dimensionality of the group.
Furthermore, Lengyel & van Gemert (2021) show that group convolution ﬁlters in the original formulation of the G-CNN by Cohen &
Welling (2016a) exhibit considerable redundancies along the group axis for the p 4 m and Z 2 groups.
Similar observations motivated depthwise separable convolutions (Chollet, 2017), which not only increased parameter efﬁciency but also model performance; observed correlations between weights are explicitly enforced with further parameter sharing through the use of kernels separable along spatial and channel dimensions.
We address the observations of redundancy along with the scalability issues of regular G-CNNs in their current form.
Our paper contains the following contributions:
• We introduce separable group convolutions for afﬁne Lie groups R n (cid:111) H , sharing the kernels for translation elements x ∈ R n along subgroup elements h ∈ H .
See Fig.
1 for an overview.
• We propose the use of a SIREN (Sitzmann et al., 2020) as kernel parameterisation in the Lie algebra - imposing a ﬁxed number of parameters per convolution kernel, regardless of the resolution at which this kernel is sampled, and ensuring smoothness over the Lie group.
• Separable group convolutions allow us to build Sim(2) -CNNs, which we thoroughly experiment with.
We show equivariance to Sim(2) increases accuracy over a range of vision benchmarks.
• To achieve equivariance to continuous afﬁne Lie groups, we propose a random sampling method over subgroups H for approximating the group convolution operation.
First, we position this work within the area of equivariant deep learning by giving an overview of related works, and explaining which current issues we are addressing with this work.
We derive separable group convolutions, and show how they may be applied to continuous groups.
Lastly, we apply these ideas by experimenting with implementations for roto-translations in 2D ( SE(2) ), dilation and translation in 2D ( R 2 (cid:111) R + ) and dilation, rotation and translation in 2D ( Sim(2) ).
R ELATED W ORK
Group equivariant convolutional neural networks. Broadly speaking, research on G-CNNs can be divided into two approaches.
First, Regular G-CNNs use the left-regular representation of the group of interest to learn representations of scalar functions over the group manifold, or a quotient space of the group.
The left-regular representation acts on the convolution kernels, yielding an orbit of the kernel under the group action.
Convolving the input using these transformed ﬁlters, a feature map deﬁned over the group is obtained at each layer.
This approach most naturally extends the conventional CNN, where convolution kernels are transformed under elements of the translation group.
Regular G-CNNs have been implemented for discrete groups (Cohen & Welling, 2016a;
Winkels & Cohen, 2018; Worrall & Brostow, 2018), compact continuous groups (Marcos et al., 2017;
Bekkers et al., 2018) and arbitrary non-compact continuous Lie groups (Bekkers, 2019; Finzi et al., 2020; Romero et al., 2020).
However, practical implementations for continuous groups often require some form of discretisation of the group, possibly introducing discretisation artefacts, and requiring a choice of resolution over the group.
For the second class, steerable G-CNNs, representation theory is used to compute a basis of equivariant functions for a given group, which are subsequently used to parameterise convolution kernels (Cohen & Welling, 2016b; Weiler et al., 2018a;b; Sosnovik et al., 2019; 2021a).
Although steerable G-CNNs decouple the cardinality of the group from the dimensionality of the feature maps, this approach is only compatible with compact groups.
The current paper may, in approach, be compared to Bekkers (2019) and Finzi et al.
(2020), who deﬁne convolution kernels on the Lie algebra of continuous groups to enable convolutions on their manifold.
Similarly, we make use of the Lie algebra and exponential map to obtain convolution kernels on the group, but separate the kernels by subgroups.
Bekkers (2019) deﬁnes a set of basis vectors in the Lie algebra, which, when combined with the exponential map, allow for the identiﬁcation of group elements by a vector in R n .
Subsequently, a set of B-splines is deﬁned on the algebra, which form a basis to expand convolution kernels in.
A linear combination of these bases creates a locally continuous function on the Lie algebra deﬁning a convolution kernel and its behaviour under transformations of the group.
Although this method allows for direct control over kernel smoothness, the learned convolution ﬁlters are limited in their expressivity by their basis functions.
Finzi et al.
(2020) instead use an MLP to learn convolution kernels on the Lie algebra, which in addition allows them to handle point cloud data.
The MLP is constructed to learn kernel values at (arbitrary) relative offsets in the Lie algebra.
In contrast, we propose to use SIRENs (Sitzmann et al., 2020) to parameterise convolution kernels, as they have been shown to outperform other forms of MLPs in parameterising convolution kernels (Romero et al., 2021), and offer more explicit control over kernel smoothness: a desirable property for addressing discretisation artefacts that occur when modelling features on continuous groups (see App. A.6).
Separable ﬁlters in machine learning In image processing, spatially separable ﬁlters have long been known to increase parameter- and computational efﬁciency, and learning such constrained ﬁlters may even increase model performance (Rigamonti et al., 2013).
In Sifre & Mallat (2014), authors investigate SE(2)-invariant feature learning through scattering convolutions, and propose separating the group convolution operation for afﬁne groups into a cascade of two ﬁlterings, the ﬁrst along the spatial dimensions R n , and the second along subgroup dimension H .
From this, authors derive a separable approach to the convolution operation with learnable ﬁlters as used in CNNs.
This formulation has since been named the depthwise-separable convolution (Chollet, 2017), a special case of the Network-In-Network principle (Lin et al., 2013) which forms the basis for the success of the Inception architectures (Szegedy et al., 2015).
In depthwise separable convolutions, each input channel ﬁrst gets convolved using a (set of) kernel(s) with limited spatial support.
Afterwards, a 1x1 convolution is used to project the feature set detected in the input channels to the output space.
Chollet (2017) speculates that the Inception architectures are successful due to the explicit separation of spatial and channel mapping functions, whereas in conventional CNNs, kernels are tasked with simultaneously mapping inter-channel and spatial correlations.
Haase & Amthor (2020) argue that the original formulation of the depthwise-separable convolution reinforces inter-kernel correlations, but does not in fact leverage intra-kernel correlations.
Subsequently, they propose an inverse ordering of the operations given in depthwise-separable convolutions, sharing the same spatial kernel along the input channels, and show convincing results.
Extending this investigation of learned convolution ﬁlters to the original G-CNN (Cohen & Welling, 2016a),
Lengyel & van Gemert (2021) remark on the high degree of correlation found among ﬁlters along the rotation axis, and propose to share the same spatial kernel for every rotation feature map.
We attempt to generalise this approach, proposing separable convolutions on arbitrary afﬁne Lie groups.
B ACKGROUND
In the following section we give the theoretical background for implementing group convolutions for arbitrary afﬁne Lie groups.
We assume familiarity with the basics of group theory and provide the relevant concepts in Appx.
A.1 and Appx. A.4.
For simplicity of notation, we initially assume that our input signal/feature map f has a single channel.
Lifting convolutions. To preserve information on the pose of features in the input, an equivariant convolution operation is achieved by lifting a function from the input space to (a homogeneous space of) the group.
As we are interested in real signals, speciﬁcally image data living on R 2 , we assume the Lie group of interest H is taken in semidirect product with the domain of our data; G = R 2 (cid:111) H .
In group convolutions, a given kernel is left-acted by all transformations in G , thereby generating a signal on the higher dimensional space G instead of R 2 .
Hence, the output feature maps disentangle poses through a domain expansion, e.g. positions plus rotations or scales.
For a given group element h ∈ H , kernel k , and location (cid:126)x in the input domain R 2 , the lifting convolution is given by:
[Equation]
where g =( x , h ) and k h
= 1 | det h | L h [ k ] is the kernel k : R 2 → R transformed via the action of group element h via L h [ k ]( x ) :
=
k ( h − 1 x ) , and with det h the determinant of the matrix representation of h that acts on R d .
The output of lifting convolutions yields a G -feature map with the original two spatial input dimensions ( R 2 ), and an additional group dimension ( H ).
See Fig.
Group convolutions. Now that the data is lifted to the domain of the group, we continue with group convolutions in subsequent layers.
Given a kernel k (now a function on G ), Haar measures d˜ g and d˜ h on the group G and sub-group H respectively, group convolutions are given by:
[Equation]
Evaluating this convolution for every group element g ∈ G , we again obtain a function deﬁned on G .
As we know G = R 2 (cid:111)
H , we can factor this operation into a transformation of a kernel k by a group element h ∈ H , k h =
L h ( k ) , followed by a convolution at every spatial location in f .
See Fig.
Achieving invariance. Using lifting and group convolution operations, we can construct convolutional layers that co-vary with the action of the group and explicitly preserve pose information throughout the representations of the network.
In most cases, we ultimately want a representation that is invariant to transformations of the input in order to achieve invariance to these identity-preserving transformations.
This is achieved by aggregating the information at all group elements in a feature map with an operation invariant to the group action, e.g., max-, mean- or sum-projection.
In practice, this is done after the last group convolution, and is followed by one or more fully connected layers.
S EPARABLE G ROUP C ONVOLUTIONS ON L IE G ROUPS
Redundancies in group convolution ﬁlters. Similar to Haase & Amthor (2020) Lengyel & van
Gemert (2021)
we investigate parameter efﬁciency of learned convolution kernels to motivate separable ﬁlters.
We train an SE(2) -equivariant CNN on the Galaxy10 dataset (see Sec. 5 for experimental details) and analyse the resulting group convolution kernels.
We apply PCA by treating the values of the group convolution kernel for each subgroup element h ∈ H as distinct spatial kernels with k × k features.
The ratio of variance explained by the ﬁrst principle component gives an indication of the variability of the group convolution kernel along the subgroup axis.
If the ratio of explained variance is high, the distinct spatial convolution kernels along the subgroup axis are well-characterised by a single shared kernel.
In Fig.
2, results are shown for the group convolution layers in our SE(2) -CNN before and after training.
We ﬁnd that during the training process, redundancy along the subgroup axis increases considerably.
This motivates sharing a single spatial kernel along the subgroup elements, which can be achieved by separating the group convolution operation into a convolution over the subgroup H , followed by a convolution over the spatial domain R 2 .
Separable group convolutions.
Let us assume that the convolution kernel k : G → R in Eq. 2 is separable. That is, k ( g )=
k R 2 ( x ) k H ( h ) .
In this case, we can derive the factorised separable group convolution as (see Appx.
A.2 for the full derivation):
[Equation]
Here k H is a convolution kernel over the group H , and k R 2 is a convolution kernel over the spatial domain R 2 .
We set k ϕ = k ϕH k ϕ R 2 , with k ϕH constant along R 2 and k ϕ R 2 constant along H .
This can be thought of as parameterising a convolution kernel k over the group G by sharing same spatial kernel k ϕ R 2 weighed by a value k ϕH ( h ) at every input group element h ∈ H , see Fig.
Importantly, this factorisation greatly increases the efﬁciency of the group convolution operation.
Since we can precompute the inner integral in Eq. 3, convolving over a single channel of a feature map f of size | H |× x × y goes from O ( | H | 2 × x × y × k 2 ) to O ( | H |× x × y × ( | H | + k 2 )) .
See Fig.
13 for a visual intuition of the separable group convolution.
Deﬁning convolution kernels on Lie algebras.
In order to perform lifting- and group convolutions (Eqs. 1, 2), we need to evaluate our convolution kernels k at relative offsets g (cid:48) on the group.
As in Bekkers et al.
; Weiler et al.
(2018b)
; Bekkers (2019); Finzi et al.
(2020), we express our group convolution kernel k in analytical form, as a function of relative group elements yielding kernel values.
Motivated by ﬁndings in Romero et al. (2021), we use a Sinusoidal Representation Network (SIREN) (Sitzmann et al., 2020) as kernel parameterisation.
SIRENs lead to great performance improvements over MLPs with ReLU , LeakyReLU and Swish when parameterising convolution kernels (we replicate this comparison for G-CNNs in Appx.
C.4).
Furthermore, the SIREN offers explicit control of kernel smoothness through a frequency multiplier parameter ω 0 : an important property discussed in Appx.
A.6.
Since the afﬁne Lie group can be non-euclidean, and neural networks are functions generally deﬁned over euclidean spaces, we resort to deﬁning the kernel function on the Lie algebra of our group of interest (Finzi et al., 2020).
The kernel function k θ maps points in the Lie algebra (which may be associated with the relative offsets on the group we are convolving over by the exponential map) to kernel values, k ϕ : g → R .
For a given element g , we have: k ϕ ( g )=SIREN(log g ) , see Fig.
The separable kernels follow the same principle, separated by subgroups, see Appx. B.1.
Approximating equivariance for compact continuous and non-compact continuous groups. In the case of small discrete subgroups H , such as the group C 4 of rotations by 90 degrees, we are able to perform lifting- and group convolution operations exactly equivariant to the action of the group, as the integral given in Eq.
2 is tractable.
We consider image data in particular, which is deﬁned over a discrete grid Z 2 , a subgroup of R 2 , making both integrals over the group discrete sums:
[Equation]
with ∆˜ x and ∆˜ h denoting the volume elements corresponding to the grid points.
In the case of continuous groups, it is possible to either make a discretisation of the subgroup H , or to approximate the group convolution by means of random sampling.
To obtain a volumetrically uniform sampling grid over the group, we sample a set of n equidistant points in the Lie algebra as in Bekkers (2019), and map those to the group using the exponential map to obtain a grid H :=
[ h e , ..., h n ] , see Fig.
For noncompact H such as the dilation group R + , we localize the support in the Lie algebra.
In the case of compact continuous groups such as SO(2) we approximate the integral by convolving over a uniformly spaced grid H which is perturbed by left-multiplication with a uniformly randomly sampled group element h (cid:15) ∼ d˜ h , i.e., h (cid:15) (cid:55)→ H (cid:15) :
= h (cid:15) H .
By uniform sampling of H we obtain left-invariant ∆˜ h which only scales the overall convolution result, allowing us to omit it from 4.
We further let ∆ ˜ x = 1 and obtain the discrete separable group convolution for continuous groups:
[Equation]
During training and inference, randomly sampling over the rotation group yields an unbiased estimation (Wu et al., 2019), making the network approximately equivariant to the continuous group SO(2) . Discretisation makes the network exactly equivariant only to a discretised subgroup of SO(2) , biasing the network to be equivariant to a ﬁxed subset of transformations.
In our experiments we show that randomly sampling outperforms discretisation when modelling equivariance to SO(2) .
Channel support of separable group convolution kernels In CNNs (and G-CNNs), a feature map f l at layer l generally has multiple channels C l ; f : R 2 → R C l .
Factorising the group convolution operation into distinct subgroup and spatial dimensions presents us with a choice: we can either (1) deﬁne both k ϕH and k ϕ R 2 over the input channels, or choose to (2) reduce the support of our spatial kernel to a single channel, and share a reweighing of it along the input channels in an inverse depth-wise separable manner, similar to Haase & Amthor (2020).
An ablation study detailed in Appx.
C.2 shows that on a ﬁxed parameter budget, additional depthwise separation (2) consistently outperforms (1) deﬁning both kernels over the input channels.
As such, we will keep to the additional depthwise separation and refer to this implementation as the separable group convolution.
Expressivity of separable group convolutions. Separable convolution kernels are strictly less expressive than their non-separable counterpart, but to what extent would this limit the expressivity of their learned representations?
In separable group convolutions, we are sharing a weighted version of a single spatial kernel along the input group axis.
In contrast, we could view non-separable group convolutions as having the ability to learn distinct spatial conﬁgurations of features along the group axis.
For a visual example, see Appx.
A.5.
Although this reduction in expressivity could theoretically prove limiting in the application of G-CNNs on vision problems, in the next section we experimentally show that separable group convolutions in fact often outperform their non-separable counterpart.
E XPERIMENTS
We empirically motivate the use of seperable group convolutions by studying the performance of three G-CNNs that incorporate equivariance to three distinct groups.
As our goal is to isolate the effect of separating the group convolution, we use the same shallow ResNet architecture throughout all experiments, only varying the sampling resolution over the group, see Appx.
B.1 for details.
After the last convolution block, we apply max-projection over the remaining group dimensions to achieve an invariant representation.
We experiment with three different groups acting on R 2 : the roto-translation group ( SE(2) ), the translation-dilation group ( R 2 (cid:111) R + ) and the group of rotations, dilations and translations ( Sim(2) ).
For SE(2) , we ﬁrst evaluate the inﬂuence of random sampling versus discretisation of SO(2) .
Next, we assess the difference in performance of separable and non-separable group convolutions on transformed MNIST variants; MNIST-rot, MNIST-scale for SE(2) and R 2 (cid:111) R + respectively.
Due to the reduction in computational complexity separable group convolutions bring, we are able to model equivariance to higher-dimensional groups: we implement two Sim(2) -CNNs and evaluate them on MNIST-rot-scale.
Lastly, to investigate the advantages equivariance brings in more complex problem settings, we experiment with all setups on three vision benchmark datasets; CIFAR10, CIFAR100 (Krizhevsky et al., 2009) and Galaxy10 (Leung & Bovy, 2019).
See Appx.
B.2 for more details on the datasets and training regimes used, and Appx.
C.1 for an additional experiment empirically validating the equivariance properties of our models.
Discretisation versus random sampling over compact subgroups. We investigate random sampling to approximate the group convolution.
We trained SE(2) -CNNs on rotated MNIST for different resolutions over SO(2) , using random sampling and discretisation.
Results are shown in Fig.
We can clearly see the advantage of approximating the group convolution integral through random sampling, likely attributable to the fact that through random sampling we obtain an unbiased estimator for the convolution, whereas discretisation equates to a biased sampling of the rotation group.
Separable and non-separable G-CNN performance on MNIST variants. Following Cohen &
Welling (2016a)
; Weiler et al.
(2018b); Sosnovik et al.
; Finzi et al.
we conduct experiments on MNIST variants with separable and non-separable implementations of equivariant models for each of the three groups.
We discuss results per dataset in-depth.
(2) resolution for ble SE(2) -CNNs Rotated MNIST. The SE(2) -CNNs are evaluated on rotated MNIST, the standard benchmark for rotation equivariant models.
To assess the inﬂuence of sampling resolution over the group, we vary the number of sampled group elements from 1 to 20.
The group convolution is approximated through random sampling.
Results are shown in Fig.
We see the inﬂuence of increasing the resolution over the rotation group from 1 to 4 elements, after which the non-separable model saturates, and performance starts to drop when further increasing the number of sampled rotations.
This is in line with ﬁndings by Bekkers et al.
(2018), who note that as the group resolution increases, so does the possibility for overﬁtting on speciﬁc spatial conﬁgurations.
The separable implementation saturates around 7 elements, but performance does not drop signiﬁcantly beyond this point.
This may imply that the reduction in kernel expressivity also has a regularising effect that beneﬁts generalisation.
In this experiment, separable group convolutions decisively outperform the non-separable variant.
Scaled MNIST.
The R 2 (cid:111)R + -CNNs are evaluated on MNIST scale.
The group convolution integral is approximated through discretisation.
In this experiment, we retain the same resolution over the dilation group in all experiments, but instead vary the value at which the group is truncated, ranging from 1.0 at a grid of 1 element in the Lie algebra to 3.0 at a grid of 20 group elements.
As in
Sosnovik et al.
, we found that the inter-scale interactions in the group convolution operation reduced performance, likely because of increase in equivariance error due to truncation of the group.
Therefore, we limited support of the group convolution kernel on R + to two neighbouring elements for all experiments in this paper.
Results are shown in Fig.
Here, non-separable group convolutions outperform the separable variant, suggesting that the ability of non-separable group convolutions to model spatial feature patterns over different scales is beneﬁcial for the scaled MNIST dataset.
Scaled rotated MNIST. Lastly, the Sim(2) -CNN is benchmarked on MNIST-rot-scale.
We limit the sampling resolution at each layer of the G-CNN to 2,4,6 and 8 elements for each subgroup.
We test two implementations: (1) separable, where the group convolution is factorised into two convolutions, one over R + (cid:111) SO(2) and one over R 2 , and (2) H -separable, which factorises the convolution into three steps: a convolution over R + , one over SO(2) , and one over R 2 .
As shown in Fig.
6 the separable implementation turned out to be unstable for low SO(2) resolutions.
We suspect that an approximation of SO(2) of only 2 rotation elements, paired with possible aliasing effects and equivariance error occurring over the dilation group, impedes the model from learning robust representations.
For higher resolutions over SO(2) , this effect seems to decrease.
The results for the H -separable implementation (Fig. 7), highlight a much clearer increase in performance as the resolution over both groups increases, which suggests decoupling these information mappings stabilises the learning process.
Application to computer vision benchmarks. To evaluate the value of equivariance to different transformations groups in natural image classifcation, we apply the SE(2) -, R 2 (cid:111)R + - and Sim(2) -
CNNs to three vision datasets.
For SE(2) and R 2 (cid:111)R + we use resolutions of 4 and 8 group elements, and truncate the scale group at a value of √ 3 , as we found this to work well for our experiments.
For
Sim(2) , we use a resolution of 4 elements over the SO(2) group and 4 elements over the R 2 (cid:111)R + subgroups and again truncate R + at √ 3 .
On CIFAR10, as GPU memory allows, we also experiment with using resolutions of 8 group elements for both subgroups.
As baseline, we implement the same architecture of our non-separable G-CNN, but restrict it to equivariance to translations.
Results are summarised in Table 1 and exhibit the following general pattern:
For most groups, separable group convolutions outperform non-separable ones, and in all experiments best performance is achieved by a Sim(2) -CNN.
Next, we discuss results per dataset in-depth. [Figure]
[Figure] [Table] CIFAR10. First, we evaluate our models on the CIFAR10 dataset (Krizhevsky et al., 2009).
Results in Table 1 show that on CIFAR10, the non-separable variant of our SE(2) -CNN is outperformed by the translation equivariant baseline.
With separable group convolutions and a group resolution ofelements, SE(2) -CNN show improved performance.
Interestingly, R 2 (cid:111)R + -CNNs outperform both baseline and SE(2) -CNNs, so it seems scale invariance is a better inductive bias than rotation invariance in CIFAR10.
Combining scale and rotation invariance ultimately leads to the best performance, with separable Sim(2) -CNNs and a resolution of 8 scale and 8 rotation group elements achieving 89 .
38% test accuracy.
The H -separable Sim(2) -CNN, which models no interactions between scale and rotation, performs slightly worse.
CIFAR100. To assess the impact of separating spatial and group information in more complex problem settings, we experiment with CIFAR100.
Notably, the non-separable R 2 (cid:111) R + -CNNs perform well under baseline.
We discuss this particular ﬁnding in Sec. 6.
For SE(2) -CNNs, non-separable convolutions outperform separable ones, suggesting the model beneﬁts from modelling distinct spatial conﬁgurations at different poses.
Again, best performance is achieved by the Sim(2) -CNN.
Galaxy10. Lastly, we apply our models in the domain of astro-photography, on the Galaxy dataset Leung & Bovy (2019).
Rotation and scaling symmetries exist naturally in this dataset, making it an interesting application for our invariant models.
Here, results show little difference between performance of non-separable and separable approaches.
Rotation invariance seems to be the stronger inductive bias in this dataset, but again, combining rotation and scale invariance leads to the best performance, with the H -separable Sim(2) -CNN achieving 87 .
45% test accuracy.
The higher separable Sim(2) test error seems to indicate that interactions between scale and rotation features for this dataset impede model performance, again, possibly due to overﬁtting. [Table] [Table]
Comparison in training time between separable and non-separable group convolutions. As one of the main motivators for the separable group convolution is its theoretical increase in computational efﬁciency, a comparison is made in training time between separable and non-separable G-CNNs.
We tracked processing times per epoch, which consists of a train- and inference procedure, for the rotated MNIST experiments, and show them in Fig. 8.
Separable group convolutions present a remarkable decrease in inference time, which may be leveraged to allow for model equivariance to larger groups. SOTA on rotated MNIST. To compare separable G-CNNs with related work, we ﬁnetune our models to competitive performance. With minimal adjustments to model and training regime, listed in App.
B.2, we are able to achieve state of the art performance on MNIST-rot, see Tab. 2. Competitive performance on CIFAR10.
We compare performance of our approach in a larger model. To this end, like (Cohen & Welling, 2016a), we re-implement the All-CNN-C architecture by Springenberg et al. (2014), using our Sim(2) convolution layers as drop-in replacement. To also compare performance in absolute numbers of trainable parameters, we train three conﬁgurations, see App.
B.2.
We show that Sim(2) -CNNs, with SIREN hidden sizes of 6 and 5 units, containing similar or smaller numbers of trainable parameters to the original All-CNN-C, improve model accuracy compared to CNNs or G-CNNs equivariant to Sim(2)
-subgroups R 2 and p 4 , and with data augmentation 6- Sim(2) -CNNs also outperform p 4 m -G-CNNs, see Tab. 3.
Increasing SIREN hidden size to 16 leads to even more signiﬁcant improvements in performance.
D ISCUSSION & F UTURE W ORK
On implicit kernel representations Some of the non-separable conﬁgurations generated results below baseline, which is unexpected given that G-CNNs usually improve upon regular CNNs.
We conjecture that these results are caused by the SIREN parameterisation, which take as input a coordinate vector of different units (mixing both spatial and sub-group h coordinates).
This could limit kernel expressivity, as sharing of a frequency multiplier ω 0 sets kernel smoothness to be identical along spatial and subgroup dimensions.
The fact that for the separable variant we use two distinct SIRENs would lift this restriction.
Despite useful advantages of MLP parametrisations for the kernels, such as their ﬂexibility and ease of implementation for arbitrary Lie groups, it remains an open problem to have full control over their smoothness; ideally one band-limits the kernel MLPs to the discretised resolution on each axis, which would be an important direction for future work.
Conclusion Motivated by observed redundancies in learned group convolution ﬁlters, we introduced separable group convolutions, a computationally efﬁcient implementation of regular group convolutions.
We showed that separable group convolutions not only drastically increase computational efﬁciency, but in many settings also outperform their non-separable counterpart.
Furthermore, we demonstrated the value of separable group convolutions as a solution for modelling equivariance to larger groups; Sim(2) -CNNs decivisely outperform models only equivariant to subgroups of Sim(2) , clearly reinforcing equivariance as means of model generalisation.
A T HEORY AND BACKGROUND A. G ROUP THEORETIC PRELIMINARIES
In this section group theoretical prerequisites used throughout the paper are brieﬂy refreshed.
This is by no means intended as an exhaustive exposition of the fundamentals of group theory, we only introduce those concepts relevant to the current work.
Group.
A group is deﬁned by a set G of group elements , along with a binary operator · : G × G → G , called the group product .
The group product deﬁnes a way to combine each pair of elements g 1 , g 2 ∈ G .
For the binary operator · to be considered a group product it needs to satisfy four constraints:
Closure.
G is closed under · ; for all g 1 , g 2 ∈ G we have g 1 · g 2 ∈ G . 2.
Identity.
There exists an identity element e s.t.
for each g ∈ G , we have e · g = g · e = g . 3.
Inverse.
For every element g ∈ G we have an element g − 1 ∈ G ,
s.t. g
· g − 1 = e . 4.
Associativity.
For every set of elements g 1 , g 2 , g 3 ∈ G , we have ( g 1 · g 2 ) · g 3 = g 1 · ( g 2 · g 3 ) .
Lie groups.
A Lie group is a group of which the elements form a smooth manifold.
Since the group itself is not necessarily a vector space, combination of elements through addition or subtraction is not deﬁned.
However, to each Lie group G , an algebra g may be associated, given by the tangent space of the Lie group at the identity T e ( G ) .
The Lie algebra may be interpreted as a vector space of inﬁnitesimal generators of the group, a set of elements from which we can obtain the group G , by repeated application.
Exponential and logarithmic map.
The exponential map exp : g → G is a function mapping elements from the Lie algebra to the group.
For many transformation Lie groups of interest this map is surjective, and it is possible to deﬁne an inverse mapping; the logarithmic map which maps from the group to the algebra.
Semi-direct product groups.
In practice, we are often only interested in data deﬁned over R d , and hence in this paper only consider afﬁne Lie groups of the form G = R d (cid:111) H , where R d is the translation group in d dimensions and H is a transformation Lie group of interest.
Group action.
A group G may have an action on a given space X .
Given a group element g ∈ G and a set X , the group action T g deﬁnes what happens to any element of x ∈ X when we apply the transformation given by element g to it.
This action is given by:
[Equation]
such that for any two elements g, h ∈ G , we can combine their actions into a single action; T g,h =
T g ◦ T h .
To avoid clutter, we write the action T g ( x ) as g · x .
Note that the action of a group G on domain X also extends to functions deﬁned on this domain.
The induced group action T g of an element g ∈ G maps a square integrable function f to a transformed function f (cid:48) under the action of group G .
For a given element of the transformed domain x (cid:48) ∈ X (cid:48) , the value of the transformed function f (cid:48) is obtained by applying the inverse transformation of g to the domain of the function f :
T g : f → f (cid:48) and for x (cid:48) ∈ T g ( X ) : f (cid:48) ( x (cid:48) )
= f ( g − 1 x (cid:48) )
Representations In practice, the object on which the group operates is often deﬁned as a vector space R.
We operate on 2d images, which can be seen as functions on R 2 .
This makes the object our group operates on a vector space, meaning the group actions (or elements) need to be constructed as linear transformations from this vector space onto itself.
Simply put, every group element g ∈ G needs to be represented in this vector space by a matrix.
We say that linear transformations L G → L 2 ( X ) g : L 2 → Lon some space X are a representation of a group G when they satisfy the group structure, meaning for g 1 , g 2 ∈ G, x ∈ X :
[Equation]
A representation may be seen as the action of a group on a space in matrix form.
Equivariance .
An operator is equivariant with respect to a group, if it commutes with the action of the group.
For an operator Φ : L 2 ( X ) →
L 2 ( Y [Equation]
[Equation]
A. D ERIVING SEPARABLE GROUP CONVOLUTIONS
If we set the kernel k to be separable, meaning we parameterise it by multiplying a kernel k H which is constant along the spatial domain and a kernel k R 2 which is constant along the group domain:
[Equation]
We can derive separable group convolutions as:
[Equation]
A. E XAMPLES OF DIFFERENT L IE GROUPS
We implemented models equivariant to three different groups: SE(2) , R 2 (cid:111) R + and Sim(2) .
In this section, we describe these groups in more detail, and give deﬁnitions for the logarithmic map required in obtaining G-CNNs equivariant to these groups (Bekkers, 2019).
The translation group R 2 The translation group in two dimensions R 2 , has group product and inverse for two elements g = x , g (cid:48)
=
x (cid:48) ∈ R 2 : g
[Equation]
(cid:48)
[Equation]
[Equation]
x (cid:48)
[Equation]
[Equation]
With logarithmic map:
[Equation]
The rotation group SO(2) The rotation group in two dimensions describes the set of continuous rotation transformations of the plane, and consists of all orthogonal matrices R with determinant 1. Its group product and inverse for two elements g = R θ , g (cid:48)
=
R (cid:48) θ ∈ SO(2) is given by:
[Equation]
With logarithmic map:
[Equation]
[Equation]
The dilation group R + The group of dilation transformations R + has a group product and inverse that, for two elements g
= s, g = s (cid:48)
∈ R + are given by:
[Equation]
[Equation]
The logarithmic map is given by:
[Equation]
The Special Euclidean group SE (2)
The Special Euclidean group in 2 dimensions describes the set of geometric transformations that are formed by combinations of rotations and translations in two dimensions.
Each group element can be parameterised by two variables θ and x , describing the rotation angle and translation vector, g
=
( θ, x ) ∈ G .
For two elements g, g (cid:48) ∈ SE (2) , the group product and inverse are given by:
[Equation]
As we can see, to combine the two elements g, g (cid:48)
we apply the action of the rotation part of g to the translation of g (cid:48) before we combine it with the translation of g , we combine elements semi-directly.
SE (2) is a semidirect product of the translation group R 2 and rotation group SO (2) .
We write this as
SE (2) =
R 2 (cid:111) SO (2) .
To simplify implementation, we separate the logarithmic map into logarithmic maps for SO(2) and R 2 in our implementation.
The dilation-translation group R 2 (cid:111) R + Another group of interest to our research topic is the translation-dilation group R 2 (cid:111) R + , the group of translation and dilation transformations.
Dilations occur frequently in natural images, in the form of scaling transformations of objects and scenes, as the distance between camera and object differs between images.
Group elements are parameterised by a scaling factor s and translation element x , g = ( x , s ) ∈ G .
For two elements g, g (cid:48) , group product and inverse are given by:
[Equation]
To simplify implementation, we separate the logarithmic map into logarithmic maps for R + and Rin our implementation.
The Similarity group Sim (2)
The similarity transformation group is the semi-direct product of the roto-translation group SE (2) and the isotropic scaling group R ∗ , and deﬁnes dilation-roto-translation transformations in two dimensions.
Each group element can be parameterised by three variables θ , s and x .
For two elements g, g (cid:48)
∈
Sim (2) the group product and inverse are given by:
[Equation]
In Eq.
25 we use the fact that the isotropic dilation group has no action on the rotation group.
This is a consequence of the fact that R + and SO (2) are both abelian groups, and may be taken in direct product to create the group of dilation-rotation transformations.
We again separate logarithmic maps by subgroup.
A. V ISUALISATIONS OF OPERATIONS IN ( SEPARABLE ) G-CNN S
To ease the reading experience of this work, in this section we provide some additional group theoretic perspective on regular CNNs, and additional visualisations and for a number of operations used in (separable) group convolutions.
CNNs from a group theoretic perspective We give a brief treatment of ordinary CNNs The ordinary convolution operation used in neural networks requires a deﬁnition of a convolution kernel k on R 2 , as we are modulating a signal f which itself lives on R 2 .
The kernel k is applied to f on every location in the input space R 2 to again yield a function over R 2 .
Intuitively, this is the same as saying (1) we transform the convolution kernel k under the action of every group element x ∈ R 2 , to obtain a set of kernels K =
{L x ( k ) | x ∈ R 2 } , and (2) apply the convolution operation on f using this set of transformed kernels K .
By tying the kernel weights used throughout the translation group, learned features are automatically generalised over spatial positions.
This intuition is visualised in Fig.
Lifting convolution In Fig.
10 we show a visualisation of the lifting convolution for the SE(2) group.
Group convolution In Fig.
11 we show a visualisation of the group convolution for the SE(2) group.
Separable group convolution kernel In Fig.
12 we show a visualisation of a separable group convolution kernel.
Separable group convolution In Fig.
13 we show a visualisation of the separable group convolution operation.
Deﬁning convolution kernels on Lie groups In Fig.
15 we show how we obtain a grid on the Lie group SO(3) / SO(2) by mapping from its algebra.
In Fig.
14 we show how we subsequently obtain a kernel on this group by deﬁning a SIREN on its algebra.
A. E XPRESSIVITY LIMITATIONS OF SEPARABLE GROUP CONVOLUTIONS
As discussed, separable group convolution kernels are strictly less expressive than their non-separable counterpart.
For example, in the case of the roto-translation group SE(2) , this would enable nonseparable group convolution kernels to learn to represent features that are built up of different spatial conﬁgurations at different orientations.
We draw up a simpliﬁed example illustrated in Fig.
Assume we have an elementary feature type e and a spatial kernel k with which we can recognise this feature type in its canonical pose θ 0 .
In our input f in , we have three instances of the feature e , one under the canonical pose θ and two under a 90 ◦ rotation.
Applying the lifting convolution using kernel k for the group G = R 2 (cid:111) C 4 of translations and 90 ◦ rotations yields a feature map f out deﬁned over G , with spatial feature maps for θ 0 , ..., θ 270 .
The spatial feature map f θ 0 out contains a response at a single spatial position.
In contrast, the feature map f θ 90 out contains a response at two spatial positions.
The spatial conﬁgurations for the feature maps along H are different.
A single conventional group convolution kernel could learn to recognise these distinct spatial conﬁgurations along the subgroup axis, whereas a separable group convolution kernel could not, since it simply repeats (a weighted version of) the same spatial kernel k R 2 along the group axis.
Although this reduction in expressivity could theoretically prove limiting in the application of GCNNs on vision tasks, our experiments show that in practice this rarely seems a problem, and may even help prevent overﬁtting.
A. K ERNEL SMOOTHNESS FOR GROUP CONVOLUTIONS ON CONTINUOUS GROUPS
As mentioned, using SIRENs we are able to explicitly control kernel smoothness.
We brieﬂy elaborate on the importance of kernel smoothness in G-CNNs.
In conventional CNNs, weights at distinct spatial locations are generally initialised independently.
Because the kernels are only transformed using discrete translation operations x ∈ Z 2 , translation equivariance is ensured by virtue of using the exact same weights values throughout all spatial locations.
In G-CNNs for continuous groups, the kernel is transformed under actions of a continuous transformation group of interest H to obtain equivariance.
However, in our convolution operation we are still using a discretised kernel; we are required to sample kernel values at different grid points for different elements h ∈ H .
We are no longer able to simply reuse the same weight values throughout the group as with regular CNNs.
To this end, we deﬁne our kernels in an analytical form, which we can trivially evaluate at arbitrary grid points.
Because our grid has a ﬁxed resolution, the kernels sampled from this analytical form are susceptible to aliasing effects; the analytical kernel function may exhibit higher frequencies than can be captured in the discretisation of the kernel.
We visualise the effects of aliasing in Fig.
In short, for continuous groups, the group action transforms a signal smoothly as the group is traversed.
To prevent discretisation artefacts, we want our kernel to exhibit the same smoothly transforming behaviour, hence we use SIRENs, as they offer explicit control over the smoothness of kernels in their analytical form.
B A DDITIONAL EXPERIMENTAL DETAILS B. A RCHITECTURAL DETAILS AND PARAMETERISATION
Model architecture As architecture for our experiments we use a simple ResNet model (He et al., 2016).
We use a single lifting convolution with 32 output channels, followed by two residual blocks of 2 group convolutions.
The ﬁrst block has 32 output channels, the second has 64 output channels.
After the ﬁrst residual block we apply max-pooling with kernel size 2 over the spatial dimensions of the feature map.
After the last residual block, we apply max pooling over remaining spatial and subgroup dimensions, followed by two linear layers with batchnorm and ReLU in between.
An overview is given in Fig.
Group convolution blocks and random sampling on the group In our residual blocks (He et al., 2016), we subsequently convolve the input to the block x in by two group convolutions, gconvand gconv 2 , yielding x out and apply elementwise addition with the input x in , followed by ReLU activation. When approximating the group convolution through random sampling, we must take care to deﬁne the input and output of the two group convolution layers on the same grid over the group as the skip-connection to ensure a well-deﬁned equivariant group convolution block. This may be done by adding a group shortcut layer which maps from the set of input elements on the group to the set of output elements.
We implement this as a group convolution with a 1 × 1 spatial extent. The group shortcut layer thus simultaneously serves as a channelwise projection from the input space of gconv 1 to the output space of gconv 2 and maps from the input grid on H of the ﬁrst group convolution to the output grid on H of the second group convolution. SIREN architecture and separable kernel parameterisation All our kernels are parameterised by a SIREN (Sitzmann et al., 2020).
In a SIREN, output y l for a layer l and input x l − 1 is deﬁned by:
[Equation]
[Table] In this equation, ω 0 acts as a multiplier for low dimensional frequencies found in the input domain (the grid of relative offsets on the group), which explicitly introduces higher frequencies, allowing the neural net to learn high-frequency functions (such as kernels).
We found a value for ω 0 of 10 to work well in all our experiments.
For the SIREN, we used an architecture of two hidden layers of 64 units.
In non-separable G-CNNs we have a single SIREN with a ﬁnal layer mapping to a vector R c in × c out .
In separable G-CNNs, we use two SIRENs, the ﬁrst mapping to a function the Lie algebra of the subgroup H ; R c in × c out , and the second mapping to R c out .
Formulating the kernel k for a group element g = ( x , h ) in terms of k H , k R 2 , input channel i and output channel j , and logarithmic map on H log H , we obtain:
[Equation]
Lastly, for H-separable Sim(2) -CNNs we use three SIRENs, the ﬁrst mapping to a function the Lie algebra of the subgroup R + ; R c in × c out , the second mapping SO(2) to R c out , and the third mapping Rto R c out . one mapping to SO(2) . Formulating the kernel k for a group element g =
(( x , θ ) , s ) in terms of k SO(2) , k R + , k R 2 , input channel i and output channel j , logarithmic maps log R + and log SO(2) on R + and SO(2) respectively we obtain: [Equation]
Model sizes In Tab. 4, we report the number of trainable parameters for each model conﬁguration.
Throughout our experiments, we kept the number of channels in all our conﬁgurations constant, to fairly compare the expressivity of the learned representations in non-separable and separable group convolutions.
This has as effect that the number of parameters in the separable implementations is larger than for non-separable implementations, due to our use of separate SIRENs to parameterise kernels over the different subgroups.
To ensure that the difference in number of trainable parameters does not inﬂuence the comparison between separable and non-separable group convolution layers, we explicitly chose to over-parameterise our SIREN architecture, as shown in an additional ablation in App.
C.3.
We only change SIREN hidden size when comparing our models to baselines proposed in related works as detailed in App.
B.2. B. E XPERIMENTAL DETAILS
Here, we list training regimes for all experiments.
We keep these as consistent as possible, as we are only interested in the effect of separating the group convolution operation, and in isolating the effect of incorporating equivariance into our models.
We list information on all datasets, and any dataset-speciﬁc model conﬁguration details we use in our experiments here.
Optimizer All architectures are trained with Adam optimisation (Kingma & Ba, 2014), and 1 e −weight decay.
All models are trained on a single Titan V.
Rotated MNIST
The 62.000 MNIST images (LeCun et al., 1998) are split into a training, validation and test set of 10.000, 2.000 and 50.000 images respectively, and randomly rotated to orientations between [0 , 2 π ) .
Note that in (Weiler et al., 2018b; Weiler & Cesa, 2019), the rotated MNIST dataset is augmented during training by transforming images with random continuous rotations.
We only use train-time augmentation for the state-of-the-art results obtained in Tab. 2, as detailed in App.
B.2.
All models trained on rotated MNIST, except for the state-of-the-art runs detailed in B.2, are trained for 200 epochs with a batch size of 128 and a learning rate of 1 · 10 − 4 .
Scaled MNIST
The 62.000 MNIST images are again split into a training, validation and test set of 10.000, 2.000 and 50.000 images respectively, but now randomly scaled by a factor [0.3, 1] and padded with zeros to retain the original resolution.
No data-augmentations of any kind are used in the experiments on scaled MNIST.
All models trained on scaled MNIST are trained for 200 epochs with a batch size of 128 and a learning rate of 1 · 10 − 4 .
Scaled rotated MNIST The 62.000 MNIST images are again split into a training, validation and test set of 10.000, 2.000 and 50.000 images respectively, but now randomly scaled by a factor [0.3, 1], padded with zeros to retain the original resolution and randomly rotated by orientations between [0 , 2 π ) .
For the experiments on scaled rotated MNIST we do not use data-augmentation of any kind.
All models trained on rotated scaled MNIST are trained for 200 epochs with a batch size of 128 and a learning rate of 1 · 10 − 4 .
CIFAR10 We evaluate our models on the CIFAR10 dataset, containing 62.000 32 × 32 color images in 10 balanced classes (Krizhevsky et al., 2009).
All models trained on CIFAR10 are trained forepochs with a batch size of 128 and a learning rate of 1 · 10 − 4 .
CIFAR100 We evaluate our models on the CIFAR100 dataset, containing 62.000 32 × 32 color images in 100 balanced classes (Krizhevsky et al., 2009).
All models trained on CIFAR10 are trained for 200 epochs with a batch size of 128 and a learning rate of 1 · 10 − 4 .
Galaxy
This dataset contains 21785 69x69 color images of galaxies divided into 10 unbalanced classes.
For Galaxy10, we limit batch size to 32 images, and scale learning rate accordingly to 2 .
5 · 10 − 5 , as suggested by Goyal et al.
Achieving SOTA on rotated MNIST To compare performance of our separable regular G-CNNs with previous work, we apply our implementation of SE(2) and Sim(2) -CNNs to achieve state of the art results on the rotated MNIST dataset, discussed in Sec. 5.
We make some slight adjustment to our architecture and training regime: we reduce the SIREN to a single hidden layer of 64 units. The convolution over SO(2) is approximated using random sampling, while the convolution over R + is approximated using a discretisation with a truncation of the scale group at s = √ 3 . We use a batch size of 64, and a learning rate of 1 · 10 − 4 and train for 300 epochs. To fairly assess the impact of equivariance, we train a number of models without continuous rotation augmentations.
To compare with previous SOTA, achieved by Weiler & Cesa (2019), we also train models with continuous rotation augmentations.
Weiler & Cesa (2019) does not specify performance results for models without train-time augmentations, therefore we also compare with the prior SOTA by Weiler et al.
(2018b), which trains models both with and without augmentations. We show results in Tab. 2. With SE(2) -CNNs we achieve best performance with a resolution of 20 elements over SO(2) . For the Sim(2) -CNNs, we achieve best performance with a resolution of 12 elements over SO(2) and 5 elements over R + .
These results show that without train-time augmentation Sim(2) -CNNs outperform the previous SOTA by Weiler & Cesa (2019), which was trained with continuous rotation augmentations.
This improvement further increases when we add train-time augmentation by continuous rotations, to a test error of 0.59%.
Achieving competitive performance on CIFAR10 To further compare the viability and performance of our separable approach to group convolutions, following Cohen & Welling (2016a)
we re-implement the All-CNN-C proposed by (Springenberg et al., 2014), using our separable group convolution layers as drop-in replacement for the regular convolution layers.
All experiments described in this section use this exact architecture.
In this experiment we looked to isolate the power of our separable group convolution layers in larger-scale models, which is why we deliberately chose a large but relatively simple architecture. We keep the number of channels constant with the original implementation by (Springenberg et al., 2014). Because of the way our kernels are parameterised, it is hard to make a direct comparison in model performance in terms of representation expressivity, while keeping the number of trainable parameters equal to the original model. Therefore, we train three conﬁgurations, with different
SIREN hidden sizes.
To obtain a model with approximately the same number of trainable parameters, for the ﬁrst conﬁguration we limit the hidden size of our SIRENs to 6 to arrive at 1.33m parameters, where the original model has 1.4m. Note that this limitation likely has considerable implications for the expressivity of the sampled group convolution kernels. To push our separable group convolution layers further, we also train a model conﬁguration with a SIREN hidden size of 5 units, to arrive at a model with 1.14m parameters (19% smaller than the original model by Springenberg et al. Lastly, to see how our layers would fare when unimpeded by limitations in SIREN size, we train a conﬁguration with a SIREN hidden size of 16 units.
Note that this model contains 3.22m parameters (representing an 128% increase in parameter count to the original model). All models are trained with a resolution of 8 elements randomly sampled over SO(2) and a discretisation of the scale group of 3 elements truncated at √ 3 . We train all of these conﬁgurations both with and without data augmentation by padding the original image by at most 6 pixels and randomly cropping to the original resolution, and random horizontal ﬂips. All models are trained for 300 epochs with a learning rate of 1 · 10 − 4 .
Results are shown in Tab. 3. We can see that for approximately equal parameter counts, our separable group convolution layers equivariant to Sim(2) outperform the original CNN baseline and the p 4 -G-CNN. Without data augmentation, our model with 6 hidden units is outperformed by the p 4 m -G-CNN which is also equivariant to reﬂections. With augmentation (containing reﬂection augmentations) our model outperforms all baseline models we compare to by a signiﬁcant margin.
The smaller conﬁguration with 5 units seems to limit kernel expressivity somewhat, although without data augmentation it still outperforms the translation equivariant original implementation of the
All-CNN-C. It seems data augmentation throws the model off in this particular conﬁguration.
The larger conﬁguration with 16 hidden units, both with and without data augmentation, signiﬁcantly outperforms all baselines and other conﬁgurations.
C A DDITIONAL EXPERIMENTS C. V ALIDATING MODEL INVARIANCE TO TRANSFORMATION GROUPS
Following Weiler et al.
(2018b), we empirically assess the level of model-invariance to rotation transformations, by training on upright MNIST, and subsequently assessing performance on a test set of MNIST images that have been transformed by a subgroup element h ∈ H .
Each model is trained on the MNIST training set containing 60,000 images, and tested on 10, transformed images.
Performance is tested for rotations over a range between [0 , 2 π ) , in 100 steps.
The group convolution is approximated through random sampling.
Results are shown in ﬁgures 19 and 20 for 1, 2 and 8 group elements for the separable and non-separable variants.
We can see that equivariance error is very similar for separable and non-separable variants, and greatly reduces with increased resolution over the group.
We do note that for 8 elements, the non-separable group convolution achieves slightly better performance.
C. S UPPORT OF THE SEPARABLE CONVOLUTION KERNELS OVER THE CHANNEL DIMENSIONS
As explained in Sec. 4, we can choose to either deﬁne both k H and k R 2 over the input channel dimension, or incorporate depthwise separability as well; additionally sharing the same spatial kernel k R 2 over the input channels as well as the group input elements.
In an additional ablation study we ﬁnd that at any resolution above | H | = 1 , additional depthwise separability outperforms deﬁning both kernels over the input channel dimensions on a ﬁxed parameter budget.
For this experiment on rotated MNIST, we use the same depthwise group separable SE(2) − CNN as in Sec. 5, with the architecture described in App.
B.1.
For the non-depthwise group separable
SE(2) − CNN, we use a single residual group convolution block of 48 channels instead of two of 32 and 64 channels respectively to arrive at approximately the same number of parameters (815k for the implementation without depthwise separability, 803k for the implementation with depthwise separability).
We train the models for 150 epochs, using adam optimisation, with a learning rate of 1 · 10 − 4 .
We repeat the same experiment for R 2 (cid:111) R + -CNNs on CIFAR10, for a resolution of 1, 2, and 4 elements.
Architectures and training regimes used for the non-depthwise group separable and depthwise group separable experiments are identical to above.
Results, shown in Fig.
21 and Fig. 22 for rotated MNIST and CIFAR10 respectively, highlight that additional depthwise separability outperforms non-depthwise group separable convolutions on a ﬁxed parameter budget, which is why we stick with additional depthwise separation of the spatial kernel, and name this approach the separable group convolution.
C. H OW DOES SIREN ARCHITECTURE INFLUENCE PERFORMANCE ?
As explained, we keep the architecture of our SIRENs constant over all experiments.
Because we use multiple SIRENs in separable architectures, and we want to isolate the effect of separability of the group convolution operation, we deliberately choose to over-parameterise the SIREN architectures, so as not to advantage the separable implementations by their increased parameter numbers.
To this end, we look for the largest size of hidden layer that does not negatively impact performance in the non-separable implementation.
We assess the impact of the size of the hidden layers in our SIREN on performance on rotated
MNIST for separable and non-separable SE(2) -CNN implementations.
We sample at a resolution over SO(2) of 8 elements.
Interestingly, results in ﬁgure 23 show that even with very small SIRENsa hidden size of 4 or 8 - performance of the non-separable implementations is remarkable.
Separable implementations clearly underperform for small SIREN sizes, leading us to hypothesize that the restriction in kernel expressivity due separability paired with under-parameterisation is detrimental to performance in these conﬁgurations.
For non-separable implementations, performance barely increases from 16 hidden units on, and starts to decrease after a size of 64 hidden units.
For this reason, we choose a size of 64 hidden units in all
SIRENs in our experiments.
We repeat the same experiment for R 2 (cid:111) R + -CNNs on CIFAR10, and visualise the results in Fig. Here, performance for very small SIREN conﬁgurations is less impressive, but we still see a similar saturation around a SIREN size of 64 hidden units.
C. C OMPARING ACTIVATION FUNCTIONS FOR CONTINUOUS CONVOLUTION KERNELS
To assess the impact of our use of SIRENs on the performance of our separable G-CNNs, we perform an additional ablation on rotated MNIST in which we compare with other activation functions.
We train separable SE(2) -CNNs with a resolution 8 group elements over SO(2) , and only vary activation functions in the MLP parameterising the convolution kernels.
The convolution on SO(2) is approximated through random sampling.
We look at other activation functions used in parameterising convolution kernels; ReLU (Wu et al., 2019), LeakyReLU
[Table]
and SiLU (Finzi et al., 2020), see Tab. 5.
These results conclusively show the power of sine as activation function, one of the reasons we chose to use SIRENs in our work. A CTIVATION S EPARABLE [Table]
C. R ANDOM SAMPLING OVER NON - COMPACT GROUPS
As explained in section 4, for non-compact groups we approximate the group convolution through a discretisation of the transformation group of interest.
Motivation for this decision is the fact that in non-compact groups, we have to deal with boundary effects of truncating the group, introducing equivariance error into the convolution operation.
For dilation, further motivation for approximation through discretisation is the loss of information that occurs in for example natural images, as a result of downscaling a signal on a ﬁxed resolution grid.
Random sampling over the dilation group would have as effect that the representation built by a group convolution layer contains different spatial resolutions of information at every sampling step.
This results in subsequent layers receiving noisy information at each iteration, which would impede model performance, with training being highly unstable for low resolutions over R + .
To empirically reinforce this hypothesis, we performed the same experiment with our R 2 (cid:111) R + -CNN on MNIST-scale described in section 5 with random sampling over the dilation group.
Results, shown in ﬁgure 25, highlight that indeed, random sampling over the dilation group leads to signiﬁcantly worse performance.
We also noticed great instability during the training process, seen in the variance of the results achieved with random sampling.
C. D O SEPARABLE GROUP CONVOLUTIONS REDUCE REDUNDANCY IN G-CNN S ?
As explained, convolving with separable group convolution kernels is analogous to a group convolution with a kernel which shares a reweighting of a single spatial kernel along the group axis.
This approach was motivated by redundancy observations in regular group convolution kernels along the group axis, seen in Fig.
To assess whether the separable group convolution solves these redundancy issues in G-CNNs, we may want to perform a similar test of the variability of our introduced separable implementation along the group axis.
Note that, if we were to reconstruct the full group convolution kernel k from k H and k 2 R and apply the same PCA variance test as used to obtain Fig.
2, we would ﬁnd that each group convolution could be fully explained by the ﬁrst principal component (corresponding to the reshared spatial kernel).
Instead, we measure the variability of our separable kernels by assessing the variance of k H , which lies along this axis.
Results are shown in ﬁgure 26 for the separable
SE(2) -CNN trained on Galaxy10 dataset with a resolution of 8 elements on SO(2) .
Here, we observe the variance increasing in all layers over the training process.
This indicates that indeed, separable group convolutions reduce parameter redundancy in G-CNNs.