Relaxing Equivariance Constraints with Non-stationary Continuous Filters
Abstract
Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example.
Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent.
The type of symmetry is typically fixed and has to be chosen in advance.
Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries.
In such cases, equivariance constraints can be overly restrictive.
In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping.
The proposed parameterisation can be thought of as a building block to allow adjustable symmetry structure in neural networks.
Compared to non-equivariant or strict-equivariant baselines, we experimentally verify that soft equivariance leads to improved performance in terms of test accuracy on CIFAR-10 and CIFAR-100 image classification tasks.
Introduction
Symmetric properties, such as equivariances and invariances, can be embedded into neural network architectures to provide inductive biases that leads to better data-efficiency and improved generalisation.
Convolutional layers are known to provide translation equivariance in simple Euclidean spaces, and recent works have allowed various extensions to more complex groups and domains.
However, symmetries are typically fixed, must be specified in advance and can not be adjusted.
Symmetries embedded in network architectures enforce a hard constraint on the functions a neural network can represent.
This can be an effective way to encode prior knowledge for problems that are inherently symmetric.
However, hard symmetry constraints can become prohibitive if a problem does not strictly follows symmetries.
For example, convolutional layers cannot encode potentially relevant absolute positional information, and ‘6’s and a ‘9’s become difficult to distinguish under rotation invariance.
Relaxed symmetry constraints can mitigate such symmetry misspecifications without completely losing the useful inductive biases that symmetries provide.
We propose to relax equivariance constraints by generalising the convolution operator with nonstationary filters that can depend on absolute group elements.
Unlike other approaches, the resulting layer that can efficiently interpolate between ( i ) non-invariant linear products akin to a fully-connected layer, ( ii ) strict group equivariant convolutions, and ( iii ) strict group invariant mappings (Fig. 1).
The importance of this work is twofold.
First of all, relaxable symmetry constraints can directly improve the performance in cases where strict symmetries are misspecified and result in an overly restrictive model class.
Secondly, automatically learning symmetry structure from data is an interesting problem.
Work in this field which often focuses on invariances [33, 4, 32, 28, 12], which are easier to parameterise than equivariances.
Parameterisations that allow smooth and adjustable symmetry constraints could extends such methods to layer-by-layer learnable equivariances. [Figure] [Figure] ~ ~  ~ ~  ~ ~ ~
[Figure] ~ [Figure]  Related Work
Group equivariance.
Equivariance constraints applied to layers provide a strong inductive bias that enforce transformations in the input to result in equivalent transformations in the output.
For compact groups it can be shown that this constraint naturally leads to (group) convolutions [15].
Many works have allowed for equivariances to groups other than translation, including continuous roto-translations [37, 35, 16], discrete roto-translations [5, 3], scale [36], and permutations [38] and non-Euclidean domains, such as spheres [6], points clouds [10] and graphs [26].
Symmetry misspecification and approximate equivariance.
Although symmetry constraints can be very effective in machine learning, they can become prohibitive if data does not exactly follow the enforced symmetry.
For instance, Liu et al.
[17] raised an ‘intriguing failing’ of convolutional neural networks by showing that they can not encode absolute coordinate information.
Adding explicit coordinate information was proposed as an ad-hoc solution to the problem, effectively breaking strict equivariance constraints.
To achieve the same result, Finzi et al.
[8] proposed to break equivariance of a convolutional layer by summing it with a non-equivariant fully-connected layer, requiring many additional parameters.
Alternatively, Romero and Lohit [23] consider a sparse local support around the group identity element to break equivariance.
This is effective for sparsely sampled subgroups, such as rotation, but becomes less practical for more densely sampled groups such as translation where the local support could lead to very sparse feature maps.
Lastly, Wang et al.
[34] propose a relaxed convolutional operator similar to our non-stationary approach, but require a low-rank factorisation which potentially limits the expressivity of the feature maps.
The same work briefly discusses a mathematical description without such factorisation similar to our proposal which is deemed “too large a trainable weight space to be practical”.
This is true for discrete convolutional kernels.
Instead we define our kernels as continuous kernels [25] parameterised with a finite number of parameters.
Consequently, our weight space becomes tractable and does not pose a problem at all.
Physics and filtering.
Symmetries play a central role in physics.
In [34] approximate equivariances are used to allow more robust models for dynamical systems.
In Noether networks [1] conservation laws are inferred by learning symmetries from data.
Our work could offer new insights in how symmetry constraints could be relaxed in such applications.
In geophysics and seismology, there are some interesting parallels between non-stationary filtering methods and approximate equivariances in machine learning.
In particular, [19] discusses generalisations of the convolution in the context of non-stationary filtering very similar to how we propose to relax equivariances in neural networks.
Automatic symmetry discovery.
An interesting problem is to automatically learn symmetry structure in neural networks architectures based on data.
For instance, Lorraine et al.
[18] learn data augmentations by differentiating the validation loss using the implicit function theorem and Zhou et al.
[39] showed how symmetry structure could be learned through a meta-learning outer loop.
In [4] and [8] symmetries are selected using a training loss directly, using an additional regulariser.
Symmetry discovery methods often focus on learning global invariances [33, 4, 28, 32, 12], which are easier to parameterise than layer-wise equivariances.
We propose a method to continuously relax equivariances, which can be useful to extend symmetry discovery methods to learn layer-wise equivariance.
Background 3.1 Groups
A Lie group provides a natural way to describe continuous symmetry, as it forms a continuous manifold of which underlying elements are equipped with a group structure.
Lie groups do not necessarily form a vector space.
However, to every Lie group G we can associate an underlying vector space called the Lie algebra g .
The Lie algebra corresponds to the tangent space of the group at the identity, has the same dimensionality as the group, and captures the local structure of the group.
Because the Lie algebra is a vector space, elements a ∈ g can be expanded in a basis in the Lie algebra a = (cid:80) dim( G ) i
= α
i A i with coefficients α ∈ R dim( G ) .
The exponential map exp : g → G maps elements from the Lie algebra to the Lie group.
We can also define a logarithm log : G → g map from the Lie group to the Lie algebra.
Such a choice always exists, but is not necessarily smooth or unique.
3.2 Equivariance and invariance
Equivariance is the property of a mapping such that transformations to the input result in equivalent transformations to the output.
If changes are invertible, they can be described as the action of a group G on some space X .
Formally, we say that a function h : X → X is equivariant to the group G if h ( g · x ) = g · h ( x ) for all g ∈ G, x ∈ X .
If the output of the function is completely independent to the action of the group G on the input, we say that the function is invariant to the group G .
Formally, a function h : X → X is invariant to the group G if h ( g · x )
= h ( x ) for all g ∈ G, x ∈ X .
3.3 The group convolution
Let f : G
→ R be an input signal and k θ : G → R be a convolutional kernel parameterised by θ .
The group convolution is defined as:
[Equation]
where we integrate with respect to the Haar measure of the group µ .
Group convolutional structure is not just a sufficient, but also a necessary condition for equivariance to the action of a compact group [15].
The regular convolution for translation equivariance is the special case where the group is the translation group G = T(n) ∼ = R n , and the group action is given by addition, i.e., v − 1 u = u − v .
The convolutional kernel k θ ( v − 1 u ) , which we will sometimes refer to as ‘filter’ for stylistic purposes, is called stationary because it can only depend on u and v through v − 1 u .
This invariance of the kernel is an important constraint and in fact a requirement for h to be equivariant .
Breaking the stationarity of the kernel allows us to relax the equivariant symmetry constraints, which is one of the central ideas of this paper.
3.4 Group lifting
The input and output signals of a neural network are typically not defined on the group G , but rather on an input and output space X and Y .
Using a group lifting and projection procedure [15, 7], we can define our model on the group and still apply it on the input and output space.
We define a lifting operator ↑ G to map input signals f : X → R to functions on the group f ↑ G : G → R .
Similarly, we define a projection operator that maps function on the group f ′ :
G → R to functions on the output space f ′ ↓ Y : Y → R .
Method 4.1 Non-stationary Integral Operator
The group convolution operation of Eq.
(1) is strictly equivariant due to the stationarity of the kernel.
That is, the kernel k only depends on relative group element v − 1 u .
For translation G =T(2) , the kernel only depends on relative coordinates ( u − v ) and not on the absolute position in the image.
To relax equivariance constraints, we let the kernel also depend on the absolute group element:
[Equation]
The main difference with the group convolution of Eq.
(1) is that the kernel k θ : G × G
→ R now has two input group elements.
We will refer to the first input argument v − 1 u ∈ G as the stationary component and the second input argument v ∈ G as the non-stationary component .
In case of translation, this change lets the kernels depend also on the absolute coordinate of the input image on which it is applied.
Consequently, h ( u ) does no longer describe the regular convolution, but rather a non-stationary integral operator, for which the convolution remains a limiting case.
For the purpose of this work, there is no meaningful difference between expressing the non-stationary component in the input domain k θ ( v − 1 u, v ) or the output domain k θ ( v − 1 u, u ) .
We will stick with the input domain.
One could also think of writing the kernel in an even more general form: k θ ( u, v ) .
This, as such, does not lead to a more expressive kernel in the sense that they can both represent the same class of functions (for a proof, see App. B.1).
Further, in this more general way of writing the non-stationary kernel it becomes harder to formulate a parameterisation that allows for controllable symmetry constraints, so we do not consider it.
The non-stationary integral operator of Section 4.1 can represents interesting special cases through an independence of k θ with respect to the first and second input arguments.
Remark 1 (Linear product / Fully-connected) .
If the kernel equals an impulse response in the stationary component multiplied by some function k ′ θ in the non-stationary component k θ ( v − 1 u, v ) =
δ ( v − 1 u ) k ′ θ ( v ) , then the operator h ( u ) =
( k θ • f )( u ) corresponds to a linear product with k ′ θ .
If we have a kernel k θ ( v − 1 u, v ) =
δ ( v − 1 u ) k ′ θ ( v ) , then we have that:
[Equation]
which is the linear product between k ′ θ ( u ) and input f ( u ) .
This is akin to an element-wise product continuously defined on G .
Optionally, this could be followed by an invariant layer to average pool activations.
If the operation is performed over channels and we regard these as output features, this corresponds to a fully-connected layer in the discrete case G =
Z n , with weights given by k ′ θ .
Remark 2 (Group equivariance / Convolution) .
If the kernel does not depend on the non-stationary component, then the operator h ( u ) =
( k θ • f )( u ) is equivalent to a strict G-equivariant convolution.
If the kernel only depends on the first stationary component, then k θ ( v − 1 u, v ) =
k ′ θ
v − 1 u ) , and:
[Equation]
which equals the group convolution which is known to be strictly group equivariant.
Remark 3 (Group Invariance / Pooling) .
If the kernel is independent of both stationary and nonstationary components, the operator h ( u ) =
( k θ • f )( u ) is strictly G-invariant.
If the kernel does not depend on both inputs, it must be constant k θ ( v − 1 u, u )
= C , and we obtain:
[Equation]
which is a scaled average ‘global-pooling’ over group actions and leads to strict group invariance.
We obtain partial invariance or ‘local-pooling’ if the kernel is only non-zero constant in some local support of the stationary domain.
4.2 Parameterising the kernel
Lie algebra basis.
In Eq.
(2), we have proposed a non-stationary integral operator with a kernel k θ : G × G → R that takes two input arguments: the stationary group element v − 1 u and a nonstationary group element v .
We can parameterise the kernel k θ in terms of real vector spaces, which are more practical to work with, by defining elements in an explicit Lie algebra basis.
To do so, we first define a logarithm function log and express the kernel k
= ˆ k (log g, log h ) in terms of a kernel in the Lie algebra
ˆ k : g × g
→ R , which always form vector spaces.
For rotation G =SO(2) , the exponential map exp sending Lie algebra elements (on a line) to Lie group elements (on a circle) is not left invertible, but is right invertible, as a function.
As the right inverse is not unique, we choose to use the principal log sending Lie group elements to Lie algebra elements closest to the identity.
In general, such a choice for the logarithm always exists but is not necessarily unique or continuous.
We choose a vector space basis { A i } dim( G ) i = to express Lie algebra elements a ∈ g in terms of coefficients α ∈ R dim( G ) that relate to Lie group elements g ∈ G by:
[Equation]
Through this construction, each group element g corresponds through a choice of log to a unique Lie algebra element a which can be implemented as a real vector α .
Functions in the Lie algebra.
We parameterise the proposed non-stationary kernel k θ : G × G → R , which takes two input arguments, in the Lie algebra kernel
ˆ k : g × g → R acting on a product space of two vector spaces.
We refer to the first vector space related to the stationary component as the filter space , as it corresponds to space in which filters of conventional convolutions are defined.
And refer to the second vector space of the non-stationary component as the domain space , because elements directly relate to absolute group elements in the input domain.
In Section 4.1 we showed that an independence of the kernel with respect to the first and second input argument correspond to strict equivariant and invariant symmetry constraints.
We would, therefore, like to choose ˆ k such that we can control the dependence on the stationary and nonstationary components, as this will allow for explicit control over symmetry constraints.
To do so, we parameterise ˆ k using a set of random Fourier features [22] with a frequency parameter ω for each dimension.
The frequency parameter gives us control over the spectral properties of the function in filter space and domain space.
But crucially, a frequency of ω =0 results in a constant infinite lengthscale and therefore a function ˆ k that is independent to the associated input dimension.
This mechanic allows us to select the special cases discussed in Section 4.1 and interpolate between them!
Together, we have a frequency parameter ω ∈ R dim( G ) that controls spectral properties in filter space and a frequency parameter ω ′ ∈ R dim( G ) to control spectral properties in domain space.
We propose a weight-space parameterisation that allows for explicit control over the symmetry properties of the layer through frequency parameters ω and ω ′ .
Weight-space implementation.
We choose to parameterise our kernel in a finite D -dimensional random Fourier features (RFF) [22, 30] basis γ ω : g → R 2 D :
[Equation]
embedding Lie algebra elements a ∈ g as real vectors using their coefficients α ∈ R dim( G ) where the frequency can be explicitly controlled by frequency parameter ω ∈ R dim( G ) .
Values of W ∈
R D × dim( G ) are randomly initialised and can be kept fixed.
We parameterise the Lie algebra kernel
ˆ k : g × g → R with a neural network NN θ : R 4D → R taking concatenated Fourier features as input:
[Equation]
where γ ω ( a v − 1 u ) are the Fourier features on the filter space and γ ω ′ ( a v ) are the Fourier features on the domain space, respectively parameterised by ω and ω ′ frequency parameter vectors.
Directly representing the kernel with an MLP on α without Fourier features will likely hamper performance as MLPs are known to suffer from ‘spectral bias’ [2, 21], making it difficult to encode
[Figure]
high-frequency functions.
Sinusoidal activations in random Fourier features [22, 30] and SIRENs [29] alleviate this issue and have been found suitable encodings of positional information in neural networks [31], including (place-coded) continuous kernels in neural networks [25, 24].
Intuition behind frequency parameters.
The frequency parameters ω and ω ′ control the frequencies in the respective ‘filter space’ and ‘domain space’.
Specific values of ω and ω ′ correspond to a layer h that exactly is or interpolates between special cases discussed in Section 4.1.
Most notably, for ω ′ =
the layer becomes a strictly equivariant convolution and for both ω = ω ′ = {+
+}
the layer performs invariant pooling.
To understand why this is the case, it can be helpful to consider how different frequencies affect the kernel.
A Fourier feature with zero frequency parameter is constant γ 0 , by definition.
For ω ′ = 0 we therefore have a constant Fourier feature for the non-stationary component γ ω ′ ( a v )= γ 0 and a kernel that is solely defined in the filter space.
If both ω = ω ′ = 0 , we also have a constant Fourier feature for the non-stationary component γ ω ( a v − 1 u )=
γ 0 and a kernel that is constant everywhere, which results in invariant pooling.
Positive values for ω ′ introduce non-stationarity that effectively relaxes the strict equivariance constraints of the convolution at ω ′
= 0 . Lastly, note that individual scalars in ω ′ correspond to particular subgroups and we can relax equivariance constraints of specific subgroups by letting ω ′ be greater than zero in the associated dimension.
Example for translation group Let us consider relaxing equivariance of the translation group
G = T (2) , as an example.
Since dim(T(2))
= 2 , we have a two dimensional filter space and domain space and therefore also two-dimensional filter space frequencies ω
= [ ω x , ω y ] T ∈ R 2 and two domain space frequencies ω ′
=
[ ω ′ x , ω ′ y
T ∈ R 2 .
If ω ′ =[0 , 0] T , the layer reduces to a translationally equivariant convolutional layer.
In other words, for any value of ω or weights θ the kernel remains stationary (as we have ω ′ = 0 ). In this case the layer equals a convolution where the same filter is applied to all locations in the input.
Higher values of ω , i.e. ω x , ω y > 0 , correspond to more rapid changes in x- and y-direction in filter space, but it remains the same filter signal that is applied independent of the input location.
If, instead, we let the components of the domain space frequencies ω ′ become non-zero, i.e. ω ′ x , ω ′ y > 0 , then the kernel values can also depend on the absolute coordinate in the input image.
Higher ω ′ result in a more rapidly changing kernel relative to different absolute locations in the input image.
This non-stationarity breaks equivariance constraints in a continuous way.
For sufficiently high ω ′ , the function can become akin to a linear product that has enough variance to place an independent weight on each pixel location.
In practice, this would also require a sufficiently flexible NN θ to represent such function. An illustration of the different symmetry modes for translation G =T(2) is shown in Fig.
S t r i c t e q u
i v a r i a n c e ' =v =
° v = 36° v = 72° v = 108° v = 144° v = 180° v = 216° v = 252° v = 288° v = 324° v = 360°
S o f t e q u
i v a r i a n c e ' >
s a m p l e d
Implementation
Filter parameterisation and frequencies All models, including the T(2)-equivariant and strict-SE(2) equivariant baselines, follow the same architecture and settings as prior work by [23].
Importantly, all models use continuous kernels that are parameterised using Fourier basis features, which is common in literature. As all baselines use the same architectures, training settings and Fourier representations for the kernels, we can attribute gains in performance solely to the relaxation of equivariance. To select ω , we use the same frequencies as used to initialise filter frequencies in [23].
To select ω ′ for the models with relaxed equviariance, we use cross-validation and limit the search space by considering equal frequencies along translational axes, ie.
ω ′ x = ω ′ y , and share frequencies across layers. This scales the frequencies along the x- and y-axis equally in the Fourier basis, but does not constrain the kernel itself to be isotropic. Independent frequencies per layer is interesting future work.
Lastly, to ensure the kernel is smooth in SO(2) , we let ω r , ω ′ r and associated entries in W be integer valued, similar to circular harmonics.
An overview of the components in ω and ω ′ for different groups can be found in Table 1.
Sampling.
The integral in Eq.
(2) is not tractable in closed form and we therefore choose to approximate it by using samples, similar to Finzi et al.
We deterministically evaluate group elements in T(2) that correspond to (lifted) pixel locations and randomly sample 4, 8 or 16 group elements uniformly over SO(2) .
For [23], the uniform distribution over SO(2) covers the entire group at the start of training but has an adjustable range that is learned by early-stopping on the validation loss.
All models used in the experiments use continuously parameterised kernels.
Locally-supported in filter space.
Similar to conventional convolutional networks, we only define the kernel in a local subset S of the filter space, forcing the kernel to be zero k θ =0 outside this local support.
Doing so, makes the approximation of the integral in Eq.
(2) much cheaper as we only need to sample the group elements within the support.
Another consequence is that we no longer obtain strictly invariant global-pooling of Eq.
Instead, a kernel that is only a constant non-zero value in a local support represents a local-pooling layer, where the pool size equals the support size.
In the experiments we let S be a disk in T(2) with a diameter equal to 7 pixels in the input domain X .
[Table]
[Table]
Experiments 6.1 Verifying soft-equivariance using symmetry misspecification toy example
To verify that relaxed equivariance allows for less restrictive functions compared to strict equivariance, we repeat the MNIST6-180 problem of [23] and include our model.
This toy problem is designed such that it can not be solved under strict symmetry constraints.
It consists of the 6’s in the MNIST dataset with half of the dataset randomly rotated by 180 ◦ degrees.
The task is to determine whether a sample has been rotated, i.e., a binary classification problem between ‘6’s and ‘9’s.
In Table 3, we compare a rotation invariant G-CNN model that consists of multiple strictly equivariant layers followed by group pooling and compare with [23] and our model.
In line with our expectations, we find that in this artificial set-up the relaxed ResNet model quickly converges to 100% test accuracy.
The model with strict rotation symmetry is unable to distinguish between ‘6’s and ‘9’s and can thus not improve over a simple coin-flip with 50% accuracy on average.
[Table]
6.2 Image classification experiments
We assess the effectivity of relaxing equivariance constraints for translation and rotation groups T(2) ,
SO(2) and SE(2) on CIFAR-10 and CIFAR-100 image classification tasks.
As strict symmetries can lead to misspecification, we hypothesise that relaxed equivariance can improve the model performance on these tasks.
As baselines, we use a strictly equivariant G-CNNs and a Partial G-CNNs [23].
The difference in model sizes arising from different kernel input dimensions are neglectable compared to the total number of model parameters for the used architecture.
In Fig.
4, we plot performance on the validation set for different values of frequency parameter ω ′ and list final test accuracies in
Table 2. From Table 2, we can see that our method outperforms the baseline on both datasets across all settings.
This holds both with and without data augmentation, showing that the benefit of relaxed equivariance does not disappear when using augmentations.
On T(2) , we find that the improvement is negligible small and it would be interesting to see whether this gap becomes larger with increased number of basis functions or other choices of NN θ .
We observe particularly large improvements in test accuracy for soft- SO(2) on CIFAR-10 and CIFAR-100.
On CIFAR-100, relaxing equivariance improves test accuracy of the non-invariant baseline by 6 percentage points when using augmentation and improves by 8 percentage, without augmentation.
[Figure] To further assess the effect of equivariance relaxation has on model performance [Figure] we can examine the impact different values for of non-stationary frequency parameter ω ′ on model performance.
In Fig.
4, we show validation accuracy for different ω ′ averaged over three seeds with error bars corresponding to standard error ( σ √ 3 ) and show final test accuracy for model with best validation score.
The scale of the found optimal frequency parameters should not directly be interpret as they also depend on the chosen basis, parameterisation and initialisation of NN θ .
Yet, we still observe that some relaxation of equivariance with frequency parameter ω ′ ̸ = 0 leads to better results when compared to strict equivariant ω ′ = 0 setting.
That relaxing equivariance results in higher test accuracy in these cases indicates that strict equivariance could be misspecified for these problems to which our approach offers a solution.
Conclusion
In this work, we have proposed a generalisation of the group convolution that allows for smooth parameter-efficient relaxation of otherwise strict symmetry constraints.
The main idea is to use a nonstationary kernel that also depends on the absolute input group element, breaking strict equivariance constraints.
Moreover, we show that we can obtain explicit control over the symmetry constraints through tunable frequency parameters, by representing group elements in a Fourier feature space.
Although we have demonstrated a method to adjust equivariances within layers, we have so far used cross-validation in order to set their optimal value.
There have been some recent approaches to automatically learn symmetries in neural networks from training data [4, 28, 32, 12], but typically focus on learning invariances which are easier to parameterise.
The method proposed in this paper could offer ways to parameterise networks with learnable equivariance in such methods.
This paves the way for automatic layer-by-layer symmetry discovery as part of the training procedure.
By relaxing equivariance properties we can leverage useful inductive bias that symmetries can provide, while preventing possible symmetry misspecification if data does not fully obey an embedded symmetry.
Experimentally, we have demonstrated that neural network layers with adjustable symmetry constraints can improve test accuracy on CIFAR-10 and CIFAR-100 image classification tasks, outperforming strict-equivariant models with up to 8 percentage points in test accuracy on
CIFAR-100.
More importantly, we hope that the parameterisation can be a generally useful building block in machine learning applications that require smooth relaxations of symmetry constraints.
A Training details A.1 Network architecture
In all experiments, we use the same architecture and training settings as prior work by [23].
The used architectures have a similar number of parameters within a 4% difference (see Table 4).
The architecture consists of a simple ResNet model [11].
A complete overview of the used architecture can be found in Sec. 5 and Fig.
3 of [23].
The network consists of a lifting layer, followed by two residual blocks with spatial max-pooling using a kernel size 2 after each group equivariant layer. After the last block, max pooling is applied over spatial and subgroup dimensions, followed by two linear layers with batch norm [13] and ReLU non-linearities.
Continuously parameterising convolutional kernels in the context of neural networks was first proposed in [27], but requires less flexible isotropic kernels.
In [9], continuously parameterising kernels with more flexible MLPs with Swish activation functions was proposed. Later works [25, 23] showed that SIRENs [29], which have a random Fourier feature basis by effectively replacing Swish activation functions with sinusoidal activations, greatly improved performance in these settings. It has been shown that random Fourier features approximate the feature basis of the radial basis function [22] (exact in the infinite-width limit), and can provide an effective spectral bias to learn continuous signals [31].
From a deep learning perspective, this encompasses kernels parameterised by small MLPs with sinusoidal activation functions in the first layer, such as SIRENs [29]. We use a shallow MLP with 32 hidden units and cosine activation functions for NN ( · ) , such that the continuous kernels are the same 3-layer SIREN as used in [23]. An ablation study of different activation functions for the architecture used in this study can be found in Table 4 of Appendix F in [23]. For a more complete overview, we refer to [25] for an ablation study for regular convolutional kernels, [23] for group convolutional kernels and partial equivariant kernels, and [14] for (separable) for group convolutional kernels.
We use the same Fourier feature parameterisation of continuous kernels in all models. A.3 Training settings
For the CIFAR-10 and CIFAR-100 datasets, we use the default train, validation and test split.
We use a mini-batch size of 64 in all experiments. We optimise for 300 epochs with Adam ( β 1 =0 .
9 , β 2 =0 .
with a learning rate of 0 . 001 , cosine annealed to zero with 5 epochs of linear warm-up and a weight decay of 0.0001.
For the strict and partial equivariant baselines, we use the same architecture as used in [23].
As can be seen from Table 4 reporting exact parameter counts, the models that were compared have approximately the same parameter counts. [Table] [Equation] =
[Equation] [Table] A.5 Runtimes
In Table 5, we report training and inference runtimes of our own implementation. Do note that these results are hardware and implementation specific. We used pytorch [20], which allows models to be efficiently run on an NVIDIA RTX 3090 24gb GPU. Adding strict rotational equivariance ( SE(2)
= T(2) ⋊ SO(2) ) is slower than only using T(2) equivariance, as it requires an extra dimension in the feature maps for samples of the rotation subgroup. Relaxing equivariance T(2) ⋊
soft- SO(2) , on the other hand, only requires an extra dependency in the kernel and does not come with an added computational cost. In this case, we show that relaxing equivariance can increase performance without an increase in train or inference time.
We hypothesize that further engineering efforts, such as dedicated low-level CUDA implementations for (relaxed) convolutional operations, could further improve runtime performance.
[Table]
B Mathematical details B.1 Relaxed kernel inputs ( v − 1 u, u ) and ( v − 1 u, v ) equally expressive as more general ( u, v )
To relax the convolution operator, we let the kernel not only depend on v − 1 u but on also directly on the input group element v or output group element u : k 1 ( v − 1 u, u ) or k 2 ( v − 1 u, v ) .
We choose this form, as it allows us to parameterise the kernel in a way that can efficiently interpolate to convolutional kernels by letting it solely depend on the first argument ( v − 1 u ) .
We could consider writing our relaxed kernel in a more general form k ( u, v ) .
However, upon a change-of-variables, this general form is just as expressive as the relaxed kernel in the sense that they can describe the same class of functions.
To show this, we consider the reparameterisation k
= k 1 ◦ f 1 or k = k 2 ◦ f 2 .
It suffices to show that bijections exist f 1 : ( a, b ) (cid:55)→ ( b − 1 a, a ) and f 2 : ( a, b ) (cid:55)→ ( b − 1 a, b ) between the forms: ( v − 1 u, u ) f 1 ⇆ f − 1( u, v ) f 2 ⇄ f − 1( v − 1 u, v )
If we choose inverse f − 1: ( a, b ) (cid:55)→ ( b, ba − 1 ) and f − 1: ( a, b ) (cid:55)→ ( ba, b ) , it follows that
[Equation]
This confirms the bijection and we conclude that kernels that depend on ( u, v ) do, as such, not represent a broader function class than kernels that depend on ( v − 1 u, u ) or ( v − 1 u, v ) .
Observe that no such bijections can be found to ( v − 1 u ) , as convolutional kernels are less expressive.