Sparse Convolutions on Lie Groups
: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Convolutional neural networks have proven very successful for a wide range of modelling tasks.
Convolutional layers embed equivariance to discrete translations into the architectural structure of neural networks.
Extensions have generalised continuous Lie groups beyond translation, such as rotation, scale or more complex symmetries.
Other works have allowed for relaxed equivariance constraints to better model data that does not fully respect symmetries while still leveraging on useful inductive biases that equivariances provide.
How continuous convolutional filters on Lie groups can best be parameterised remains an open question.
To parameterise sufficiently flexible continuous filters, small MLP hypernetworks are often used in practice.
Although this works, it typically introduces many additional model parameters.
To be more parameter-efficient, we propose an alternative approach and define continuous filters with a small finite set of basis functions through anchor points.
Regular convolutional layers appear as a special case, allowing for practical conversion between regular filters and our basis function filter formulation, at equal memory complexity.
The basis function filters enable efficient construction of neural network architectures with equivariance or relaxed equivariance, outperforming baselines on vision classification tasks.
1. Introduction
Symmetries provide powerful inductive bias in neural networks, often improving data efficiency and generalisation performance.
Convolutional filters are an apparent example encoding equivariance to translations through effective weight-sharing.
Extensions to other groups exist, but equivariance to Lie groups can be hard to implement efficiently due to their continuous nature.
Similar difficulties arise in relaxations of equivariance, which can be useful to model data that only partially respects symmetries.
In practice, continuous filters are often parameterised using shallow MLPs or hypernetworks, which take coordinates (or group elements) as input and output a filter response.
Although MLPs can approximate any function in theory, this might require adding a large set of parameters.
To prevent having to add many hypernetwork parameters, we propose a basis function approach and parameterise continuous kernels from a small finite set of anchor points.
Anchor points are the filter values at specific locations stored in memory and induce values at all other (continuous) locations.
In theory this yields an equally flexible model class given enough anchor points, but more crucially, we empirically find that we can represent useful continuous filters with just a few anchor points.
We demonstrate that this yields more efficient representations of regular convolutional kernels, group convolutional filters and relaxed convolutional filters.
Unlike prior work, this allows us to create group equivariant and relaxed equivariant versions of large commonly used neural network architectures, such as deep ResNets, with equivalent parameter counts.
On CIFAR-10 and CIFAR-100 image classification tasks, we show that the approach is much more memory efficient and outperforms our baselines in terms of parameter efficiency and classification performance.
2. Related Work Continuous convolutional kernels
Continuous convolutional kernels have been proposed in Schütt et al.
(2017) to model quantum interactions, but relied on isotropic kernels limiting flexibility.
Wang et al.
; Ma et al.
(2022) proposed more expressive MLP hypernetworks.
Rahimi et al.
(2007) show that the exponential basis function can be approximated (and is exact in the infinite width limit) with a finite number of random Fourier basis functions.
Parameterising kernels in such feature spaces using MLPs with sinusoidal activation functions, such as positional encodings (Vaswani et al., 2017) and SIRENs (Sitzmann et al., 2020), have been found effective to fit high-frequency signals, such as images (Tancik et al., 2020) and neural rendings (Mildenhall et al., 2020).
In the context of (group-) convolutions such parameterisations are often used to parameterise continuous kernels, and found to be very effective in practice (Romero et al., 2021; Knigge et al., 2022).
Yet, representing filters with MLPs often requires many additional model parameters compared to discrete filter grids used of classical CNNs.
We overcome this limitation and effectively parameterise continuous kernels by regressing a small set of anchor points.
Group equivariance
Classical convolutional layers embed equivariance to discrete translations.
This was generalised to groups in Cohen and Welling (2016) which demonstrated equivariance to discrete 90 degree rotations and flips.
Further extensions allow convolutions on other groups, such as continuous roto-translations (Worrall et al., 2017;
Weiler et al., 2018; Weiler and Cesa, 2019; Kondor et al., 2018), scale (Worrall and Welling, 2019), and permutations (Zaheer et al., 2017) and non-Euclidean domains, such as spheres (Cohen et al., 2018), point clouds (Fuchs et al., 2020) and graphs Satorras et al.
Our method is very similar to Bekkers (2019) proposing to parameterise filters on Lie groups using B-splines.
A difference is that we are not limited to B-splines but generalise to arbitrary basis functions and consider filters in the context of relaxed equivariance.
Relaxed equivariance
Equivariance and invariance symmetry constraints have proven very useful in data modelling.
However, hard symmetry constraints can be overly restrictive if data does not fully respect enforced symmetries (van der Ouderaa and van der Wilk, 2021).
In digit-recognition, some robustness to rotational perturbation is desirable, whereas ‘6’s and ‘9’s may become indistinguishable under full rotational invariance.
Likewise, Wang et al.
(2022) improve real-world dynamics data modeling through relaxed symmetry constraints.
Some efforts have been made to parameterise approximate equivariance (Wang et al., 2022; Romero and Lohit, 2021).
A recent generalisation of the convolution proposed by van der Ouderaa et al.
(2022) allows for relaxed equivariance with explicit control over the amount of equivariance.
We utilise this generalisation and demonstrate how our more efficient filter parameterisation can be effectively used to parameterise filters in this setting.
Symmetry discovery
Another interesting consequence of relaxing equivariance constraints, is that symmetry structure becomes differentiable and can thus be optimised with gradients.
Recent work on automatic symmetry discovery (Benton et al., 2020; van der Ouderaa and van der Wilk, 2021; Immer et al., 2022) focus on invariances or data augmentation, which are easier to parameterise than layer-wise equivariance.
We offer an efficient way to parameterise filters for relaxed equivariance, which may allow extensions of symmetry discovery methods to learn layer-wise equivariances.
3. Background Lie groups and Lie algebra
A Lie group is a group with the structure of a differentiable manifold.
To each Lie group G we can associate a Lie algebra g : the tangent space of the Lie group at the identity element, capturing local structure of the Lie group.
The Lie algebra always forms a vector space.
The exponential map exp maps elements from the Lie algebra to the Lie group.
As the Lie algebra is always a vector space, its elements a ∈ g can be expanded in a basis a = (cid:80) dim( G ) i
= α
i A
i , with coefficients α ∈ R dim( G ) .
We define the operator a ∨ := α ∈ R D to encode Lie algebra elements as real vectors in the chosen basis.
The exponential map exp : g → G maps elements from the Lie algebra to the Lie group.
We can also define a logarithm map log : G → g that maps elements in the Lie group to the Lie algebra.
Such a choice always exists, but it is not always smooth or unique.
Equivariance and Invariance
Equivariance is the property of a mapping such that transformations to the input result in equivalent transformations in the output.
If changes are invertible, they can be described as the action of a group G on some space X .
Formally, we say that a function h : X → X is equivariant to the group G if h ( g · x ) =
g · h ( x ) for all g ∈ G , x ∈ X .
If the output of the function is independent to the action of the group G on the input, we say that the function is invariant to group G .
Formally, a function h : X → X is invariant to group G
if h ( g · x ) = h ( x ) for all g ∈ G , x ∈ X .
Group convolutions
Classical convolutions on real space encode translational equivariance.
The group convolution y encodes equivariance to other groups (Cohen and Welling, 2016):
[Equation]
with convolutional kernel f : G → R , input x : G → R , and output y : G → R all functions on group G with Haar measure µ G .
For compact groups, the equivariance constraint is both a sufficient and necessary condition for a linear map to be a convolution (Kondor and Trivedi, 2018).
In other words, no equivariance without convolution (and vice versa).
In practice, we can approximate the integral with ˆ y using N MC Monte Carlo samples (Finzi et al., 2020).
Relaxed equivariance
If modeled data does not exactly follow a symmetry, strictly enforcing it is misspecified and approximate equivariance may be more desirable.
We can quantify the amount of equivariance by the equivariance error (Wang et al., 2022):
[Equation]
for a function f : G → R .
In case of strict equivariance, we have that || f || EE =0 and for relaxed equivariance we typically want || f || EE <
ϵ for some ϵ .
To relax equivariance, we use the construction proposed by van der Ouderaa et al.
(2022) which uses filter functions (cid:102)
f G : G × G → R in Eq.
(8) that not only depends on the relative group element v − 1 u , but also on an additional absolute input group element u .
This additional argument breaks strict equivariance and the amount of equivariance can be controlled through this dependence.
[Figure]
4. Continuous filters with basis functions Classical filters
Filters in classical discrete convolution f : Z 2 → R map discrete 2dimensional filter space Z 2 to the reals.
Filter are typically only defined on a small local squared support S ⊂
Z 2 forming a grid which pixel coordinates can be summarised as row vectors in matrix S ∈ Z S 2 × 2 .
Filter weights can be stored in a square matrix U ∈ R S × S .
As we treat filters as functions, we define it as a function f ( (cid:2) xx 2 (cid:3) T ) =
[ U ] ( x 1 ,x 2 ) that indexes and returns weights within the support [ x 1 , x 2 ] T ∈ S , and 0 elsewhere.
Instead of filters in discrete space Z 2 , we will now propose how to construct filters on continuous domains R 2 .
Continuous filters with basis functions
Our goal is to parameterise smooth filter functions on f : R 2 → R that can be efficiently parameterised with a finite amount of memory.
To do so, we consider a finite amount of P anchor points, summarised in matrix
Z ∈ R P × 2 , to which we associate values listed in u ∈ R P .
In addition, we choose a basis function ϕ : R 2 × R 2 → R and scalar noise parameter σ 2 , and induce a continuous function f from this finite set ( Z , u ) using simple basis function regression.
This defines a continuous filter f ( x ) , which closed-form formula is given by Eq.
(3), that can be evaluated at any arbitrary coordinate x ∈ R 2 .
The memory required to store the filter is no longer a function of filter size in O ( S 2 ) , but rather of the number of anchor points O ( P ) .
Converting from classical filters
The proposed basis function formulation differs from that of classical CNN filters as it enables continuous filter evaluations anywhere in R 2 .
However, we can treat a classical filter as a special case of our basis function filter allowing for straightforward conversion between the two.
To convert a classical discrete filters f classic : Z 2 → R to a continuous basis function filters f : R 2 → R we can set our anchor point coordinates equal to the original sampling grid Z = S and set the associated values equal to filter values u = vec ( U ) , with zero noise σ =0 .
This creates a basis function filter that has the exact same output at the original sampling grid, ie.
f ( x )= f classic ( x ) ∀ x ∈
Z 2 .
In this case, the number of anchor points equals the number of pixels in sampling grid P = S 2 , and the resulting memory complexity therefore remains equivalent.
Transferring weights from regular convolutional layer to implementations of basis function filters can be of practical interest.
Inference
The filter response at any location in x ∈ R 2 can be computed in closed-form:
[Equation]
where ϕ xZ :
= (cid:2) ϕ
( x , z 1 ) ϕ
( x , z 2 ) · · · ϕ ( x , z P ) (cid:3) T ∈ R P and Φ
zz ∈ R P
×
P is defined by [ Φ ZZ ] ij :
= ϕ
( z i , z j ) with z i being the i ’th row vector of Z .
Train
The inference procedure f Z , u ( x ) is differentiable w.r.t.
basis function locations Z , basis function values u , and possible hyper-parameters in ϕ .
We can train basis function layers just like other neural network layers, treating Z and u as learnable parameters and optimise them together with model parameters using back-propagation.
Sparsify
Optionally, we can further reduce the number of anchor points to P ′ <
P to a new set of locations Z ′ ∈ R P ′ × 2 .
We pick Z ′ with some scheme, such as random sampling or (row-wise) subset of Z , and find new basis function values u ′ ∈ R P
′ that minimise the squared distance between new predictions f
Z ′ , u ′ ( z i ) and original basis functions:
[Equation]
We can find the following closed-form minimum norm solution for new anchor point values:
[Equation]
with A =
( Φ ZZ ′ ( Φ Z ′ Z ′
+ σ 2 I ) − 1 , and
[Equation]
, where Φ ZZ ′ :
= ϕ ( Z , Z ′
) ∈ R P × P ′ , and Φ
[Equation]
Instead of only taking the arg min w.r.t.
u ′ , an even tighter fit may be obtained by optimizing both variables Z ′ and u ′ .
Unfortunately, there is no closed-form least-squares solution in this case, and would have to resort to other minimization strategies, such as gradient descent.
[Figure]
5. Basis functions on Lie Groups
In the previous chapter, we discussed how to extend discrete convolutional filters on Z 2 to continuous filters on R 2 .
Now, we will further extend this to more general groups f : G → R , which can be useful to implement efficient group convolutions.
Again, we consider a finite amount of points but now place them on the group
Z
=
{ z i } Pi =1 ⊂ G , and list the values associated to the points in u ∈ R P .
We can define a smooth function on the group f : G → R from the finite set of anchor points and values ( Z, u ) with basis function regression by defining a basis function that takes group elements as input ϕ G : G × G → R .
Now the difficulty lies in defining a basis function on the group.
To do so, we choose to parameterise it in the Lie algebra g associated to Lie group G :
[Equation]
where we choose some logarithm log map from group elements in G to Lie algebra g , and use the ( · ) ∨ : g → R dim( G ) operator to obtain a real vector that represents the group element in some chosen Lie algebra basis.
Note that we take the difference between two elements in the
Lie group g − 1 g ′ and that if two elements are the same g = g ′ this maps to the identity element log( g − 1 g )= 0 ∈ g in the Lie algebra.
This is desirable, as a more naive approach of mapping to the Lie algebra first, i.e. mapping to ϕ (log( g ′ )
− log( g )) , would depend on the location of the identity (see Fig.
3( b )).
Using this construction we can define filters on the group in terms of regular basis functions in real vector space ϕ : R dim( G ) → R , such as the ϕ RBF .
This generalises Bekkers (2019) proposing the same technique for B-spline basis functions.
Inference
Similar to basis function kernels for regular convolutions, we can find a value at any continuous location in filter space g ∈ G in closed-form:
[Equation]
where we define ϕ gZ := (cid:2) ϕ G ( g, z 1 ) ϕ G ( g, z 2 ) · · · ϕ G ( g, z P ) (cid:3) T ∈ R P , define Φ ZZ ∈ R P × P by [ Φ ZZ ] ij = ϕ G ( z i , z j ) , and a variance noise term σ 2 is added for numerical stability
[Table]
6. Basis functions for relaxed equivariance
In this work, we consider the method proposed by van der Ouderaa et al.
(2022) to relax equivariance, letting convolutional filters depend on an additional absolute group elements:
[Equation]
where filter (cid:102)
f G : G × (cid:101) G
→ R , input x : G → R and output y : G → R are defined on group G .
Unlike classic convolutions, filters (cid:102)
f G now operate on a group product space, taking not one but two group elements as input v − 1 u ∈ G and u ∈ (cid:101) G .
We can choose the relaxed subgroups through our choice of (cid:101) G ≤ G , and have (cid:101) G =
G if all subgroups are relaxed.
To emphasise the difference between regular convolutions and the operator of Eq.
(8) that relaxes equivariance, we add a (cid:101) -symbol to (cid:101) y and the filters (cid:102)
f G : G × (cid:101) G
→ R in case of relaxed equivariance.
We proceed by defining continuous filters for relaxed equivariance by performing basis function regression on the group product space G × (cid:101) G .
To do so, we consider a finite amount of points in this space (cid:101)
Z = { ( z i , (cid:101) z
i ) } Pi =1 ⊆ G × (cid:101) G , with associated values listed as u ∈ R P .
Thus, this requires defining basis functions on the product space of the group product space:
[Equation]
where we choose a logarithm log -map from the Lie group G to the Lie algebra g and operators ( · ) ∨ : R dim( G ) → R and ( · ) (cid:101) ∨ : R dim( (cid:101) G )
→ R to represent elements as real vectors by expanding in a chosen Lie algebra basis.
We have defined basis functions on group product spaces in terms of used stationary basis functions in real vector space ϕ : R dim( G )+dim( (cid:101) G )
→ R .
Similar to Sections 4 and 5, we can construct filters for relaxed equivariance in closed-form:
[Equation]
where ϕ ( g,g ′ )
Z :=
[ (cid:102) ϕ
G (( g ′ , z ′ 1 ) , ( g, z 1 )) · · · (cid:102) ϕ
G (( g ′ , z ′ P ) , ( g, z P )))
] T ∈ R P , and define [ Φ ZZ ] ij = (cid:102) ϕ G (( g ′ , z ′ i ) , ( g, z j )) with shape Φ ZZ ∈ R P × P , and σ 2 for numerical stability.
Note about smoothness in Lie group
One might ask whether filters on Lie groups defined in the Lie algebra (as in Sections 5 and 6) are still smooth on the Lie group.
Even if the function in the Lie algebra is smooth, smoothness is not guaranteed on the Lie group.
An example for SO(2) is depicted in Figs.
3( a ) and 3( b ).
However, filters will still be smooth close to the identity and possible discontinuities are not likely to form problems if filters are only locally supported around the origin.
This is typically true for classic convolutional filters, but is not often done for other groups, such as SO(2) , in literature.
This problem arises in many commonly used filter parameterisations, such as regular MLPs and encodings that use random Fourier feature spaces with non-integer frequencies, such as positional encodings or
SIRENs Sitzmann et al.
Our proposed basis function filters can still be guaranteed to be smooth on SO(2) by carefully choosing a periodic basis ϕ PERIODIC (see Fig.
3( c )).
[Figure]
7. Results Analysis of parameter efficiency
To assess parameter efficiency of basis function filters we compare test accuracy for different number of anchor points P .
The parameter count of basis function filters is directly proportional to P and can be chosen freely, unlike the fixed parameter count of classical convolutional filters which correspond to the area of the chosen filter support (e.g. 25 parameters for a 5 × 5 filter).
We compare test accuracy of regular convolutional layers (‘grid convs’) with basis function filters at varying parameter counts.
We initialise equally in both cases and reduce the parameter counts before training using the sparsification procedure described in Eqs.
(4) and (5) of Section 4, uniformly sampling anchor point locations within the filter support.
Fig.
4 shows test accuracy against number of parameters on CIFAR data with and without data augmentation.
We demonstrate a reduction of model parameters up to a factor of ∼ 10 × with a few percentage points drop in test accuracy in the most extreme case: 3-4% without data augmentation and 8-9% with data augmentation.
Surprisingly, we have merely a single P =1 anchor point left in this setting.
Per layer, that amounts to a single anchor point location shared over channels and a single anchor point value parameter for each filter channel.
More details in App.
C.
Large equivariant networks with basis function kernels
The parameter-efficiency of basis function filters allow us to construct group equivariant versions of large commonly used convolutional architectures at parameter counts equivalent to regular non-equivariant models.
To assess effectiveness, we compare test accuracy of such equivariant versions for different groups: regular discrete translation, discrete rotations p 4 , continuous planar roto-translations
SE(2) , and continuous planar scalings R
2 ⋊ R + .
Table 1 shows test accuracy for models trained with and without data augmentation.
Equivariant networks for rotation groups outperform the regular baseline for SE(2) given enough (typically >8) Monte Carlo samples
N MC .
We do not observe such improvement for planar scaling and hypothesise that could be due to sampling error of small filters.
This would be an issue with sampled regular representations in general and not of our basis function parameterisation as such.
Overall, we demonstrate that basis function filters can effectively be used to construct equivariant versions of large commonly used convolutional architectures.
Embedding group symmetries into the architecture yields higher test accuracies, as expected.
Relaxed equivariance and symmetry discovery
Lastly, we investigate the use of basis function filters to parameterise relaxations of equivariance.
We consider roto-translation
G =SE(2) and consider different amounts of relaxation of subgroup SO(2) by varying (cid:101) ω .
Basis function filters allow us to repeat the cross-validation experiments in van der Ouderaa et al.
(2022) on large commonly used ResNet architectures.
We directly report test accuracy in the upper part of Table 2.
Instead fixing the amount of relaxation, we also consider learning (cid:101) ω from training data by treating it as a parameter and optimising it with backpropagation together with the model parameters.
As (cid:101) ω directly controls layer-wise equivariance constraints, this allows gradient-based learning of symmetry constraints.
As suggested by van der
Ouderaa et al.
(2022), we add a regularizing λ
|| (cid:101) ω
|| 2 to the cross-entropy loss that encourages (cid:101) ω to be low and thus encourages stricter symmetry constraints.
This is analogous to the regularisation proposed in Augerino (Benton et al., 2020) to learn invariances through data augmentations and similarly requires tuning of λ .
We report test accuracy for different values for λ in the bottom part of Table 2.
We observe that learned equivariance constraints (cid:101) ω achieve similar performance to the best fixed setting for (cid:101) ω and outperform baselines in most cases.
This suggests that our efficient sparse filters can be an efficient parameterisation for relaxed equivariance.
Like most symmetry discovery methods, we still need to parameterise the class of symmetries that can be learned and limit ourselves to learning the amount of rotational equivariance to the SO(2) subgroup in an SE(2) -equivariant network.
Also, the used regularisation requires tuning an additional hyperparameter.
Immer et al.
(2022) observe that selecting the right amount of invariance with regularisation used in Augerino can be difficult and proposes an alternative loss motivated by marginal likelihood approximations.
Investigating such objectives could improve layer-wise equivariance learning in combination with the proposed filter parameterisation is interesting future work.
Regardless, our experiments indicate that basis function filters can effectively parameterise filters for relaxed equivariance in large networks.
We improve over van der Ouderaa et al.
(2022) which had to consider smaller architectures due to added hypernetwork parameters
and we recommend using the proposed filter parameterisation instead.
[Table]
8. Conclusion
This work proposes a practical parameterisation for continuous (group) convolutional filters with sampled regular representations.
Existing methods often rely on MLP hypernetworks which may require many additional model parameters compared to classical filters.
Instead, we propose to infer continuous filters from a small finite set of anchor points utilising basis function regression.
The approach is general and enables parameter-efficient continuous filters in regular convolutional networks (CNNs), group-equivariant convolutional layers (G-CNNs) and filters for relaxed equivariance.
We assess parameter efficiency of basis function filters at reduced parameter counts.
Futhermore, the proposed basis function filters allow us to construct equivariant versions of large commonly used convolutional neural network architectures that can outperform regular baselines on CIFAR-10 and CIFARclassification tasks.
Furthermore, the approach allows for relaxations of equivariance without increasing the parameter count.
In this last setting, we demonstrate efficient symmetry discovery by automatically learning layer-wise equivariance constraints from training data
Appendix A. Additional related work Basis function regression
Basis function regression is a well known procedure 1 which we use to infer values y ∗ ∈ R at arbitrary continuous points in D -dimensional space x ∗ ∈ R D from a finite number of P points in X ∈ R P × D with known values y ∈ R P .
We use a basis function ϕ
( x ′ , x ) : R D × R D → R and perform inference in closed-form:
[Equation]
(11) where vector ϕ x ∗ Z ∈ R P and matrix
Φ ZZ ∈ R P × P are constructed using the basis function [ ϕ xZ ] i := ϕ
( x , Z i ) and [ Φ ZZ ] ij :=
ϕ ( Z i : , Z j : ) and regularizing observational variance scalar σ 2 .
Here Z i : ∈ R D denotes the i’th row of Z .
If ϕ ( x ′ , x ) can be rewritten as ϕ ′ ( x − x ′ ) , the basis function is called ‘stationary’.
In this case, we will simply write the basis function as the one-argument function ϕ : R D → R .
One of the most commonly used basis functions ϕ in regression is the stationary squared exponential radial basis function ϕ RBF ( a ) =
σ 2 exp (cid:0) − 2 ω 2 a T a (cid:1) , with scalar variance σ 2 ∈ R and frequency ω 2 ∈ R .
Appendix A. Implementation Basis function factorisation
We choose to factor the basis functions as ϕ ( a ) =
(cid:81)
Di =
ϕ
i ( a i ) with a ∈ R D and represent individual dimensions using radial or periodic basis functions:
[Equation]
(12) where we set ϕ i
= ϕ
PERIODIC for dimensions that correspond to the SO(2) subgroup and ϕ i =
ϕ RBF for dimensions that correspond to translation R 2 or scale R + subgroups.
Note that we have D = dim( G ) for strict equivariance and D = dim( G )+ dim( (cid:101) G ) for relaxed equivariance.
Factorising basis functions across dimensions differs from factorising the filter itself across dimensions, which is often called a ‘separable filter’ (Knigge et al., 2022).
Separable filters have efficiency benefits, but limit the functions a filter can represent.
Unlike separable kernels, our proposed factorisation of basis functions still yield filters that can approximate any function, given enough anchor points.
Intuition behind frequency parameters
To understand the effect of frequency parameters on filters, it can be helpful to separately consider the first frequencies, overloading notation ω :
= [ ω 1 , . . .
, ω dim G ] T ∈ R dim( G ) , and the remaining dim( (cid:101) G ) frequency components as (cid:101) ω ∈ R dim( (cid:101) G ) .
The frequencies ω control the spectral properties of regular filters.
For example, frequencies for translation G =T(2) would be two-dimensional ω ∈ R 2 and influence the respective frequencies along the x- and y-axis of the two-dimensional filter.
On the other hand, the additional frequencies for relaxed equivariance (cid:101) ω control the amount of equivariance of relaxed subgroups, where higher frequencies correspond to more relaxation.
Conversely, setting (cid:101) ω
= 0 corresponds to no relaxation in which case the layer becomes equivalent to a regular (group) convolution (van der Ouderaa et al., 2022), which is guaranteed to be strictly equivariant || f || EE =0 .
Group elements as real vectors
We map Lie group elements to Lie algebra elements through a choice of log and expand them through ( · ) ∨ or ( · ) (cid:101) ∨ in a Lie algebra basis so they can be implemented as real vectors.
The choice of basis is somewhat arbitrary and can be set according to the application.
We choose our basis such that a length of 1 corresponds to the apothem of a sampled filter support for T(2) , doubling in scale for R + and a full rotational orbit for SO(2) .
For relaxed equivariance, the same basis is used except for T(2) where group elements correspond to the input domain and therefore the apothem of the input image support is used instead of that of the sampled filter support.
We always initialise frequency components as 1, if not specified otherwise.
Example for relaxed rotational equivariance
In our experiments, we will consider relaxing the SO(2) subgroup of SE(2) = T(2) ⋊ SO(2) .
In this case, we have that dim( G ) = dim(T(2)) + dim(SO(2))
= 2 + 1 = 3 and dim( (cid:101) G ) = dim(SO(2))
= 1 .
Therefore, D = dim( G )+dim( (cid:101) G )
=
3+1 = 4 .
We will use ϕ 1 = ϕ 2 = ϕ RBF for the dimensions corresponding to translation T(2) subgroup, and ϕ 3 = ϕ 4 = ϕ PERIODIC for the dimensions that correspond to rotation SO(2) and relaxation of rotation SO(2) subgroup.
Thus, the frequency vectors in this case are of dimensionality ω ∈ R dim( G ) =
R 3 and (cid:101) ω ∈ R dim( (cid:101) G )
= R 1 .
Appendix B. Appendix: Training details B.1. Architecture
For the architectures, we used the standard ResNet-18 and ResNet-152 architecture He et al.
(2016) implementations of PyTorch Paszke et al.
(2017) with the following parameter counts.
[Table]
B.2. Dataset details
We evaluate with the CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009).
The
CIFAR-10 dataset consists of 60,000 32x32 colour images.
CIFAR-100 consists of 60, 32x32 colour images in 100 classes.
For both datasets, we use the default 50,000/10, split for training and testing and apply standard zero mean and unit standard deviation normalisation using training set statistics.
B.3. Optimization details
In all experiments, we optimise using Adam (Kingma and Ba, 2014) with an initial learning rate of 0.001 cosine decayed to zero over 400 epochs.
Appendix C. Parameter reduction with sparsified convolutions
Unlike grids of regular convolutions
[Table]
the P basis coordinates (rows of Z ) of basis function kernels are no longer required to lie in a grid.
Furthermore, the number of basis functions per filter can be chosen arbitrarily and can be lower than the number of pixel grids in regular convolutions.
In Eqs.
(4) and (5) of Section
[Table]
we formulate a procedure to sparsify parameters of convolutional kernels, similar to ’pruning’.
Note that we reduce the number of parameters that describe the convolutional filters and (and thus reduce storage cost)
[Table]
but the size of the sampled filter during training/inference remains the same.
In Section 7, we evaluate parameter efficiency of basis function kernels by evaluating test performance of a
ResNet-18 model with reduced number of basis function coordinates.
In Tables 4 and
[Table]
we report more detailed results of this experiment including exact parameter counts of reduced models as well as test accuracy.
Mean and standard error are reported over 3 random seeds
[Table] [Table]
accuracy of ResNet18 model trained with reduced number of parameters, with and without data augmentation.
Total parameter count and parameter count per filter are reported, as well as standard error over three seeds
[Equation]