Relaxing Equivariance Constraints with Non-stationary Continuous Filters
Anonymous Author(s)
AffiliationAddress email
Abstract
Equivariances provide useful inductive biases in neural network modeling, with thetranslation equivariance of convolutional neural networks being a canonical example.
Equivariances can be embedded in architectures through weight-sharing andplace symmetry constraints on the functions a neural network can represent.
Thetype of symmetry is typically fixed and has to be chosen in advance.
Although sometasks are inherently equivariant, many tasks do not strictly follow such symmetries.
In such cases, equivariance constraints can be overly restrictive.
In this work,we propose a parameter-efficient relaxation of equivariance that can effectivelyinterpolate between a (i) non-equivariant linear product, (ii) a strict-equivariantconvolution, and (iii) a strictly-invariant mapping.
The proposed parameterizationcan be thought of as a building block to allow adjustable symmetry structure inneural networks.
Compared to non-equivariant or strict-equivariant baselines, weexperimentally verify that soft equivariance leads to improved performance interms of test accuracy on CIFAR-10 and CIFAR-100 image classification tasks.
Introduction
Symmetric properties, such as equivariances and invariances, can be embedded into neural networkarchitectures to provide inductive biases that leads to better data-efficiency and improved generalization.
Convolutional layers are known to provide translation equivariance in simple euclidean spaces,and recent works have allowed various extensions to more complex groups and domains.
However,symmetries are typically fixed, must be specified in advance and can not be adjusted.
Symmetries embedded in network architectures enforce a hard constraint on the functions a neuralnetwork can represent.
This can be an effective way to encode prior knowledge for problems thatare inherently symmetric.
However, hard symmetry constraints can become prohibitive if a problemdoes not strictly follows symmetries.
For example, convolutional layers cannot encode potentiallyrelevant absolute positional information, and ‘6’s and a ‘9’s become difficult to distinguish underrotation invariance.
Relaxed symmetry constraints can mitigate such symmetry misspecificationswithout completely losing the useful inductive biases that symmetries provide.
We propose to relax equivariance constraints by generalizing the convolution operator with nonstationary filters that can depend on absolute group elements.
Unlike other approaches, the resultinglayer that can efficiently interpolate between ( i ) non-invariant linear products akin to a fully-connectedlayer, ( ii ) strict group equivariant convolutions, and ( iii ) strict group invariant mappings (Fig. 1).
[Figure]
~ ~  ~ ~  ~ ~ ~
[Figure]
~
[Figure]
The importance of this work is twofold.
First of all, relaxable symmetry constraints can directlyimprove the performance in cases where strict symmetries are misspecified and result in an overly restrictive model class.
Secondly, automatically learning symmetry structure from data is an interestingproblem.
Work in this field which often focuses on invariances [27, 4, 26, 22, 10], which are easierto parameterize than equivariances.
Parameterizations that allow smooth and adjustable symmetryconstraints could extends such methods to layer-by-layer learnable equivariances.
Related Work
Group equivariance.
Equivariance constraints applied to layers provide a strong inductive biasthat enforce transformations in the input to result in equivalent transformations in the output.
Forcompact groups it can be shown that this constraint naturally leads to (group) convolutions [11].
Many works have allowed for equivariances to groups other than translation, including continuousroto-translations [31, 29, 12], discrete roto-translations [5, 3], scale [30], and permutations [32] andnon-euclidean domains, e.g., spheres [6], points clouds [9] and graphs [21].
Symmetry misspecification and approximate equivariance.
Although symmetry constraints canbe very effective in machine learning, they can become prohibitive if data does not exactly follow theenforced symmetry.
For instance, Liu et al.
[13] raised an ‘intriguing failing’ of convolutional neuralnetworks by showing that they can not encode absolute coordinate information.
Adding explicitcoordinate information was proposed as an ad-hoc solution to the problem, effectively breaking strictequivariance constraints.
To achieve the same result, Finzi et al.
[8] proposed to break equivarianceof a convolutional layer by summing it with a non-equivariant fully-connected layer, requiring manyadditional parameters.
Alternatively, Romero and Lohit [18] consider a sparse local support aroundthe group identity element to break equivariance.
This is effective for sparsely sampled subgroups,such as rotation, but becomes less practical for more densely sampled groups such as translationwhere the local support could lead to very sparse feature maps.
Lastly, Wang et al.
[28] proposea relaxed convolutional operator similar to our non-stationary approach, but require a low-rankfactorization which potentially limits the expressivity of the feature maps.
The same work brieflydiscusses a mathematical description without such factorization similar to our proposal which isdeemed “too large a trainable weight space to be practical”.
This is true for discrete convolutionalkernels.
Instead we define our kernels as continuous kernels [20] parameterized with a finite numberof parameters.
Consequently, our weight space becomes tractable and does not pose a problem at all.
Physics and filtering.
Symmetries play a central role in physics.
In [28] approximate equivariancesare used to allow more robust models for dynamical systems.
In Noether networks [1] conservationlaws are inferred by learning symmetries from data.
Our work could offer new insights in how symmetry constraints could be relaxed in such applications.
In geophysics and seismology, there aresome interesting parallels between non-stationary filtering methods and approximate equivariances inmachine learning.
In particular, [15] discusses generalizations of the convolution in the context ofnon-stationary filtering very similar to how we propose to relax equivariances in neural networks.
Automatic symmetry discovery.
An interesting problem is to automatically learn symmetrystructure in neural networks architectures based on data.
For instance, Lorraine et al.
[14] learn dataaugmentations by differentiating the validation loss using the implicit function theorem and Zhou et al.[33] showed how symmetry structure could be learned through a meta-learning outer loop.
In [4] and[8] symmetries are selected using a training loss directly, using an additional regularizer.
Symmetrydiscovery methods often focus on learning global invariances [27, 4, 22, 26, 10], which are easier toparameterize than layer-wise equivariances.
We propose a method to continuosly relax equivariances,which can be useful to extend symmetry discovery methods to learn layer-wise equivariance.
Background
Groups.
A Lie group provides a natural way to describe continuous symmetry. It forms a continuousmanifold of which the underlying elements are equipped with a group structure.
Lie groups do notnecessarily form a vector space.
However, to every Lie group G we can associate an underlyingvector space called the Lie algebra g .
The Lie algebra corresponds to the tangent space of thegroup at the identity, has the same dimensionality as the group, and captures the local structure ofthe group.
Because the Lie algebra is a vector space, elements a ∈ g can be expanded in a basisa =
(cid:80) dim( G ) i
= α
i A i with coefficients α ∈ R dim( G ) .
The exponential map exp : g → G maps elementsfrom the Lie algebra to the Lie group.
We can also define a logarithm map log : G → g that mapselements in the Lie group to the Lie algebra.
Such a choice always exists, but is not necessarilysmooth or unique.
Equivariance and invariance.
Equivariance is the property of a mapping such that transformationsto the input result in equivalent transformations to the output.
If changes are invertible, they can bedescribed as the action of a group G on some space X .
Formally, we say that a function h : X → Xis equivariant to the group G if h ( g · x ) = g · h ( x ) for all g ∈ G, x ∈ X .
If the output of the function is completely independent to the action of the group G on the input, wesay that the function is invariant to the group G .
Formally, a function h : X → X is invariant to thegroup G
if h ( g · x ) = h ( x ) for all g ∈ G, x ∈ X .
The group convolution.
Let f : G
→
R be an input signal and k θ : G → R be a convolutionalkernel parameterized by θ .
The group convolution is defined as:
[Equation]
where we integrate with respect to the Haar measure of the group µ .
The group convolution wasfound to be both a sufficient and necessary condition for equivariance to a compact group [11].
Theregular convolution for translation equivariance is the special case where the group is the translationgroup G = T(n) ∼ = R n , and the group action is given by addition, i.e., v − 1 u = u − v .
The convolutional kernel k θ ( v − 1 u ) , which we will sometimes refer to as ‘filter’ for stylistic purposes,is called stationary because it can only depend on u and v through v − 1 u .
This invariance of thekernel is an important constraint and in fact a requirement for h to be equivariant .
Breaking thestationarity of the kernel allows us to relax the equivariant symmetry constraints, and it is one of thecentral ideas of this paper.
Group lifting.
The input and output signals of a neural network are typically not defined on the group
G , but rather on an input and output space X and Y .
Using a group lifting and projection procedure[11, 7], we can define our model on the group and still apply it on the input and output space.
Wedefine a lifting operator ↑ G to map input signals f : X → R to functions on the group f ↑ G : G → R .
Similarly, we define a projection operator that maps function on the group f ′ :
G → R to functionson the output space f ′ ↓ Y : Y → R .
Method
4.1 Non-stationary Integral Operator
The group convolution operation of Eq.
(1) is strictly equivariant due to the stationarity of the kernel.
That is, the kernel k only depends on relative group element v − 1 u .
For translation G =T(2) , thekernel only depends on relative coordinates ( u − v ) and not on the absolute position in the image.
Torelax equivariance constraints, we let the kernel also depend on the absolute group element:
[Equation]
The main difference with the group convolution of Eq.
(1) is that the kernel k θ : G × G
→ R nowhas two input group elements.
We will refer to the first input argument v − 1 u ∈ G as the stationarycomponent and the second input argument v ∈ G as the non-stationary component .
In case oftranslation, this change lets the kernels depend also on the absolute coordinate of the input image onwhich it is applied.
Consequently, h ( u ) does no longer describe the regular convolution, but rather anon-stationary integral operator, for which the convolution remains a limiting case.
For the purpose of this work, there is no meaningful difference between expressing the non-stationarycomponent in the input domain k θ ( v − 1 u, v ) or the output domain k θ ( v − 1 u, u ) .
We will stick withthe input domain and some discussion on the difference can be found in [15].
One could also thinkof writing the kernel in an even more general form: k θ ( u, v ) .
This, as such, does not lead to a moreexpressive kernel (they are isomorphic, see ?? ).
This more general description will make it harder toparameterize kernels with controllable symmetry constraints, so we do not consider it.
The non-stationary integral operator of Section 4.1 can represents interesting special cases through anindependence of k θ with respect to the first and second input arguments.
Remark 1 (Linear product / Fully-connected) .
If the kernel equals an impulse response in thestationary component multiplied by some function k ′ θ in the non-stationary component k θ ( v − 1 u, v ) =
δ ( v − 1 u ) k ′ θ ( v ) , then the operator h ( u ) =
( k θ • f )( u ) corresponds to a linear product with k ′ θ .
If we have a kernel k θ ( v − 1 u, v ) =
δ ( v − 1 u ) k ′ θ ( v ) , then we have that:
[Equation]
which is the linear product between k ′ θ ( u ) and input f ( u ) .
This is akin to an element-wise productcontinuously defined on G .
Optionally, the layer could be followed by an invariant layer to averagepool activations.
In the discrete case G = Z n , this corresponds to a fully-connected layer where theweights are given by k ′ θ .
Remark 2 (Group equivariance / Convolution) .
If the kernel does not depend on the non-stationarycomponent, then the operator h ( u ) =
( k θ • f )( u ) is equivalent to a strict G-equivariant convolution.
If the kernel only depends on the first stationary component, then k θ ( v − 1 u, v ) =
k ′ θ
v − 1 u ) , and:
[Equation]
which equals the group convolution which is known to be strictly group equivariant.
Remark 3 (Group Invariance / Pooling) .
If the kernel is independent of both stationary and nonstationary components, the operator h ( u ) =
( k θ • f )( u ) is strictly G-invariant.
If the kernel does not depend on both inputs, it must be constant k θ ( v − 1 u, u )
= C , and we obtain:
[Equation]
which is a scaled average ‘global-pooling’ over group actions and leads to strict group invariance.
We obtain partial invariance or ‘local-pooling’ if the kernel is only non-zero constant in some localsupport of the stationary domain.
4.2 Parameterizing the kernel
Lie algebra basis.
In Eq.
(2), we have proposed a non-stationary integral operator with a kernelk θ : G × G
→ R that takes two input arguments: the stationary group element v − 1 u and a nonstationary group element v .
We can parameterize the kernel k θ in terms of real vector spaces, whichare more practical to work with, by defining elements in an explicit Lie algebra basis.
To do so, wefirst define a logarithm function log and express the kernel k
= ˆ k (log g, log h ) in terms of a kernelin the Lie algebra
ˆ k : g × g
→ R , which always form vector spaces.
For rotation G =SO(2) , theexponential map exp sending Lie algebra elements (on a line) to Lie group elements (on a circle) isnot left invertible, but is right invertible, as a function.
As the right inverse is not unique, we chooseto use the principal log sending Lie group elements to Lie algebra elements closest to the identity.
Ingeneral, such a choice for the logarithm always exists but is not necessarily unique or continuous.
Wechoose a vector space basis { A i } dim( G ) i = to express Lie algebra elements a ∈ g in terms of coefficientsα ∈ R dim( G ) that relate to Lie group elements g ∈ G by:
[Equation]
Through this construction, each group element g corresponds through a choice of log to a unique Liealgebra element a which can be implemented as a real vector α .
Functions in the Lie algebra.
We parameterize the proposed non-stationary kernel k θ : G × G → R ,which takes two input arguments, in the Lie algebra kernel
ˆ k : g × g → R acting on a cross-productof two vector spaces.
We refer to the first vector space related to the stationary component as thefilter-space , as it corresponds to space in which filters of conventional convolutions are defined.
And refer to the second vector space of the non-stationary component as the domain-space , becauseelements directly relate to absolute group elements in the input domain.
In Section 4.1 we showed that an independence of the kernel with respect to the first and second inputargument correspond to strict equivariant and invariant symmetry constraints.
We would, therefore,like to choose ˆ k such that we can control the dependence on the stationary and non-stationarycomponents, as this will allow for explicit control over symmetry constraints.
One option wouldbe to let ˆ k be a Gaussian Processes (GP) with an RBF-kernel with a frequency parameter ω foreach dimension — set to be the inverse of the conventional lengthscale parameter ω = 1 l 2 .
Thefrequency parameter gives us control over the spectral properties of the function in filter-spaceand domain-space.
But crucially, a frequency of ω =0 results in a constant infinite lengthscale andtherefore a function ˆ k that is independent to the associated input dimension.
This mechanic allows usto select the special cases discussed in Section 4.1 and interpolate between them!
Together, we havea frequency parameter ω ∈ R dim( G ) that controls spectral properties in filter-space and a frequencyparameter ω ′ ∈ R dim( G ) to control spectral properties in domain-space.
Instead of a GP, we willconsider a weight-space variation in this work that equivalently has frequency parameters ω , ω ′ withthe same discussed properties.
Weight-space implementation.
The infinite number of basis functions in a GP feature space can beapproximated with a finite number of D random fourier features (RFF) [17, 24] γ ω : g → R 2 D :
[Equation]
embedding Lie algebra elements a ∈ g as real vectors using their coefficients α ∈ R dim( G ) in a waythat is controlled by frequency parameter ω ∈ R dim( G ) .
Values of W ∈ R D × dim( G ) are randomlyinitialized and can be kept fixed.
We parameterize the Lie algebra kernel
ˆ k : g × g → R with a neuralnetwork NN θ : R 4D → R taking concatenated Fourier features as input:
[Equation]
where γ ω ( a v − 1 u ) are the Fourier features of the stationary component on the filter-space and γ ω ′ ( a v )are the Fourier features of the non-stationary component on the domain-space.
[Figure]
Directly representing the kernel with an MLP on α without Fourier features will likely hamperperformance as MLPs are known to suffer from ‘spectral bias’ [2, 16], making it difficult to encodehigh-frequency functions.
Sinusoidal activations in random fourier features [17, 24] and SIRENs[23] alleviate this issue and have been found suitable encodings of positional information in neuralnetworks [25], including (place-coded) continuous kernels in neural networks [20, 19].
Moreover, itcan be shown that if we initialize W with independent Gaussian noise from N (0 , 1) and let NN θ bea single linear layer, ˆ k corresponds to a GP with RBF-kernel in the infinite width limit D → ∞ [17].
Intuition behind frequency parameters.
The frequency parameters ω and ω ′ control the frequencies in the respective ‘filter-space’ and ‘domain-space’.
Specific values of ω and ω ′ correspondto a layer h that exactly is or interpolates between special cases discussed in Section 4.1.
Mostnotably, for ω ′ =
the layer becomes a strictly equivariant convolution and for both ω = ω ′ =
0 thelayer performs invariant pooling.
To understand why this is the case, it can be helpful to consider howdifferent frequencies affect the kernel.
A Fourier feature with zero frequency parameter is constantγ 0 , by definition.
For ω ′ = 0 we therefore have a constant Fourier feature for the non-stationarycomponent γ ω ′ ( a v )= γ 0 and a kernel that is solely defined in the filter-space.
If both ω = ω ′ = 0 , wealso have a constant Fourier feature for the non-stationary component γ ω ( a v − 1 u )
= γ 0 and a kernelthat is constant everywhere, which results in invariant pooling.
Positive values for ω ′ introducenon-stationarity that effectively relaxes the strict equivariance constraints of the convolution at ω ′ =
Note that individual scalars in ω ′ correspond to particular subgroups and we can relax equivarianceconstraints of specific subgroups by letting ω ′ be greater than zero in the associated dimension.
Example for translation group Let us consider relaxing equivariance of the translation group
G = T (2) , as an example.
Since dim(T(2))
= 2 , we have a two dimensional filter-space and domainspace and therefore also two-dimensional filter-space frequencies ω =
[ ω x , ω y ] T ∈ R 2 and twodomain-space frequencies ω ′
=
[ ω ′ x , ω ′ y
T ∈ R 2 .
If ω ′ = 0 , the layer reduces to a translationallyequivariant convolutional layer.
In other words, for any value of ω or weights θ the kernel remainsstationary (due to ω ′ = 0 ) which means the layer equals a convolution where the same filter is appliedto all locations in the input.
Higher values of ω , i.e. ω x , ω y > 0 , correspond to more rapid changes inx- and y-direction in filter-space.
If, instead, we let the components of the domain-space frequenciesω ′ become non-zero, i.e. ω ′ x , ω ′ y > 0 , then the kernel values can also depend on the absolutecoordinate in the input image.
Higher ω ′ result in a more rapidly changing kernel relative to differentabsolute locations in the input image.
This non-stationarity breaks equivariance constraints in acontinuous way.
For sufficiently high ω ′ , the function can become akin to a linear product that hasenough variance to place an independent weight on each pixel location.
An illustration of the differentsymmetry modes for translation G =T(2) is shown in Fig.
S t r i c t e q u
i v a r i a n c e ' =v =
° v = 36° v = 72° v = 108° v = 144° v = 180° v = 216° v = 252° v = 288° v = 324° v = 360°
S o f t e q u
i v a r i a n c e ' >
s a m p l e d
Implementation
Parameterization in T(2) , SO(2) and SE(2)
Although we define the Eq.
(2) for general groups,we empirically validate the approach by relaxing symmetries in T(2) , SO(2) and SE(2) groups.
Wedefine frequencies in x-direction ω x , ω ′ x , y-direction ω y , ω ′ y and rotation ω r , ω ′ r .
To ensure the kernelis smooth in SO(2) , we let ω r , ω ′ r and associated entries in W be integer valued, similar to circularharmonics.
In Table 1 we give an overview of ω and ω ′ for different models.
Sampling.
The integral in Eq.
(2) is not tractable in closed form and we therefore choose toapproximate it by using samples, similar to Finzi et al.
We deterministically evaluate groupelements in T(2) that correspond to (lifted) pixel locations and randomly sample 4, 8 or 16 groupelements uniformly over SO(2) .
For [18], the uniform distribution over SO(2) covers the entire groupat the start of training but has an adjustable range that is learned by early-stopping on the validationloss.
Both approaches have continuous convolutional kernels that can be evaluated anywhere andthus do not require any interpolation of filter values.
Locally-supported in filter-space.
Similar to conventional convolutional networks, we only definethe kernel in a local subset S of the filter-space, forcing the kernel to be zero k θ =0 outside this localsupport.
Doing so, makes the approximation of the integral in Eq.
(2) much cheaper as we only needto sample the group elements within the support.
Another consequence is that we no longer obtainstrictly invariant global-pooling of Eq.
Instead, a kernel that is only a constant non-zero value ina local support represents a local-pooling layer, where the pool size equals the support size.
In theexperiments we let S be a disk in T(2) with a diameter equal to 7 pixels in the input domain X .
Choosing frequency parameters.
For fair comparison with Romero and Lohit [18], we follow thesame
ResNet architecture and initialization of ω and NN θ .
To select ω ′ , we use cross-validationand limit the search space by considering equal filter-space frequencies ω x = ω y = ω r , translation axesω ′
x = ω ′ y , and share frequencies across layers.
Equal frequencies in x and y only scale the Fourierbasis and do not constrain the kernel to be isotropic.
Independent frequencies per layer is interestingfuture work.
[Table] [Table]
[Table] [Table]
T
[Table]
[Table] [Table]
soft ω x
[Equation]
ω
[Equation]
, ω r
[Equation]
′ x
[Equation] [Equation]
r
[Table]
Experiments6.1 Verifying soft-equivariance using symmetry misspecification toy example
To verify that relaxed equivariance allows for less restrictive functions compared to strict equivariance,we repeat the MNIST6-180 problem of [18] and include our model.
This toy problem is designedsuch that it can not be solved under strict symmetry constraints.
It consists of the 6’s in the MNISTdataset with half of the dataset randomly rotated by 180 ◦ degrees.
The task is to determine whethera sample has been rotated, i.e., a binary classification problem between ‘6’s and ‘9’s.
In Table 3,we compare a rotation invariant G-CNN model that consists of multiple strictly equivariant layersfollowed by group pooling and compare with [18] and our model.
In line with our expectations, wefind that in this artificial set-up the relaxed ResNet model quickly converges to 100% test accuracy.
The model with strict rotation symmetry is unable to distinguish between ‘6’s and ‘9’s and can thusnot improve over a simple coin-flip with 50% accuracy on average.
[Table]
6.2 Image classification experiments
We assess the effectivity of relaxing equivariance constraints for translation and rotation groups T(2) ,
SO(2) and SE(2) on CIFAR-10 and CIFAR-100 image classification tasks.
As strict symmetries canlead to misspecification, we hypothesize that relaxed equivariance can improve the model performanceon these tasks.
As baselines, we use a strictly equivariant G-CNNs and a Partial G-CNNs [18].
Thedifference in model sizes arising from different kernel input dimensions are neglectable compared tothe total number of model parameters for the used architecture.
In Fig.
4, we plot performance on thevalidation set for different values of frequency parameter ω ′ and list final test accuracies in Table 2.
From Table 2, we can see that our method outperforms the baseline on both datasets across all settings.
On T(2) , we find that the improvement is negligible small and it would be interesting to see whetherthis gap becomes larger with increased number of basis functions or other choices of NN θ .
We findparticularly large improvements in test accuracy for soft- SO(2) on CIFAR-10 and CIFAR-100.
On
CIFAR-100, our relaxed equivariance approach improves test accuracy of the non-invariant baselineby 8 percentage points.
To further assess the effect of equivariance relaxation has on model performance, we can examine theimpact different values for of non-stationary frequency parameter ω ′ on model performance.
In Fig.
4,we show validation accuracy for different ω ′ averaged over three seeds with error bars correspondingto standard error ( σ √ 3 ) and show final test accuracy for model with best validation score.
The scale ofthe found optimal frequency parameters should not directly be interpret as they also depend on thechosen basis, parameterization and initialization of NN θ .
Yet, we still observe that some relaxationof equivariance with frequency parameter ω ′ ̸ = 0 leads to better results when compared to strictequivariant ω ′ = 0 setting.
That relaxing equivariance results in higher test accuracy in these casesindicates that strict equivariance could be misspecified for these problems to which our approachoffers a solution.
Conclusion
In this work, we have proposed a generalization of the group convolution that allows for smoothparameter-efficient relaxation of otherwise strict symmetry constraints.
The main idea is to use a nonstationary kernel that also depends on the absolute input group element, breaking strict equivarianceconstraints.
Moreover, we show that we can obtain explicit control over the symmetry constraintsthrough tunable frequency parameters, by representing group elements in a Fourier feature space.
Although we have demonstrated a method to adjust equivariances within layers, we have so far usedcross-validation in order to set their optimal value.
There have been some recent approaches toautomatically learn symmetries in neural networks from training data [4, 22, 26, 10], but typicallyfocus on learning invariances which are easier to parameterize.
The method proposed in this papercould offer ways to parameterize networks with learnable equivariance in such methods.
This pavesthe way for automatic layer-by-layer symmetry discovery as part of the training procedure.
By relaxing equivariance properties we can leverage useful inductive bias that symmetries canprovide, while preventing possible symmetry misspecification if data does not fully obey an embedded symmetry.
Experimentally, we have demonstrated that neural network layers with adjustablesymmetry constraints can improve test accuracy on CIFAR-10 and CIFAR-100 image classificationtasks, outperforming strict-equivariant models with up to 8 percentage points in test accuracy on
CIFAR-100.
More importantly, we hope that the parameterization can be a generally useful buildingblock in machine learning applications that require smooth relaxations of symmetry constraints.