Learning Partial Equivariances from Data
Anonymous Author(s)
AffiliationAddress email
Abstract
Group Convolutional Neural Networks (G-CNNs) constrain learned features torespect the symmetries in the selected group, and lead to better generalization whenthese symmetries appear in the data.
If this is not the case, however, equivarianceleads to overly constrained models and worse performance.
Frequently, transformations occurring in data can be better represented by a subset of a group thanby a group as a whole, e.g., rotations in [− 90 ○ , 90 ○ ] .
In such cases, a model thatrespects equivariance partially is better suited to represent the data.
In addition,relevant transformations may differ for low and high-level features.
For instance,full rotation equivariance is useful to describe edge orientations in a face, but partialrotation equivariance is better suited to describe face poses relative to the camera.
In other words, the optimal level of equivariance may differ per layer.
In this work,we introduce Partial G-CNNs : G-CNNs able to learn layer-wise levels of partialand full equivariance to discrete, continuous groups and combinations thereof aspart of training.
Partial G-CNNs retain full equivariance when beneficial, e.g., forrotated MNIST, but adjust it whenever it becomes harmful, e.g., for classificationof 6 / 9 digits or natural images.
We empirically show that partial G-CNNs pair
G-CNNs when full equivariance is advantageous, and outperform them otherwise.
Introduction
The translation equivariance of Convolutional Neural Networks (CNNs) [30] has proven an importantinductive bias for good generalization on vision tasks.
This is achieved by restricting learned featuresto respect the translation symmetry encountered in visual data, such that if an input is translated,its features are also translated, but not modified.
Group equivariant CNNs (G-CNNs) [7] extendequivariance to other symmetry groups.
Analogously, they restrict the learned features to respect thesymmetries in the group considered such that if an input is transformed by an element in the group,e.g., a rotation, its features are also transformed, e.g., rotated, but not modified.
Nevertheless, the group to which G-CNNs are equivariant must be fixed prior to training, andimposing equivariance to symmetries not present in the data leads to overly constrained models andworse performance [6].
The latter comes from a difference in the data distribution, and the familyof distributions the model can describe.
Consequently, the group must be selected carefully, and itshould correspond to the transformations that appear naturally in the data.
Frequently, transformations appearing in data can be better represented by a subset of a group thanby a group as a whole, e.g., rotations in [− 90 ○ , 90 ○ ] .
For instance, natural images much more likelyshow an elephant standing straight or slightly rotated than an elephant upside-down.
In some cases,group transformations even change the desired model response, e.g., in the classification of the digits6 and 9.
In both examples, the data distribution is better represented by a model that respects rotationequivariance partially .
That is, a model equivariant to some, but not all rotations.
Moreover, the optimal level of equivariance may change per layer.
This results from changes inthe likelihood of some transformations for low and high-level features.
For instance, whereas the
Consequently, all output components are part of the output for any group transformation of the
In a partial group convolution, however, the domain of the output is a learned subset S and all values outside of S are discarded.
As a result, parts of the output in a partial group convolution can change for different group transformations of the input.
In this figure, the output components within S for a 0 ○ rotation (outlined in orange) gradually leave S for stronger transformations of the input.
For strong transformations –here a 180 ○ rotation–, the output components within S are entirely different.
This difference allows partial group convolutions to distinguish among input transformations.
By controlling the size of S , the level of equivariance of the operation can be adjusted.
orientations of edges in an human face are properly described with full rotation equivariance, the posesof human faces relative to the camera are better represented by rotations in a subset of the circle.
The previous observations indicate that constructing a model with different levels of equivariance ateach layer may be advantageous.
Weiler and Cesa [45] empirically observed that manually tuningthe level of equivariance at different layers leads to accuracy improvements for non-fully equivarianttasks.
Nevertheless, manually tuning layer-wise levels of equivariance is not straightforward andrequires iterations over several possible combinations of equivariance levels.
Consequently, it isdesirable to construct a model able to learn optimal levels of equivariance directly from data.
In this work, we introduce Partial Group equivariant CNNs (Partial G-CNNs): a family of equivariantmodels able to learn layer-wise levels of equivariance directly from data.
Instead of sampling groupelements uniformly from the group during the group convolution –as in G-CNNs–, Partial G-CNNslearn a probability distribution over group elements at each group convolutional layer in the network,and sample group elements during group convolutions from the learned distributions.
By tuning thelearned distributions, Partial G-CNNs adjust their level of equivariance at each layer during training.
We evaluate Partial G-CNNs on illustrative toy tasks and vision benchmark datasets.
We show thatwhenever full equivariance is beneficial, e.g., for rotated MNIST, Partial G-CNNs learn to remainfully equivariant.
However, if equivariance becomes harmful, e.g., for classification of 6 / 9 digitsand natural images, Partial G-CNNs learn to adjust equivariance to a subset of the group to improveaccuracy.
Partial G-CNNs improve upon conventional G-CNNs when equivariance reductions areadvantageous, and match their performance whenever their design is optimal.
In summary, our contributions are:
• We present a novel design for the construction of equivariant neural networks, with which layer-wiselevels of partial or full equivariance can be learned from data.• We empirically show that Partial G-CNNs perform better than conventional G-CNNs for tasks forwhich full equivariance is harmful, and match their performance if full equivariance is beneficial.
Background
This work expects the reader to have a basic understanding of concepts from group theory such asgroups, subgroups and group actions.
Please refer to Appx.
A if you are unfamiliar with these terms.
Group equivariance.
Group equivariance is the property of a map to respect the transformations in agroup.
We say that a map is equivariant to a group if whenever the input is transformed by elementsof the group, the output of the map is equally transformed but not modified.
Formally, for a group Gwith elements g ∈ G acting on a set X , and a map ϕ
∶ X → X , we say that ϕ is equivariant to G if:
[Equation]
For example, the convolution of a signal f ∶ R → R and a kernel ψ ∶ R → R is translation equivariantbecause L t ( ψ ∗ f )=
ψ ∗ L t f , where L t translates the function by t : L t f ( x )=
f
( x − t ) .
That is, if theinput is translated its convolutional descriptors are also translated but not modified
The group convolution.
To construct neural networks equivariant to a group G , we require anoperation that respects the symmetries in the group.
The group convolution is such a mapping, whichgeneralizes the convolution for equivariance to general symmetry groups.
Formally, for any u ∈ G ,the group convolution of a signal f ∶ G → R and a kernel ψ ∶ G → R is given by:
[Equation]
where µ G (⋅) is the (invariant) Haar measure of the group.
The group convolution is G -equivariant inthe sense that for all u,v,w ∈ G , it holds that:
[Equation]
The lifting convolution.
Regularly, the input of a neural network is not readily defined on the groupof interest G , but on a sub-domain thereof X , i.e., f ∶ X → R .
For instance, medical images arefunctions on R 2 although equivariance to 2D-translations and planar rotations is desirable.
In thiscase X = R 2 and the group of interest is G = SE ( 2 ) .
Consequently, we must first lift the input from Xto G in order to use group convolutions.
This is achieved via the lifting convolution defined as:
[Equation]
Practical implementation of the group convolution.
The group convolution requires integrationover a continuous domain and, in general, cannot be computed in finite time. As a result, it is generallyapproximated.
Two main strategies exist to approximate group convolutions with regular group representations: group discretization [7] and Monte Carlo approximation [16].
The former approximatesthe group convolution with a fixed group discretization.
Unfortunately, the approximation becomesonly equivariant to the transformations in the discretization and not to the intrinsic continuous group.
A Monte Carlo approximation, on the other hand, ensures equivariance –in expectation– to thecontinuous group. This is done by uniformly sampling transformations { v j } , { u i } from the groupduring each forward pass, and using these transformations to approximate the group convolution:
[Equation]
Note that this Monte Carlo approximation requires the convolutional kernel ψ to be defined on thecontinuous group .
As the domain cannot be enumerated, independent weights cannot be used toparameterize the convolutional kernel.
Instead, Finzi et al.
[16] parameterize it with a small neuralnetwork, i.e., ψ ( x )=
MLP ( x ) .
This allows them to map all elements v − 1 j u i to a defined kernel value.
Partial Group Equivariant Networks3.1 (Approximate) partial group equivariance
Before defining the partial group convolution, we first formalize what we mean by partial groupequivariance.
We say that a map ϕ is partially equivariant to G , if it is equivariant to transformationsin a subset of the group S ⊂ G , but not necessarily to all transformations in the group G .
That is, if:
[Equation]
Different from equivariance to a subgroup of G –a subset of the group that also fulfills the groupaxioms–, we do not restrict the subset S to be itself a group.
As explained in detail in Sec. 3.3, partial equivariance holds, in general, only approximately , andit is exact only if S is a subgroup of G .
This results from the set S not being necessarily closedunder group actions.
In other words, partial equivariance is a relaxation of group equivariance similarto soft invariance [43]: the property of a map to be approximately invariant.
We opt for the wordpartial in the equivariance setting to emphasize that (approximate) partial group equivariance arisesby restricting the domain of the signals in a group convolution to a subset, i.e., a part, of the group.
3.2 The partial group convolution
Let S ( 1 ) , S ( 2 ) be subsets of a group G and p ( u ) be a probability distribution on the group, which isnon-zero only on S ( 2 ) .
The partial group convolution from a function f ∶ S ( 1 ) → R to a functionh ∶ S ( 2 ) → R is given by:
[Equation]
In contrast to group convolutions whose inputs and outputs are always defined on the entire group, i.e.,f,h ∶ G → R , the domain of the input and output of the partial group convolution can also be subsets of the group .
By learning these subsets, the model can become ( i ) fully equivariant ( S ( 1 ) , S ( 2 ) =
G ) , ( ii )partially equivariant ( S ( 1 ) , S ( 2 ) ≠ G ) , or ( iii ) forget some equivariances ( S ( 2 ) a subgroup of G ) .
3.3 From group convolutions to partial group convolutions
In this section, we show how group convolutions can be extended to describe partial equivariances.
Vital to our analysis is the equivariance proof of the group convolution [7, 9].
In addition, we mustdistinguish between the domains of the input and output of the group convolution, i.e., the domainsof f and h in Eq. 2.
This distinction is important because they may be different for partial groupconvolutions.
From here on, we refer to these as the input domain and the output domain .
Proposition 3.1.
Let L w f ( u )=
f ( w − 1 u ) .
The group convolution is G -equivariant in the sense that:
[Equation]
Proof.
[Equation]
127 In the first line, the change of variables ¯ v ∶= w − 1 v is used.
This is possible because the groupconvolution is a map from the group to itself, and thus if w,v ∈ G , so does w − 1 v .
Moreover, as the Haar measure is an invariant measure on the group, we have that µ G ( v )=
µ G ( ¯ v ) , for all v, ¯ v ∈ G .
Going from the group G to a subset S .
Crucial to the proof of Proposition 3.1 is the fact the groupconvolution is an operation from input signals to output signals on the group.
As a result, w − 1 u isa member of the output domain for any w ∈ G applied to the input domain.
Consequently, a grouptransformation applied to the input can be reflected by an equivalent transformation on the output.
Now, consider the case in which the output domain is not the group G , but instead an arbitrary subset
S ⊂ G , e.g., rotations in [− π 2 , π 2 ] .
Following the proof of Proposition 3.1 with u ∈ S , and v ∈ G , weobserve that the operation is equivariant to transformations w ∈ G as long as w − 1 u is a member of S .
However, if w − 1 u does not belong to the output domain S , the output of the operation cannot reflectan equivalent transformation to that of the input, and thus equivariance is not guaranteed (Fig. 1).
Bytuning the size of S , partial group convolutions can adjust their level of equivariance.
Note that equivariance is only obtained if Eq.
7 holds for all elements in the output domain.
That is, ifw − 1 u is a member of S , for all elements u ∈ S .
For partial group convolutions, this is, in general, notthe case as the output domain S is not necessarily closed under group transformations.
Nevertheless,we can precisely quantify how much the output response will change for any input transformationgiven an output domain S .
Intuitively, this difference is given by the difference in the parts of theoutput feature representation that go in and out of S by the action of input group transformations.
The stronger the transformation and the smaller the size of S , the larger the equivariance differencein the output is (Fig. 1).
The formal treatment and derivation are provided in the supplement.
Going from a subset S ( 1 ) to another subset S ( 2 ) .
Now, consider the case in which the domain ofthe input and the output are both subsets of the group, i.e., v ∈ S ( 1 ) and u ∈ S ( 2 ) .
Analogous to theprevious case, equivariance to input transformations w ∈ G holds at positions u ∈ S ( 2 ) for whichw − 1 u are also members of S ( 2 ) .
Nevertheless, the input domain is not longer restricted to be closed,and thus the input can also change for different group transformations.
To see this, consider a partial group convolution from an input subset S ( 1 ) to the group G (Fig. 2).
Even if the output domain is the group, differences in the output feature map can be seen.
This resultsfrom differences observed in the input feature map f for different group transformations of the input.
Similar to the previous case, we can precisely quantify how much the output response changes for anarbitrary subset S ( 1 ) in the input domain.
Intuitively, the difference is given by the change in the partsof the input feature representation that go in and out of S ( 2 ) by the action of the input group transformation.
The stronger the transformation and the smaller the size of S ( 1 ) , the larger the difference in theoutput is (Fig. 2).
The formal treatment and derivation of this quantity are provided in the supplement.
Special case: lifting convolutions.
To conclude, we note that the lifting convolution (Eq. 3) is aspecial case of a partial group convolution with S ( 1 ) =
X and S ( 2 ) =
G .
Note however, that the liftingconvolution is fully group equivariant.
This is because the domain of the input – X – remains equalfor all group transformations of the input. Consequently, no input component leaves the subset forany group transformation of the input.
Consequently, even if the output domain is the group, i.e., S ( 2 ) = G , a partial group convolution does not produce exactly equivariant outputs by design.
The difference from exact equivariance in the output response is proportional to the strength of the input transformation and the size of S ( 1 )
3.4 Learning group subsets via probability distributions on group elements
So far, we have discussed the properties of the partial group convolution in terms of group subsetswithout specifying how these subsets can be learned.
In this section, we describe how these subsetscan be learned by learning a certain probability distribution on the group.
We provide specificexamples for discrete groups, continuous groups, and combinations thereof.
Vital to our approach is the Monte Carlo approximation to the group convolution presented in Sec. 2:
[Equation]
As shown in the supplement, this approximation is equivariant to G in expectation if the elements inthe input and output domain are uniformly sampled from the Haar measure, i.e., u i ,v j ∼ µ G (⋅) .
Approach.
Our main observation is that we can prioritize sampling specific group elements duringthe group convolution by learning a probability distribution p ( u ) over the elements of the group.
When group convolutions draw elements uniformly from the group, each group element is drawn withequal probability and thus, the resulting approximation is fully equivariant in expectation.
However,we can also define a different probability distribution that draws some samples with larger probability.
For instance, we can sample from a certain region, e.g., rotations in [− π 2 , π 2 ] , by defining a probabilitydistribution on the group p ( u ) which is uniform in this range, but zero otherwise.
The same principlecan be used to forget an equivariance by letting this distribution collapse to a single point, e.g., theidentity, along the corresponding group dimension.
In other words, learning a probability distribution p ( u ) on the group that is non-zero only in asubset of the group can be used to effectively learn this subset.
Specifically, we define a probabilitydistribution p ( u ) on the output domain of the group convolution in order to learn a subset of the group S ( 2 ) upon which partial equivariance is defined.
Note that we only need to define a distribution onthe output domain of each layer.
This is because neural networks apply layers sequentially, and thusthe distribution on the output domain of the previous layer defines the input domain of the next layer.
Distributions for one-dimensional continuous groups.
We take inspiration from Augerino [3], anuse the reparameterization trick [26] to parameterize continuous distributions.
In particular, we usethe reparameterization trick on the Lie algebra of the group [15] to define a distribution which isuniform over a connected set of group elements [ u − 1 ,...,e,...,u ] , but zero otherwise.
To this end,we define a distribution u ⋅ [− 1 , 1 ] with learnable u on the Lie algebra g , and map it to the group viathe pushforward of the exponential map
exp ∶ g
→ G .
This give us a distribution which is uniformover a connected set of elements [ u − 1 ,...,e,...,u ] , but zero otherwise.
For instance, we can learn a distribution on the rotation group SO ( 2 ) , which is uniform between[− θ,θ ] and zero otherwise by defining a probability distribution θ ⋅ [− 1 , 1 ] with learnable θ on the
Lie algebra, and mapping it to the group.
If we parameterize group elements as scalars g ∈ [− π,π ) ,the exponential map is the identity, and thus p ( g )=
U ( θ ⋅ [− 1 , 1 )) .
If we sample group elements fromthis distribution during the calculation of the group convolution, the output domain will only containelements in [− θ,θ ) and the output feature map will be partially equivariant.
Distributions for one-dimensional discrete groups.
We can define a probability distribution on adiscrete group as the probability of sampling from all possible element combinations.
For instance,for the mirroring group { 1 , − 1 } , this distribution assigns a probability to each of the combinations { 0 , 0 } , { 0 , 1 } , { 1 , 0 } , { 1 , 1 } indicating whether the corresponding element is sampled (1) or not(0).
For a group with elements { e,g 1 ,...,g n } , however, this means sampling from 2 n + 1 elements,which is computationally expensive and potentially unstable to train.
Instead, we define elementwise Bernoulli distributions over each of the elements { g 1 ,...,g n } , and learn the probability p i ofsampling each element g i .
The probability distribution on the group can then be formulated as thejoint probability of the element-wise Bernoulli distributions p ( e,g 1 ,...,g n ) =
∏ ni = 1 p ( g i ) .
To learn the element-wise Bernoulli distributions, we use the Gumbel-Softmax trick [25, 34], and usethe Straight-Through Gumbel-Softmax to back-propagate through sampling.
If all the probabilitiesare equal to 1, i.e., { p i = 1 } ni = 1 , the group convolution will be fully equivariant.
Whenever probabilitiesstart declining, group equivariance becomes partial.
In the limit, if all probabilities become zero, i.e.,{ p i = 0 } ni = 1 , then only the identity is sampled and this equivariance is effectively forgotten.
Probability distributions for multi-dimensional groups.
There exist several multi-dimensionalgroups with important applications, such as the orthogonal group O ( 2 ) –parameterized by rotationsand mirroring–, or the dilation-rotation group –parameterized by scaling and rotations–. For multidimensional groups, we construct a probability distribution over the entire group as a combination ofindependent probability distributions along each of the group axes .
For a group G with elementsg decomposable along n dimensions g =( d 1 ,...,d n ) , we decompose the probability distribution as:p ( g )=
∏ ni = 1 p ( d i ) , where the probability p ( d i ) is defined given the type of space: continuous ordiscrete.
For instance, for the orthogonal group O ( 2 ) defined by rotations r and mirroring m , i.e.,g = ( r,m ) , r ∈ SO ( 2 ) ,m ∈ {± 1 } , we define the probability distribution on the group as p ( g )= 225 p ( r ) ⋅ p ( m ) , where p ( r ) is a continuous distribution, and p ( m ) is a discrete one as defined above.
3.5 Partial Group Equivariant Networks
To conclude this section, we illustrate the structure of Partial
G-CNNs.
We build upon Finzi et al.
[16] and extend their continuous G-CNNs to discrete groups.
This is achieved by parameterizing the convolutional kernels on (continuous) Lie groups,and applying the action of discrete groups directly on the grouprepresentation of the kernels.
In addition, we replace the isotropic lifting of Finzi et al.
[16]with lifting convolutions (Eq. 3).
Inspired by Romero et al.[38], we parameterize convolutional kernels as implicit neuralrepresentations with SIRENs [40].
This parameterization leadsto higher expressivity, faster convergence, and better accuracythan ReLU , LeakyReLU and Swish MLP parameterizationsused so far for continuous G-CNNs, e.g., [39, 16] –see Tab.
The architecture of Partial G-CNNs is shown in Fig.
3,and varies only in the number of blocks and channels used.
Related work
Group equivariant neural networks.
The seminal work of G-CNNs [7] has inspired several methodsequivariant to many different groups.
Existing methods show equivariance to planar [11, 49, 47],spherical rotations [46, 8, 12–14], scaling [48, 41, 37], and other symmetry groups [35, 4, 18].
Groupequivariant self-attention has also been proposed [19, 36, 23].
Common to these methods is that theyare fully equivariant and the group must be fixed prior to training.
Contrarily, Partial G-CNNs learntheir level of equivariance from data and can represent full, partial and no equivariance.
Invariance learning.
Learning the right amount of global invariance from data has been exploredby learning a probability distribution over continuous test-time augmentations [3], or by using themarginal likelihood of a Gaussian process [43, 42].
Contrary to these approaches, Partial G-CNNsaim to learn the right level of equivariance at every layer and do not require additional loss terms.
Partial G-CNNs relate to Augerino [3] in the form that probability distributions are defined oncontinuous groups.
However, Partial G-CNNs are intended to learn partial layer-wise equivariancesand are able to learn probability distributions on discrete groups.
There also exist learnable dataaugmentation strategies [33, 20, 32, 5] that can find transformations of the input that optimize the taskloss.
We can also view Augerino as learning a smart data augmentation technique which we comparewith.
In contrast to these methods, Partial G-CNNs find optimal partial equivariances at each layer. [Table] [Table]
Equivariance learning.
Learning equivariant mappings from data has been explored by metalearning of weight-tying matrices encoding symmetry equivariances [50, 1] and by learning the Liealgebra generators of the group jointly with the parameters of the network [10].
These approachesutilize the same learned symmetries across layers.
MSR [50] is only applicable to (small) discretegroups, and requires long training times.
L-Conv [10] is only applicable to continuous groups and isnot fully compatible with current deep learning components, e.g., pooling, normalization.
Unlikethese approaches, Partial G-CNNs can learn levels of equivariance at every layer, are fully compatiblewith current deep learning components, and are applicable for discrete groups, continuous groupsand combinations thereof.
We note, however, that Zhou et al.
[50], Dehmamy et al.
[10] learn thestructure of the group from scratch.
Contrarily, Partial G-CNNs start from a (very) large group andallows layers in the network to constrain their equivariance levels to better fit the data.
Finzi et al.
[17]incorporate soft equivariance constraints by combining outputs of equivariant and non-equivariantlayers running in parallel, which incurs in large parameter and time costs.
Differently, Partial G-CNNsaim to learn from data the optimal amount of partial equivariance directly on the group manifold.
Experiments
Experimental details.
We parameterize all our convolutional kernels as 3-layer SIRENs [40] with32 hidden units.
All our networks are constructed with 2 residual blocks of 32 channels each, batchnormalization [24] following the structure shown in Fig.
We intentionally select our networks to besimple as to better assess the effect of partial equivariance.
We avoid learning probability distributionson the translation part of the considered groups, and assume all spatial positions to be sampled inorder to use fast PyTorch convolution primitives in our implementation. 3 Additional experimentaldetails such as specific hyperparameters used and complementary results can be found in Appx.
E, F. Toy tasks: MNIST6-180 and MNIST6-M. First, we validate whether Partial G-CNNs can learnpartial equivariances.
To this end, we construct two illustrative datasets: MNIST6-180 , and MNIST6-M .
MNIST6-180 is constructed by extracting the digits of the class 6 from the MNIST dataset [30], androtating them on the circle.
The goal is to predict whether the number is a six, i.e., a rotation in[− 90 ○ , 90 ○ ] was applied, or a nine otherwise.
Similarly, we construct MNIST6-M by mirroring digitsover the y axis.
The the goal is to predict whether a digit was mirrored or not.
As shown in Tab. 1, G-CNNs are unable to solve these tasks as discrimination among group transformations is required.
Specifically, SE ( 2 ) -CNNs are unable to solve MNIST6-180, and Mirror -CNNs–G-CNNs equivariant to reflections– are unable to solve MNIST6-M.
Furthermore, E ( 2 ) -CNNscannot solve any of the two tasks, because E ( 2 ) -CNNs incorporate equivariance to both rotations andreflections.
Partial G-CNNs, on the other hand, easily solve both tasks with corresponding base groups.
This indicates that Partial G-CNNs learn to adjust the equivariance levels in order to solve the tasks.
In addition, we verify the learned levels of equivariance for a Partial SE ( 2 ) -CNN on MNIST6-180.
To this end, we plot the probability of assigning the label 6 to test samples of MNIST6-180 rotated onthe whole circle.
Fig 4 shows that the network learns to predict “6” for rotated samples in [− 90 ○ , 90 ○ ] ,and “9” otherwise.
Note that Partial G-CNNs learn the expected levels of partial equivariance withoutadding regularization loss terms to encourage them as in Benton et al.
Benchmark image datasets.
Next, we validate Partial G-CNNs on classification datasets: RotMNIST[29], CIFAR-10 and CIFAR-100 [28].
Results on PatchCam [44] can be found in Appx. F.
We construct Partial G-CNNs with base groups SE ( 2 ) and E ( 2 ) , and varying number of elementsused in the Monte Carlo approximation of the group convolution and compare them to equivalent
G-CNNs and ResNets (equivalent to T ( 2 ) -CNNs).
Our results (Tab. 3) show that Partial G-CNNsare competitive with G-CNNs when full-equivariance is advantageous (rotated MNIST and Patch-
[Table]
Regular G-CNNs cannot solve the task
Camelyon).
However, for tasks in which the data does not naturally exhibit full rotation equivariance(CIFAR-10 and CIFAR-100), Partial G-CNNs consistently outperform fully equivariant G-CNNs.
The need for learning partial layer-wise equivariances.
Next, we evaluate ( i ) the effect of learningpartial equivariances instead of soft invariances, and ( ii ) the effect of learning layer-wise levels ofequivariances instead of a single level of partial equivariance for the entire network.
For the former, we compare Partial G-CNNs to equivalent ResNets with Augerino [3] (Tab. 2).
Weextend our strategy to learn distributions on discrete groups to the Augerino framework to allow it tohandle groups with discrete components, e.g., E ( 2 ) .
For the latter, we construct regular G-CNNs andreplace either the final group convolutional layer by a T ( 2 ) convolutional layer, or the global maxpooling layer at the end by a learnable MLP (Tab. 4).
If determining the level of equivariance at theend of the network is sufficient, these models should perform comparably to Partial G-CNNs.
Tab. 2 shows that Augerino is competitive to Partial G-CNNs on rotated MNIST, but falls behind by alarge margin on CIFAR-10 and CIFAR-100.
This result can be explained by how these datasets areconstructed.
RotMNIST is constructed by rotating MNIST digits globally, thus it is not surprisingthat a model able to encode global invariances can match Partial G-CNNs.
The invariance andequivariance relationships in natural images, however, are more complex, as they can be local as well.
Consequently, tackling different levels of equivariance at each layer using Partial G-CNNs leads tobenefits over using a single level of global invariance for the entire network.
Although the T ( 2 ) and MLP alternatives outlined before could solve the MNIST-180 and MNIST-Mtoy datasets, we observed that Partial G-CNNs perform consistently better on the visual benchmarksconsidered (Tab. 4).
This indicates that learning layer-wise partial equivariances is beneficial overmodifying the level of equivariance only at the end of the model.
In addition, it is important tohighlight that Partial G-CNNs can become fully-, partial-, and non-equivariant during training. Alternative models, on the other hand, are either unable to become fully equivariant ( T ( 2 ) models)or very unlikely to do so in practice ( MLP models).
SIRENs as group convolution kernels.
To conclude, we validate SIRENs as parameterization forgroup convolutional kernels.
Tab.
6 shows that SE ( 2 ) -CNNs with SIREN kernels outperform SE ( 2 ) -
CNNs with ReLU , LeakyReLU and Swish kernels by a large margin on all datasets considered.
Discussion
Memory consumption in partial equivariant settings.
G-CNNs fix the number of samples used toapproximate the group convolution prior to training.
In Partial G-CNNs we fix a maximum number ofsamples and adjust the number used at every layer based on the group subset learned.
Consequently,a partial group convolution with a learned distribution p ( u )=
U ( π 2 [− 1 , 1 ]) uses half of the elementsused in a corresponding group convolution.
This reduction in memory and execution time leads toimprovements in training and inference time for Partial G-CNNs on partial equivariant settings.
Weobserve reductions up to 2.5 × in execution time and memory usage on CIFAR-10 and CIFAR-100.
Sampling per batch element.
In our experiments, we sample once from the learned distributionp ( u ) at every layer, and use this sample for all elements in the batch.
A better estimation of p ( u )can be obtained by drawing a sample per batch element.
Though this method may lead to fasterconvergence and better estimations of the learned distributions, it comes at a prohibitive memory cost
[Table] as independent convolutional kernels must be rendered per batch element. Consequently, we onlyconsider a single sample at each layer for the entire batch in our experiments. Better kernels with implicit neural representations.
Inspired by Romero et al.
[38], we replace
LeakyReLU , ReLU and Swish kernels used so far for continuous group convolution kernels with a
SIREN [40].
Our results show that SIRENs are better at modelling group convolutional kernels.
Going from a small group subset to a larger one, e.g., the group G .
What does it mean andwhy is it advantageous?
In Sec.
3.3 we described that a partial group convolution can go from agroup subset S ( 1 ) to a larger group subset S ( 2 ) , e.g., the whole group G .
Nevertheless, once a layerbecomes partially equivariant, subsequent layers cannot become fully equivariant even for S ( 2 ) =
G .
Interestingly, we observe that Partial G-CNNs often learn to disrupt equivariance halfway in thenetwork, and return to the whole group afterwards (Fig. 5).
This behavior is actually advantageous.
Full equivariance restricts group convolutions to apply the same mapping on the entire group.
As aresult, once the input is transformed, the output remains equal up to the same group transformations.
Inpartial equivariance settings, Partial G-CNNs can output different feature representations for differentinput transformations (Figs. 1, 2).
Consequently, Partial G-CNNs can use the group dimensionto encode different feature mappings.
Specifically, some kernel values are used for some inputtransformations and other ones are used for other input transformations.
This means that when Partial G-CNNs go back to a larger group subset from a smaller one, they are able to use the group axis toencode additional transformation-dependent features, thus resulting in increased model expressivity.
In Appx. F, we evaluate the effect of enforcing monotonically decreasing group subsets as a functionof depth.
That is, Partial G-CNNs whose subsets at deeper layers are equal or smaller than those at previous ones.
Our results show that this monotonicity actually leads to slightly worse results comparedto the unconstrained case, thus supporting the use of unconstrained learning of group subsets.
Limitations and future work
Partial equivariances for other group representations.
The theory of learnable partial equivariancesproposed here is only applicable to architectures using regular group representations, e.g., [7, 36].
Nevertheless, other type of representations exist with which exact equivariance to continuous groupscan be obtained: irreducible representations [49, 47, 45].
We consider extending the learning ofpartial equivariances to irreducible representations a valuable extension of our work.
Unstable training on discrete groups.
Although we can model partial equivariance on discretegroups with our proposed discrete probability distribution parameterization, we observed that thesedistributions can be unstable to train.
To cope with this, we utilize a 10x lower learning rate for theparameters of the probability distributions (See Appx.
E.3 for details). Nevertheless, finding goodways of learning discrete distributions is an active field of research [21, 22], and advances in this fieldcould be used to further improve the learning of partial equivariances on discrete groups.
Scaling partial equivariance to large groups. Arguably the main limitation of G-CNNs withregular representations is their computational and memory complexity, which prevents the use of verylarge groups, e.g., simultaneous rotation, scaling, mirroring and translations. Partial equivarianceis particularly promising for large groups as the network is initialized with a prior towards beingequivariant to the entire group, but is able to focus on those relevant to the task at hand.
We considerlearning partial equivariances on large groups an interesting direction for further research. Groupconvolutions can be scaled to large groups with, e.g., separable kernels [31, 27]. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments.
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[No]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes]
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[No]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]