S TEERABLE P ARTIAL D IFFERENTIAL O PERATORS FOR E QUIVARIANT N EURAL N ETWORKS
A BSTRACT
Recent work in equivariant deep learning bears strong similarities to physics.
Fields over a base space are fundamental entities in both subjects, as are equivariant maps between these ﬁelds.
In deep learning, however, these maps are usually deﬁned by convolutions with a kernel, whereas they are partial differential operators (PDOs) in physics.
Developing the theory of equivariant PDOs in the context of deep learning could bring these subjects even closer together and lead to a stronger ﬂow of ideas.
In this work, we derive a G -steerability constraint that completely characterizes when a PDO between feature vector ﬁelds is equivariant, for arbitrary symmetry groups G .
We then fully solve this constraint for several important groups.
We use our solutions as equivariant drop-in replacements for convolutional layers and benchmark them in that role.
Finally, we develop a framework for equivariant maps based on Schwartz distributions that uniﬁes classical convolutions and differential operators and gives insight about the relation between the two.
I NTRODUCTION
In many machine learning tasks, the data exhibits certain symmetries, such as translation- and sometimes rotation-invariance in image classiﬁcation.
To exploit those symmetries, equivariant neural networks have been widely studied and successfully applied in the past years, beginning with
Group convolutional neural networks (Cohen & Welling, 2016; Weiler et al., 2018b).
A signiﬁcant generalization of Group convolutional networks is given by steerable CNNs (Cohen & Welling, 2017;
Weiler et al., 2018a; Weiler & Cesa, 2019), which unify many different pre-existing equivariant models (Weiler & Cesa, 2019).
They do this by representing features as ﬁelds of feature vectors over a base space, such as R 2 in the case of two-dimensional images.
Layers are then linear equivariant maps between these ﬁelds.
This is very reminiscent of physics.
There, ﬁelds are used to model particles and their interactions, with physical space or spacetime as the base space.
The maps between these ﬁelds are also equivariant, with the symmetries being part of fundamental physical laws.
It is also noteworthy that these symmetries are largely ones that appear the most often in deep learning, such as translation and rotation equivariance.
These similarities have already led to ideas from physics being applied in equivariant deep learning (Lang & Weiler, 2021).
However, one remaining difference is that physics uses equivariant partial differential operators (PDOs) to deﬁne maps between ﬁelds, such as the gradient or Laplacian.
Therefore, using PDOs instead of convolutions in deep learning would complete the analogy to physics and could lead to even more transfer of ideas between subjects.
Equivariant PDO-based networks have already been designed in prior work (Shen et al., 2020; Smets et al., 2020; Sharp et al., 2020).
Most relevant for our work are PDO-eConvs
(Shen et al., 2020), which can be seen as the PDO-analogon of group convolutions.
However, PDO-eConvs are only one instance of equivariant PDOs and do not cover the most common PDOs from physics, such as the gradient, divergence, etc.
Very similarly to how steerable CNNs (Cohen & Welling, 2017; Weiler et al., 2018a; Weiler & Cesa, 2019) generalize group convolutions, we generalize PDO-eConvs by characterizing the set of all translation equivariant PDOs between feature ﬁelds over Euclidean spaces.
Because of this analogy, we dub these equivariant differential operators steerable PDOs .
These steerable PDOs and their similarity to steerable CNNs also raise the question of how equivariant
PDOs and kernels relate to each other, and whether they can be uniﬁed.
We present a framework for equivariant maps that contains both steerable PDOs and convolutions with steerable kernels as special cases.
We then prove that this framework deﬁnes the most general set of translation equivariant, linear, continuous maps between feature ﬁelds, complementing recent work (Aronsson, 2021) that describes when equivariant maps are convolutions.
Since formally developing this framework requires the theory of Schwartz distributions, we cover it mainly in Appendix E, and the main paper can be read without any knowledge of distributions.
However, we reference the main results from this framework in the paper where appropriate.
In order to make steerable PDOs practically applicable, we describe an approach to ﬁnd complete bases for vector spaces of equivariant PDOs and then apply this method to the most important cases.
We have also implemented steerable PDOs for all subgroups of O(2) ( https://github.com/ ejnnr/steerable_pdos ).
Our code extends the E2CNN library 1 (Weiler & Cesa, 2019), which will allow practitioners to easily use both steerable kernels and steerable PDOs within the same library, and even to combine both inside the same network.
Finally, we test our approach empirically by comparing steerable PDOs to steerable CNNs.
In particular, we benchmark different discretization methods for the numerical implementation.
In summary, our main contributions are as follows:
• We develop the theory of equivariant PDOs on Euclidean spaces, giving a practical characterization of precisely when a PDO is equivariant under any given symmetry.
• We unify equivariant PDOs and kernels into one framework that provably contains all translation equivariant, linear, continuous maps between feature spaces.
• We describe a method for ﬁnding bases of the vector spaces of equivariant PDOs, and provide explicit bases for many important cases.
• We benchmark steerable PDOs using different discretization procedures and provide an implementation of steerable PDOs as an extension of the E2CNN library.
1. R ELATED WORK
Equivariant convolutional networks Our approach to equivariance follows the one taken by steerable CNNs (Cohen & Welling, 2017; Weiler et al., 2018a; Weiler & Cesa, 2019;
Brandstetter et al., 2021).
They represent each feature as a map from the base space, such as R d , to a ﬁber R c that is equipped with a representation ρ of the point group G .
Compared to vanilla CNNs, which have ﬁber R , steerable CNNs thus extend the codomain of feature maps.
A different approach is taken by group convolutional networks (Cohen & Welling, 2016;
Hoogeboom et al., 2018; Weiler et al., 2018b).
They represent each feature as a map from a group H acting on the input space to R .
Because the input to the network usually does not lie in H , this requires a lifting map from the input space to H .
Compared to vanilla CNNs, group convolutional networks can thus be understood as extending the domain of feature maps.
When H = R d (cid:111) G is the semidirect product of the translation group and a pointwise group G , then group convolutions on H are equivalent to G -steerable convolutions with regular representations.
For ﬁnite G , the group convolution over G simply becomes a ﬁnite sum.
LieConvs (Finzi et al., 2020) describe a way of implementing group convolutions even for inﬁnite groups by using a Monte
Carlo approximation for the convolution integral.
Steerable CNNs with regular representations would have to use similar approximations for inﬁnite groups, but they can instead also use (non-regular) ﬁnite-dimensional representations.
Both the group convolutional and the steerable approach can be applied to non-Euclidean input spaces—LieConvs deﬁne group convolutions on arbitrary Lie groups and steerable convolutions can be deﬁned on Riemannian manifolds (Cohen et al., 2019b; Weiler et al., 2021) and homogeneous spaces (Cohen et al., 2019a).
One practical advantage of the group convolutional approach employed by LieConvs is that it doesn’t require solving any equivariance constraints, which tends to make implementation of new groups easier.
They also require somewhat less heavy theoretical machinery.
On the other hand, steerable
CNNs are much more general.
This makes them interesting from a theoretical angle and also has more practical advantages; for example, they can naturally represent the symmetries of vector ﬁeld input or output.
Since our focus is developing the theory of equivariant PDOs and the connection to physics, where vector ﬁelds are ubiquitous, we are taking the steerable perspective in this paper.
Equivariant PDO-based networks The work most closely related to ours are PDO-eConvs (Shen et al., 2020), which apply the group convolutional perspective to PDOs.
Unlike LieConvs, they are not designed to work with inﬁnite groups.
The steerable PDOs we introduce generalizes PDO-eConvs, which are obtained as a special case by using regular representations.
A different approach to equivariant PDO-based networks was taken by Smets et al.
Instead of applying a differential operator to input features, they use layers that map an initial condition for a
PDE to its solution at a ﬁxed later time.
The PDE has a ﬁxed form but several learnable parameters and constraints on these parameters—combined with the form of the PDE—guarantee equivariance.
Sharp et al.
(2020) also use a PDE, namely the diffusion equation, as part of their DiffusionNet model, which can learn on 3D surfaces.
Interestingly, the time evolution operator for the diffusion equation is exp( t ∆) , which can be interpreted as an inﬁnite power series in the Laplacian, very reminiscent of the ﬁnite Laplacian polynomials that naturally appear throughout this paper.
Studying the equivariance of such inﬁnite series of PDOs might be an interesting direction for future work.
We clarify the relation between PDO-based and kernel-based networks in some more detail in Appendix F.
S TEERABLE PDO S
In this section, we develop the theory of equivariant PDOs.
We will represent all features as smooth ﬁelds f : R d → R c that associate a feature vector f ( x ) ∈ R c , called the ﬁber at x , with each point x ∈ R d .
We write
F i = C ∞ ( R d , R c i ) for the space of these ﬁelds f in layer i .
Additionally, we have a group of transformations acting on the input space R d , which describes under which symmetries we want the PDOs to be equivariant.
We will always use a group of the form H =
( R d , +) (cid:111) G , for some G ≤ GL( d, R ) .
Here, ( R d , +) refers to the group of translations of R d , while G is some group of linear invertible transformations.
The full group of symmetries H is the semidirect product of these two, meaning that each element h ∈ H can be uniquely written as h
= tg , where
t ∈ R
d is a translation and g ∈ G a linear transformation.
For example, if G = { e } is the trivial group, we consider only equivariance under translations, as in classical CNNs, while for G = SO( d ) we additionally consider rotational equivariance.
Each feature space F i has an associated group representation ρ
i : G → GL( c i , R ) , which determines how each ﬁber R c i transforms under transformations of the input space.
Brieﬂy, ρ i associates an invertible matrix
ρ i ( g ) to each group element g , such that ρ i ( g )
ρ i ( g (cid:48) )
=
ρ i ( gg (cid:48) ) ; more details on representation theory can be found in Appendix B.
To see why these representations are necessary, consider the feature space F = C ∞ ( R 2 , R 2 ) and the group G = SO(2) .
The two channels could simply be two independent scalar ﬁelds, meaning that rotations of the input move each ﬁber but do not transform the ﬁbers themselves.
Formally, this would mean using trivial representations ρ ( g )
= 1 for both channels.
On the other hand, the two channels could together form a vector ﬁeld, which means that each ﬁber would need to itself be rotated in addition to being moved.
This would correspond to the representation ρ ( g ) =
g .
These two cases are visualized in Fig.
In general, the transformation of a feature f ∈ F i under an input transformation tg with t ∈ R d and g ∈ G is given by
[Equation]
The g − 1 ( x − t ) term moves each ﬁber spatially, whereas the ρ i ( g ) is responsible for the individual transformation of each ﬁber.
For a network, we will need maps between adjacent feature spaces F i and F i +1 .
Since during this theory section, we only consider single layers in isolation, we will drop the index i and simply denote the layer map as Φ : F in → F out .
We are particularly interested in equivariant maps Φ , i.e. maps that commute with the action of H on the feature spaces:
[Equation]
We call Φ translation -equivariant if Eq.
(2) holds for h ∈ ( R d , +) , i.e. for pure translations.
Analogously, Φ is G -equivariant if it holds for linear transformations h ∈ G .
Because H is the semidirect product of R d and G , a map is H -equivariant
if and only if it is both translation- and G -equivariant.
2. PDO S AS MAPS BETWEEN FEATURE SPACES
We want to use PDOs for the layer map Φ , so we need to introduce some notation for PDOs between multi-dimensional feature ﬁelds.
As shown in Fig.
1, such a multi-dimensional PDO can be interpreted as a matrix of one-dimensional PDOs.
Speciﬁcally, a PDO from C ∞ ( R d , R c in ) to C ∞ ( R d , R c out ) is described by a c out × c in matrix.
For example, the 2D divergence operator, which maps from R 2 to R =
R 1 can be written as the 1 × 2 matrix ( ∂∂ 2 ) .
This is exactly analogous to convolutional kernels, which can also be interpreted as c out × c in matrices of scalar-valued kernels.
To work with the one-dimensional PDOs that make up the entries of this matrix, we use multi-index notation, so for a tuple
α = ( α 1 , . .
. , α d ) ∈ N d 0 , we write ∂ α :=
∂ α 1 1 . . .
∂ α d d .
A general onedimensional PDO is a sum (cid:80)
α c α ∂ α , where the coefﬁcients c α are smooth functions
c α : R d → R (so for now no spatial weight sharing is assumed).
The sum ranges over all multi-indices α , but we require all but a ﬁnite number of coefﬁcients to be zero everywhere, so the sum is effectively ﬁnite.
As described, a PDO between general feature spaces is then a matrix of these one-dimensional PDOs.
2. E QUIVARIANCE CONSTRAINT FOR PDO S
We now derive a complete characterization of the PDOs that are H -equivariant in the sense deﬁned by
Eq.
Because a map is equivariant under the full symmetries H
=
( R d , +) (cid:111)
G if and only if it is both translation equivariant and G -equivariant, we split up our treatment into these two requirements.
First, we note that translation equivariance corresponds to spatial weight sharing, just like in CNNs (see Appendix G for the proof):
Proposition 1.
A c out × c in -PDO Φ with matrix entries (cid:80)
α c ijα ∂ α is translation equivariant if and only if all coefﬁcients c ijα are constants, i.e. c ijα ( x ) =
c ijα ( x (cid:48) ) for all x, x (cid:48) ∈ R d .
So from now on, we restrict our attention to PDOs with constant coefﬁcients and ask under which circumstances they are additionally equivariant under the action of the point group G ≤ GL( d, R ) .
To answer that, we make use of a duality between polynomials and PDOs that will appear throughout this paper: for a PDO (cid:80)
α c α ∂ α , with c α ∈ R , 2 there is an associated polynomial simply given by
(cid:80)
α c α x α , where x α :=
x α 1 1 . . .
x α d d .
Conversely, for any polynomial p = (cid:80)
α c α x α , we get a PDO by formally plugging in ∂ =
( ∂ 1 , . . . ,
∂ d ) for x , yielding p ( ∂ ) :
= (cid:80)
α c α ∂ α .
We will denote the map in this direction by D , so we write D ( p ) :
= p ( ∂ ) .
We can extend this duality to PDOs between multi-dimensional ﬁelds: for a matrix P of polynomials, we deﬁne D ( P ) component-wise, so D ( P ) will be a matrix of PDOs given by D ( P ) ij :
= D ( P ij ) .
To avoid confusion, we will always denote polynomials by lowercase letters, such as p, q , and matrices of polynomials by uppercase letters, like P and Q .
As a simple example of the correspondence between polynomials and PDOs, the Laplacian operator ∆ is given by ∆ =
D ( | x | 2 ) .
The gradient ( ∂ 1 , . . . ,
∂ d ) T is induced by the d × 1 matrix ( x 1 , . . .
, x d ) T and the 3D curl ( d = 3 ) is induced by the 3 × 3 matrix [Equation]
[Equation]
Finally, note that D is a ring isomorphism, i.e. a bijection with D ( p + q )
=
D ( p ) + D ( q ) and
D ( pq ) =
D ( p ) ◦ D ( q ) , allowing us to switch viewpoints at will.
We are now ready to state our main result on the G -equivariance of PDOs:
Theorem 2.
For any matrix of polynomials P , the differential operator D ( P ) is G -equivariant
if and only if it satisﬁes the PDO G -steerability constraint ,
[Equation]
P ( x ) just means that x is plugged into the polynomials in each entry of P , which results in a real-valued matrix P ( x ) ∈ R c out × c in , and similarly for P (cid:0) ( g − 1 ) T x (cid:1) .
Appendix C provides more intuition for the action of group elements on polynomials.
Because D is an isomorphism, this result completely characterizes when a PDO with constant coefﬁcients is G -equivariant, and in conjunction with Proposition 1, when any PDO between feature ﬁelds is H -equivariant.
The proof of Theoremcan again be found in Appendix G.
To avoid confusion, we would like to point out that this description of PDOs as polynomials is only a useful trick that lets us express certain operations more easily and will later let us connect steerable
PDOs to steerable kernels.
Convolving with these polynomials is not a meaningful operation; they only become useful when ∂ is plugged in for x .
2. E XAMPLES OF EQUIVARIANT PDO S
To build intuition, we explore the space of equivariant PDOs for two simple cases before covering the general solution.
Consider R 2 as a base space with the symmetry group G = SO(2) and trivial representations ρ in and ρ out , i.e. maps between two scalar ﬁelds.
The equivariance condition then becomes p ( gx )
= p ( x ) , so the polynomial p has to be rotation invariant.
This is the case if and only if it can be written as a function of | x | 2 , i.e. p ( x ) =
q ( | x | 2 ) for some q : R ≥
→ R .
Since we want p to be a polynomial, q needs to be a polynomial in one variable.
Because | x | 2 = x 21 + x 22 , we get the
PDO D ( p ) =
q (∆) , where ∆ := ∂ 21 + ∂ 22 is the Laplace operator.
So the SO(2) -equivariant PDOs between two scalar ﬁelds are exactly the polynomials in the Laplacian, such as 2∆ 2 + ∆ + 3 .
As a second example, consider PDOs that map from a vector to a scalar ﬁeld, still with SO(2) as the symmetry group.
There are two such PDOs that often occur in the natural sciences, namely the divergence, div v := ∂
1 v 1 + ∂ 2 v 2 , and the 2D curl, curl 2D v :=
∂ 1 v 2 − ∂ 2 v 1 .
Both of these are SO(2) -equivariant
(see Appendix A).
We get additional equivariant PDOs by composing with equivariant scalar-to-scalar maps, i.e. polynomials in the Laplacian.
Speciﬁcally, q (∆) ◦ div and q (∆) ◦ curl 2D are also equivariant vector-to-scalar PDOs, for any polynomial q .
We will omit the ◦ from now on.
We show in Appendix A that these PDOs already span the complete space of SO(2) -equivariant
PDOs from vector to scalar ﬁelds.
Explicitly, the equivariant PDOs in this setting are all of the form q 1 (∆) div + q 2 (∆) curl 2D for polynomials
q 1 and q 2 .
One example of this is the PDO shown in Fig.
In these examples, as well as in other simple cases (see Appendix A), the equivariant PDOs are all combinations of well-known operators such as the divergence and Laplacian.
That the notion of equivariance can reproduce all the intuitively “natural” differential operators suggests that it captures the right concept.
B ASES FOR SPACES OF STEERABLE PDO S
For the purposes of deep learning, we need to be able to learn steerable PDOs.
To illustrate how to achieve this, consider again SO(2)
-equivariant PDOs mapping from vector to scalar ﬁeld.
We have seen in Section 2.3 that they all have the form
[Equation]
In practice, we will need to limit the order of the PDOs we consider, so that we can discretize them.
For example, we could consider only polynomials q 1 and q 2 of up to ﬁrst order, i.e. q 1 ( z ) =
c 1 z + cand q 2 ( z ) =
c 3 z
+ c 4 .
This leads to PDOs of up to order three (since ∆ is a second order PDO and div and curl 2D are ﬁrst order).
The space of such equivariant PDOs is then
[Equation]
We can now train the real-valued parameters c 1 , . . .
, c 4 and thereby learn arbitrary equivariant PDOs of up to order three.
The general principle is that we need to ﬁnd a basis of the real vector space of steerable PDOs; then we learn weights for a linear combination of the basis elements, yielding arbitrary equivariant PDOs.
Different group representations are popular in equivariant deep learning practice; for example PDOeConvs and group convolutions correspond to so-called regular representations (see Appendix M) but quotient representations have also been used very successfully (Weiler & Cesa, 2019).
We therefore want to ﬁnd bases of steerable PDOs for arbitrary representations ρ in and ρ out .
To do so, we make use of the existing work on solving the closely related G -steerability constraint for kernels .
In this section, we will ﬁrst give a brief overview of this kernel steerability constraint and then describe how to transfer its solutions to ones of the PDO steerability constraint.
3. T HE STEERABILITY CONSTRAINT FOR KERNELS
The kernel G -steerability constraint characterizes when convolution with a kernel κ : R d → R c out × c in is G -equivariant, like the PDO steerability constraint Eq.
(4) does for PDOs.
Namely,
[Equation]
has to hold.
This constraint was proven in (Weiler et al., 2018a) for orthogonal G and later in (Weiler et al., 2021) for general G .
It is very similar to the PDO steerability constraint: the only differences are the determinant and the κ ( gx ) term on the LHS where we had ( g − 1 ) T instead of g .
An explanation for this similarity is that both constraints can be seen as special cases of a more general steerability constraint for Schwartz distributions , which we derive in Appendix E. Schwartz distributions are a generalization of classical functions and convolutions with Schwartz distributions can represent both classical convolutions and PDOs.
As we prove in Appendix E, such distributional convolutions are the most general translation equivariant, continuous, linear maps between feature spaces, strictly more general than either PDOs or classical kernels.
We also show how steerable
PDOs can be interpreted as the Fourier transform of steerable kernels, which we use to explain the remaining differences between the two steerability constraints.
But for the purposes of this section, we want to draw particular attention to the fact that for G ≤ O( d ) , the two constraints become exactly identical: the determinant then becomes 1 , and ( g − 1 ) T = g .
So for this case, which is by far the most practically important one, we will use existing solutions of the kernel steerability constraint to ﬁnd a complete basis of equivariant PDOs.
3. T RANSFERRING KERNEL SOLUTIONS TO STEERABLE PDO BASES
Solutions of the kernel steerability constraint have been published for subgroups of O(2) (Lang &
Weiler, 2021; Weiler & Cesa, 2019) and O(3) (Lang & Weiler, 2021; Weiler et al., 2018a), and they all use a basis of the form
[Equation]
where A and B are index sets for the radial and angular part and the ϕ α span the entire space of radial functions.
The reason is that the steerability constraint Eq.
(7) constrains only the angular part if
G ≤ O( d ) , because orthogonal groups preserve distances.
The angular functions χ β are only deﬁned on the sphere S d − 1 .
But crucially, we show in Appendix J that they can all be canonically extended to polynomials deﬁned on R d .
Concretely, we deﬁne ˜ χ
β ( x ) :=
| x | l β χ β ( x/ | x | ) , where l β is chosen minimally
such that ˜ χ β is a polynomial.
What we prove in Appendix J is that such an l β always exists.
The radial part ϕ α in the kernel basis consists of unrestricted radial functions.
To get a basis for the space of polynomial steerable kernels, it is enough to use only powers of | x | 2 for the radial part.
Speciﬁcally, we show in Appendix H that
[Equation]
[Equation]
is a basis for the space of polynomial steerable kernels.
As a ﬁnal step, we interpret each polynomial as a PDO using the isomorphism D deﬁned in Section 2.2, yielding a complete basis of the space of steerable PDOs.
The | x | 2 k terms become Laplacian powers ∆ k , while in the examples from
Section 2.3, ˜ χ β corresponds to divergence and curl.
In Appendix I, we apply this procedure to subgroups of O(2) and O(3) to obtain concrete solutions.
E XPERIMENTS
Implementation We developed the theory of steerable PDOs in a continuous setting, but for implementation the PDOs need to be discretized, just like steerable kernels.
The method of discretization is completely independent of the steerable PDO basis, so steerable PDOs can be combined with any discretization procedure.
We compare three methods, ﬁnite differences (FD), radial basis function ﬁnite differences (RBF-FD) and Gaussian derivatives .
Finite differences are a generalization of the usual central difference approximation and are the method used by PDO-eConvs (Shen et al., 2020).
RBF-FD ﬁnds stencils by demanding that the discretization should become exact when applied to radial basis functions placed on the stencil points.
Its advantage over FD is that it can be applied to structureless point clouds rather than only to regular grids.
Gaussian derivative stencils work by placing a Gaussian on the target point and then evaluating its derivative on the stencil points.
Like RBF-FD, this also works on point clouds, and the Gaussian also has a slight smoothing effect, which is why this discretization is often used in Computer Vision.
Formal descriptions of all three discretization methods can be found in Appendix L.
In addition to discretization, the inﬁnite basis of steerable PDOs or kernels needs to be restricted to a ﬁnite subspace.
For kernels, we use the bandlimiting ﬁlters by Weiler & Cesa (2019).
For PDOs, we limit the total derivative order to two for 3 × 3 stencils and to three for 5 × 5 stencils (except for PDO-eConvs, where we use the original basis that limits the maximum order of partial derivatives).
Finally, steerable PDOs and steerable kernels only replace the convolutional layers in a classical CNN.
To achieve an equivariant network, all the other layers, such as nonlinearities or Batchnorm also need to be equivariant.
Weiler & Cesa (2019) discuss in details how this can be achieved for various types of layers.
In our experiments, we use exactly the same implementation they do.
Care also needs to be taken with biases in the PDO layers.
Here, we again follow (Weiler & Cesa, 2019) by adding a bias only to the trivial irreducible representations that up ρ out [Table]
MNIST We ﬁrst bench [Table] mark steerable PDOs on rotated
MNIST (Larochelle et al., 2007), which consists of MNIST images that have been rotated by different angles, with 12k train and 50k test images.
Our results can be found in Table 1.
The models with 5 × 5 stencils use an architecture that Weiler & Cesa (2019) used for steerable CNNs, with six
C 16 -equivariant layers followed by two fully connected layers.
The ﬁrst column gives the representation under which the six equivariant layers transform (see Appendix B for their deﬁnitions).
PDO-eConvs implicitly use regular representations (see Appendix M), but with a slightly different basis than the one we present, so we test both bases.
We also tested models that are
D 16 -equivariant in their ﬁrst layers and C 16 -equivariant in their last one but did not ﬁnd any improvements, see Appendix N. For the models with 3 × 3 stencils, we use eight instead of six C 16 -equivariant layers, in order to compensate for the smaller receptive ﬁeld and keep the parameter count comparable.
The remaining differences between kernel and PDO parameter counts come from the fact that the basis restrictions necessarily work slightly differently (via bandlimiting ﬁlters or derivative order restriction respectively).
All models were trained with 30 epochs and hyperparameters based on those by Weiler & Cesa (2019), though we changed the learning rate schedule and regularization slightly because this improved performance for all models, including kernel-based ones.
The training data is augmented with random rotations.
Precise descriptions of the architecture and hyperparameters can be found in Appendix O.
[Table] STL-10 The rotated MNIST dataset has global rotational symmetry by design, so it is unsurprising that equivariant models perform well.
But interestingly, rotation equivariance can also help for natural images without global rotational symmetry (Weiler & Cesa, 2019;
Shen et al., 2020).
We therefore benchmark steerable PDOs on STL-10 (Coates et al., 2011), where we only use the labeled portion of 5000 training images.
The results are shown in Table 2.
The model architecture and hyperparameters are exactly the same as in (Weiler & Cesa, 2019), namely a Wide-ResNet-16-8 trained for 1000 epochs with random crops, horizontal ﬂips and Cutout (DeVries & Taylor, 2017) as data augmentation.
The group column describes the equivariance group in each of the three residual blocks.
For example, D 8 D 4 D 1 means that the ﬁrst block is equivariant under reﬂections and 8 rotations, the second under 4 rotations and the last one only under reﬂections.
All layers use regular representations.
The D 8 -equivariant layers use 5 × 5 ﬁlters to improve equivariance, whereas the other layers use 3 × 3 ﬁlters.
Fluid ﬂow prediction In the previously described tasks, the input and output representations are all trivial.
To showcase the use of non-trivial output representations, we predict laminar ﬂuid ﬂow around various objects, following Ribeiro et al. In this case, the network outputs a vector ﬁeld , which behaves differently under rotations than scalar outputs, and whose equivariance cannot be represented using PDO-eConvs, since they only implement trivial and regular representations. We ﬁnd that equivariance signiﬁcantly improves performance for both kernels and PDOs, compared to vanilla non-equivariant versions. See Appendix N for details.
Equivariance errors In the continuum, steerable CNNs and steerable PDOs are both exactly equivariant.
But the discretization on a square grid leads to unavoidable equivariance errors for rotations that aren’t multiples of π 2 .
The violation of equivariance in practice is thus closely connected to the discretization error.
For ﬁnite differences, the discretization error is particularly easy to bound asymptotically, and as pointed out by Shen et al.
(2020), this places the same asymptotic bound on the equivariance error.
However, our experiments show that empirically, ﬁnite differences don’t lead to a particularly low equivariance error (kernels and all PDO discretizations perform similarly).
See
Table 4 in Appendix N for details.
Locality of PDOs While all equivariant models improve signiﬁcantly over the non-equivariant
CNN, the method of discretization plays an important role for PDOs.
The reason that FD and RBF-FD underperform kernels is that they don’t make full use of the stencil, since PDOs are inherently local operators.
When a 5 × 5 stencil is used, the outermost entries are all very small compared to the inner ones, and even in 3 × 3 kernels, the four corners tend to be closer to zero (see Appendix N for images of stencils to illustrate this).
Gaussian discretization performs signiﬁcantly better and almost as well as kernels because its smoothing effect alleviates these issues.
This ﬁts the observation that kernels and Gaussian methods proﬁt from using 5 × 5 kernels, whereas these do not help for FD and RBF-FD (and in fact decrease performance because of the smaller number of layers).
C ONCLUSION
We have described a general framework for equivariant PDOs acting on feature ﬁelds over Euclidean space.
With this framework, we found strong similarities between equivariant PDOs and equivariant convolutions, even unifying the two using convolutions with Schwartz distributions.
We exploited these similarities to ﬁnd bases for equivariant PDOs based on existing solutions for steerable kernels.
Our experiments show that the locality of PDOs can be a disadvantage compared to convolutional kernels.
However, our approach for equivariance can easily be combined with any discretization method, and we show that using Gaussian derivatives for discretization alleviates the issue.
Equivariant PDOs could also be very useful in cases where their strong locality is a desideratum rather than a drawback.
The theory developed in this work provides the necessary foundation for applications where equivariant PDOs, rather than kernels, are needed.
For example, Probabilistic Numerical CNNs (Finzi et al., 2021) use PDOs in order to parameterize convolutions on continuous input data.
Finzi et al.
also derive a constraint to make these PDOs equivariant, which is a special case of our PDO G -steerability constraint Eq.
The solutions to this constraint presented in this paper are the missing piece for implementing and empirically evaluating Probabilistic Numerical CNNs – neither of which Finzi et al.
do.
Another promising application of PDOs is an extension to manifolds.
Gauge CNNs (Cohen et al., 2019b; Kicanaoglu et al., 2019; Haan et al., 2021; Weiler et al., 2021) are a rather general framework for convolutions on manifolds; see Weiler et al.
(2021) for a thorough treatment and literature review.
As in the Euclidean case, Gauge CNNs use feature ﬁelds that transform according to some representation ρ .
The kernels are deﬁned on the tangent space and are still constrained by the
G -steerability constraint.
Because of that, our approach to equivariant Euclidean PDOs is very well-suited for generalization to manifolds and our steerable PDO solutions will still remain valid.
One advantage of PDOs in this setting is that they require no Riemannian structure and can achieve equivariance with respect to arbitrary diffeomorphisms, as is common in physics, instead of mere isometry equivariance.
R EPRODUCIBILITY S TATEMENT
All Propositions and Theorems in the main paper and in the appendix explicitly state all their assumptions.
In cases where theoretical results are stated only informally in the main paper (such as our results involving Schwartz distributions), the precise claims can be found in the appendix.
The appendix also contains complete proofs for all of our theoretical claims.
The code necessary to reproduce our experiments can be found at https://github.com/ ejnnr/steerable_pdo_experiments .
The required datasets are downloaded and preprocessed automatically.
The exact hyperparameters we used are available as pre-deﬁned conﬁguration options in our scripts.
We also include a version lockﬁle for installing precisely the right versions of all required Python packages.
Our implementation of steerable PDOs is easy to adapt to different use cases and fully documented, allowing other practitioners to test the method on different datasets or tasks. A CKNOWLEDGMENTS
We would like to thank Gabriele Cesa and Leon Lang for discussions on integrating steerable PDOs into the E2CNN library and on solutions of the kernel G -steerability constraint.
This work was supported by funding from QUVA Lab
S UPPLEMENTARY MATERIAL A S TEERABLE PDO S BETWEEN VECTOR AND SCALAR FIELDS
We give bases for the space of steerable PDOs for many important cases in Appendix K.
However, the generality of the description there obscures the connection to well-known PDOs such as the gradient or divergence.
So to complement the general solutions, we discuss a few simple cases in much more detail in this section.
We will see that Laplacian, gradient, divergence, and curl are all rotation equivariant and, more interestingly, that all rotation equivariant PDOs can be constructed by combining these (in the simple settings we cover in this section).
In many cases, we rederive the solutions even though all of them would follow immediately from the general case in Appendix K, in order to provide some intuition on why these are the only equivariant PDOs.
Readers who are only interested in an overview of the results may wish to skip to the end of this section.
A. S CALAR TO SCALAR PDO S
We have already argued in Section 2.3 that the SO(2) -equivariant PDOs between two scalar ﬁelds are precisely polynomials in the Laplacian, i.e. of the form q (∆) for an arbitrary real polynomial q ∈ R [ x ] .
The derivation given there applies without changes to SO( d ) for d > 2 and to O( d ) as well, so the same holds in these cases.
A. S CALAR TO VECTOR PDO S
We start by considering the case where ρ in is trivial (with c in = 1 ) and ρ out is the vector ﬁeld representation (i.e. c out = d ).
We can then represent PDOs by a d × 1 matrix of polynomials, i.e. a column vector.
The PDO steerability constraint becomes
[Equation]
since ρ in is trivial.
A good mental model for this subsection is to think of P as a vector ﬁeld
P : R d → R d whose entries happen to be polynomials.
The steerability constraint simply states that this vector ﬁeld must “look the same” after a rotation, see Fig. 3 for examples.
We begin by discussing the case G = SO(2) , which is somewhat different from O(2) and from d > 2 .
Any rotation equivariant vector ﬁeld on R 2 is fully determined by its values on the ray { ( x 1 , 0) | x 1 > 0 } .
Speciﬁcally, the rotation equivariant vector ﬁelds v are precisely those that can in polar coordinates be written as
[Equation]
for arbitrary radial parts f 1 , f 2 .
In Cartesian coordinates, we can write this as
[Equation]
where ˜ f
i ( z ) :
= f i ( √ z ) √ z .
We need v 1 and v 2 to be polynomials in x, y , which means that the ˜ f
i need to be polynomials.
If we then apply the D isomorphism, we get:
Proposition 3.
The SO(2) -equivariant differential operators from a scalar to a vector ﬁeld are exactly those of the form
[Equation]
where q 1 and q 2 are arbitrary polynomials.
In words, the SO(2) -equivariant PDOs are all linear combinations of the gradient and the transpose of the 2D curl, ( − ∂ 2 , ∂ 1 ) T , with coefﬁcients being polynomials in the Laplacian (rather than just real numbers)
[Figure]
The gradient is also equivariant under reﬂections, i.e. O(2) -equivariant, and it easily generalizes to higher dimensions.
However, the transpose 2D curl only appears in this particular setting: it is not reﬂection-equivariant, and it does not have an analogon in higher dimensions.
6 We summarize this in the following result:
Proposition 4.
Let G
= O( d ) for d ≥ 2 or G = SO
( d ) for d > 2 .
Then the G -equivariant differential operators from a scalar to a vector ﬁeld are exactly those of the form
[Equation]
for an arbitrary polynomial q ∈ R [ x ] .
A possible intuition for why SO(2) is a special case is that SO(2) ∼ = S
whereas SO( d ) (cid:54)∼ = S d −for d > 2 and O( d ) (cid:54)∼ =
S
d − 1 for d ≥ 2 .
Our construction above heavily makes use of the fact that rotations of R 2 correspond one-to-one to angles, i.e. points on S 1 , and this construction thus does not generalize to any other cases.
Before we prove Proposition 4, we show a helpful lemma:
Lemma 5.
Let d > 2 .
Then for any linearly independent vectors v, w ∈ R d , there is a rotation
g ∈ SO( d ) such that gv = v but gw (cid:54) = w .
For d = 2 , there is such a g ∈ O(2) .
Proof of lemma.
Since v and w are linearly independent, we can write R d = span( v ) ⊕ W , where
W is a linear subspace containing w .
Pick an element ˜ g ∈ SO
( d − 1) (or O( d − 1) = O(1) = {± 1 } in the d = 2 case) such that ˜ gw (cid:54) =
w , where ˜ g acts on W (this always exists, note that w (cid:54) = 0 ).
Then there is a g ∈ SO
( d ) (or O( d ) ) that restricts to ˜ g on W and is the identity on span( v ) (in terms of matrices with respect to a basis of the form { v, . . .
} , g would be block-diagonal, with a 1 × 1 identity block and a ( d − 1) ×
( d − 1) block for ˜ g ).
Proof of Proposition 4.
One direction is clear: the gradient is induced by the matrix of polynomials
P ( x ) = x, x ∈ R d , which is clearly equivariant.
We have also seen that polynomials of the Laplacian are equivariant, and we can compose these equivariant PDOs to get another equivariant PDO.
So what remains to show is that no other equivariant PDOs exist.
So let P be G -equivariant, with G as in Proposition 4.
Let furthermore x ∈ R d be arbitrary and g ∈ G be an element in the stabilizer of x , i.e. gx = x .
Then we have
[Equation]
In other words, for any g ∈ G with gx = x , we also have gP ( x ) =
P ( x ) .
By the lemma, x and P ( x ) are thus linearly dependent, i.e. P ( x ) =
cx for this particular x and some c ∈ R .
We can apply this reasoning to any x ∈ R d , so there is a function c : R d → R such that P ( x ) =
c ( x ) x .
Furthermore,
[Equation]
so c has to be rotation invariant.
As we have already argued, this implies that c ( x ) =
q ( x 21 + . . .
+ x 2 d ) for some function q , and since c needs to be a polynomial, so does q .
Then
[Equation]
as claimed.
If d = 1 , the lemma also holds and the proof goes through – this is just the scalar case from the previous section, which is generalized here.
But for G = SO(2) , this argument does not work because
SO (1) =
{ 1 } , so the decisive step in the proof of the lemma fails.
That is what allows the additional equivariant PDOs.
A. V ECTOR TO SCALAR PDO S
Equivariant PDOs mapping from vector to scalar ﬁelds are simply the transpose of those mapping from scalar to vector ﬁelds (for orthogonal groups G ): P is now a 1 × d matrix and the steerability constraint is
[Equation]
By transposing, we get
[Equation]
19) which is the equivariance condition for scalar to vector layers.
As solutions, we get the divergence (as the transpose of the gradient) and for G = SO(2) also the 2D curl.
They are again combined linearly with polynomials in the Laplacian as coefﬁcients.
A. V ECTOR TO VECTOR PDO S
For PDOs mapping between two vector ﬁelds, the steerability constraint is
[Equation]
Since the solution space in this case is somewhat more complicated, we will only cover G = SO(2) in these examples; see Appendix K for more solutions.
In principle, we could apply the same method that we already used before for SO(2) , choosing the radial components of P freely and using the steerability constraint to determine the angular components.
But since the computations in this case are more involved and don’t yield much additional insight, we will instead use the solutions from
Appendix K. Vector ﬁelds correspond to frequency 1 irreps, and writing out the solutions for those explicitly, we get that the equivariant PDOs are exactly linear combinations of
[Equation]
with polynomials in the Laplacian as coefﬁcients.
The ﬁrst two operators are simply the identity and a π 2 rotation matrix, both zeroth order PDOs.
Note that the rotation matrix rotates the ﬁbers of the vector ﬁeld, it does not act on the base space.
The other two operators are less interpretable, but we can replace them through a change of basis:
[Equation]
x
[Equation] [Equation]
which is the matrix describing the composition grad ◦ div .
Similarly,
[Equation]
which is the matrix describing grad ◦ curl 2D .
So if we write R for the π 2 rotation matrix (interpreted as a PDO), then the equivariant PDOs mapping between vector ﬁelds are exactly linear combinations of [Equation]
[Equation]
as always with polynomials in the Laplacian as coefﬁcients.
We can be even more economical and describe these PDOs with fewer building blocks.
For two vectors P, Q of one-dimensional PDOs, e.g. P = ∂ = ( ∂ 1 , ∂ 2 ) T , we write P ⊗ Q for the 2 ×
PDO with entries ( P ⊗ Q ) ij = P i Q j .
Then we can write for example grad ◦ div = ∂ ⊗ ∂ .
We furthermore note that
[Equation]
This means we can write the basis from above as
[Equation]
A. S UMMARY OF RESULTS
• SO( d ) - or O( d ) -equivariant PDOs between two scalar ﬁelds are exactly polynomials in the Laplacian.
• G -equivariant PDOs mapping from scalar to vector ﬁelds are exactly those of the form q (∆) grad for G = SO( d ) with d > 2 or G =
O( d ) .
For PDOs from vector to scalar ﬁelds, we similarly get q (∆) div .
• SO(2)
-equivariant
PDOs from vector to scalar ﬁelds are exactly those of the form
[Equation]
For PDOs from scalar to vector ﬁelds, we get grad instead of div and the transpose of the 2D curl instead.
• SO(2)
-equivariant PDOs between two vector ﬁelds are linear combinations of the identity, a π 2 rotation, grad ◦ div and grad ◦ curl 2D , with polynomials in the Laplacian as coefﬁcients.
B R EPRESENTATION THEORY PRIMER
This section introduces the fundamental deﬁnitions of representation theory that we need.
B. B ASIC DEFINITIONS
Deﬁnition 1.
A group representation of a group G , or representation for short, is a group homomorphism ρ : G → GL( V ) for some vector space V .
This means that ρ ( gg (cid:48) )
=
ρ ( g ) ρ ( g (cid:48) ) for all g, g (cid:48) , so multiplication of group elements in G is represented as matrix multiplication in GL( V ) .
For this paper, we only need V = R c , so we focus on this case.
In particular, some of the following deﬁnitions make use of the fact that we only consider ﬁnite-dimensional representations.
Given multiple representations of the same group, we can “stack them together” using direct sums:
Deﬁnition 2.
Let ρ 1 : G → GL( R c 1 ) and ρ 2 : G → GL( R c 2 ) be two representations of G .
Then we deﬁne the direct sum representation ρ 1 ⊕ ρ 2 : G → GL( R c 1 + c 2 ) by ( ρ 1 ⊕ ρ 2 )( g ) :
= (cid:18)
ρ 1 ( g ) ρ 2 ( g ) (cid:19) , (28) which acts independently on the subspaces R c 1 and R c 2 of R
This corresponds exactly to stacking feature ﬁelds of different types.
For example, we can stack a vector and a scalar ﬁeld into one four-dimensional ﬁeld, such that its vector and scalar part transform independently .
The representation of this four-dimensional ﬁeld will be the direct sum of the vector and scalar ﬁeld representations.
Often, two representations are formally different but can be transformed into one another using a change of basis; they then behave the same in all relevant aspects.
For example, SO(2) has a representation
[Equation]
that represents a rotation angle ϕ by a counterclockwise rotation matrix (this representation is the one used for vector ﬁelds).
However, we could just as well use
[Equation]
where the rotation is clockwise.
Using one or the other is pure convention and we would like to treat them as “the same” representation.
This is formalized as follows:
Deﬁnition 3.
Two representations ρ 1 , ρ 2 : G → GL( R c ) are equivalent if there is a matrix Q ∈
GL( R c ) such that
[Equation]
for all g ∈ G .
Intuitively, ρ 1 and ρ 2 differ only by a change of basis, which is given by Q .
B. D ECOMPOSITION INTO IRREDUCIBLE REPRESENTATIONS
We can now discuss irreducible representations, which play an important role for solving the kernel and PDO steerability constraints.
Deﬁnition 4.
A linear subspace W ⊆ R c is called invariant under a representation ρ : G → GL( R c ) if ρ ( g ) w ∈ W for all g ∈ G and w ∈ W .
In this case, we can deﬁne the restriction ρ | W : G →
GL( W ) , called a subrepresentation of ρ .
Deﬁnition 5.
A representation ρ is called irreducible if all its subrepresentations are either ρ itself or representations G → GL( { 0 } ) , where { 0 } is the trivial vector space.
For example, ρ 1 and ρ 2 are both subrepresentations of ρ 1 ⊕ ρ 2 , so direct sums are never irreducible.
A natural question is the converse: if a representation is not equivalent to a direct sum, does that mean that it is irreducible?
In other words, can all representations be split into a direct sum of irreducible ones?
In general, this is false, but it holds for the cases that interest us:
Deﬁnition 6.
A topological group is a group G equipped with a topology, such that the group multiplication G × G → G and the inverse map G → G are continuous with respect to that topology.
A compact group is a topological group that is compact as a topological space.
Theorem 6.
Let G be a compact group.
Then every ﬁnite-dimensional representation of G is equivalent to a direct sum of irreducible representations.
Since we only consider compact subgroups of O(2) and O(3) , this theorem applies to all the cases we solve.
So we can always write
[Equation]
where the ψ i are irreducible representations.
It is then easy to show (Weiler & Cesa, 2019) that a kernel k solves the G -steerability constraint
[Equation]
if and only if κ :=
Q out k Q − 1 in solves a block-wise steerability constraint between irreducible representations.
Concretely,
[Equation]
where κ ij ( x ) is the submatrix of κ ( x ) that belongs to ψ i and ψ j .
The approach to solving the steerability constraint is thus to solve Eq.
(34) for arbitrary irreducible representations ψ i and ψ j .
For general (not necessarily irreducible) ρ out
and ρ in , we then ﬁrst ﬁnd the decompositions into irreducible representations.
Each basis element κ ij of the solution to Eq.
(34) is then padded with zeros to the right size and ﬁnally transformed via k = Q − 1 out κQ in to get the ﬁnal basis elements.
See Weiler & Cesa (2019) for a more detailed discussion and visualization.
Clearly, this procedure works just as well for PDOs as it does for kernels: for PDOs, we need to ﬁnd the restriction of the kernel solution space to polynomials, and it does not matter whether we restrict on the level of irreducible representations and then combine them, or ﬁrst combine irreducible representations and then restrict.
B. S PECIFIC REPRESENTATIONS
We now deﬁne all the types of representations that occur in the main paper:
• As already mentioned in the paper, scalar ﬁelds are described by the trivial representation ρ ( g ) :
= 1 and vector ﬁelds by the representation ρ ( g ) :
= g (for G ≤ GL( R d ) ).
• For a ﬁnite group G , the regular representation is ρ : G → GL( R | G | ) , deﬁned by
[Equation]
where ( e g ) g ∈ G is the canonical basis of R | G | (for some ordering of G ).
So this representation associates one basis vector to each group element and then acts by permuting these basis vectors.
ρ ( g ) is thus always a permutation matrix.
• Quotient representations are a generalization of regular representations, deﬁned as follows: let G be a ﬁnite group and H ≤ G a subgroup.
Then we deﬁne the quotient representation ρ G/H quot : G → GL( R | G | / | H | ) by
[Equation]
where we now use a basis indexed by the cosets gH ∈ G/H for g ∈ G .
For H = { e } , we recover regular representations.
Appendix C by Weiler & Cesa (2019) provides some intuition for these quotient representations in the case where G and H are both cyclic groups C N and C M .
C I NTUITION FOR THE GROUP ACTION ON POLYNOMIALS
Our work makes heavy use of terms of the form p ( gx ) , where p is a polynomial p : R d → R , g ∈ G is a group element, and x ∈ R d .
We would now like to provide a bit more intuition for this action of the group G on polynomials.
Note that we will only cover scalar-valued polynomials in this appendix, i.e. using trivial representations.
The general case is straight-forward: the group acts on each polynomial in a matrix of polynomials the way we describe here, while ρ in ( g ) and ρ out ( g ) act via matrix multiplication.
To prevent confusion, let us reiterate that we can think of polynomials in two different ways.
The ﬁrst is as a formal expression, where x is a placeholder for things to be plugged in.
This is the approach we take when connecting polynomials to PDOs, where we plug in differential operators for x .
The second is as a speciﬁc type of function on R d —in which case x ∈ R
d is simply the argument of that function.
This second perspective is the one in which the group action on polynomials is easiest to understand.
Speciﬁcally, the action of g on x is simply matrix multiplication.
Similar to how x (cid:55)→ p ( x ) deﬁnes a function on R d , x (cid:55)→ p ( gx ) deﬁnes a different function on R d .
We could think of it as composing the function p with the group action of g on R d .
Crucially, this new function is still a polynomial in the components of x , just with different coefﬁcients.
For purposes of illustration, consider a very simple example with d = 2 and G = SO(2) .
We will use the polynomial p ( x ) =
x 22 + x
(this is just meant to be a simple non-trivial polynomial, it does not have special equivariance properties).
We can represent g as a rotation matrix parameterized by an angle θ .
The action on x is then given by
[Equation]
The left-hand side is what we now plug into our polynomial, which yields
[Equation]
We can expand this and collect the coefﬁcients for each power of x 1 and x 2 , which yields
[Equation]
If desired, we can now switch to the ﬁrst perspective, and interpret this polynomial a formal expression deﬁned by its coefﬁcients.
In that perspective, g acts on p by modifying its coefﬁcients, rather than by composition.
Computing the new coefﬁcients is straightforward analytically (given a matrix representation of g ); we just gave a simple example for a polynomial of order two.
The general case proceeds along the same lines, it just requires using the binomial theorem (or the multinomial theorem for d > 2 ).
D B ACKGROUND ON DISTRIBUTIONS
In Appendix E, we will describe our framework for equivariant maps using convolutions with
Schwartz distributions.
To facilitate that, we now give the necessary background on Schwartz distributions.
We restrict ourselves to what is absolutely necessary for our purposes; for a much more thorough introduction and for proofs, see e.g. Treves (1967).
D. B ASIC DEFINITIONS
Deﬁnition 7.
For U ⊂ R d open, D ( U ) :
= C ∞ c ( U ) is called the space of test functions on U ( C ∞ c ( U ) is the space of compactly supported smooth functions
U → R ).
A sequence ( ϕ n ) in D ( U ) is deﬁned to converge to 0 iff
(i) there is a compact subset K ⊂ U such that the support of each ϕ n is contained in K and (ii) ∂
α ϕ n → 0 uniformly for all multi-indices α .
One can deﬁne the so-called canonical LF topology on D ( U ) .
This topology induces the notion of convergence just given.
However, constructing this topology explicitly is unnecessary for our purposes since knowing when sequences converge will be enough.
Deﬁnition 8.
A linear functional T : D ( U ) → R is deﬁned as continuous if for every sequence ϕ n that converges to 0 in D ( U ) ,
Tϕ n → 0 (in R ).
Such a continuous functional is called a distribution on U .
The space of all distributions on U is written as D (cid:48)
( U ) .
This notion of continuity is also induced by the canonical LF topology and then D (cid:48)
( U ) is the topological dual of D ( U ) as the notation suggests.
So intuitively, a distribution is a “reasonable” way of linearly assigning a number in R to each test function in C ∞ c .
A function f : U → R induces a distribution T f ∈ D (cid:48)
( U ) , deﬁned by
[Equation]
Since ϕ has compact support, we don’t need many restrictions on f (for example, any locally integrable function f ∈ L 1 , loc ( U ) works).
We will also use the duality pairing
[Equation]
and under slight abuse of notation also write
[Equation]
Note that this coincides with the inner product on L 2 ( U ) , but it allows functions f that are not in Lwhile in exchange requiring ϕ to have compact support.
D. C ONVOLUTIONS
We deﬁne the translation τ x : R d → R d by τ x ( y ) =
y
+ x and set τ x f := f ◦ τ − x for functions f on R d .
This is exactly the same as the action of ( R d , +) we used in the main paper, just with more explicit notation.
Furthermore, we write ˇ f ( x ) :
= f ( − x ) .
Then we can deﬁne the convolution between a distribution and a test function: for f ∈ D ( U ) and
T ∈ D (cid:48)
( U ) , the convolution T ∗ f ∈ C ∞ ( U ) is deﬁned by (
[Equation]
[Equation] x
[Equation]
Explicitly, ( τ x ˇ f )( y ) =
f ( x − y ) .
This immediately shows that this notion of convolution extends the classical one, i.e.
T g [Equation] f
[Equation]
g
[Equation]
D. D ERIVATIVES
Deﬁnition 9.
For T ∈ D (cid:48) ( U ) , we deﬁne the distributional derivative ∂ α T as the unique distribution on U for which
[Equation]
The distributional derivative of all orders always exists, i.e. “distributions are inﬁnitely differentiable”, but of course only in this distributional sense.
At least for f ∈ C ∞ ( U ) (but also under much weaker assumptions), this deﬁnition also extends the deﬁnition of derivatives of functions, in the sense that
T ∂ α f = ∂
α T f .
D. C OMPOSITION WITH DIFFEOMORPHISMS
Let F : U → U be a diffeomorphism and T ∈ D (cid:48) ( U ) .
Then we deﬁne the composition T ◦ F ∈ D (cid:48)
( U ) as
[Equation]
where det DF − 1 is the inverse Jacobian.
As in the previous constructions, this extends the deﬁnition for classical functions in the sense that
[Equation]
This follows immediately from the transformation theorem for integrals:
[Equation] [Equation]
[Equation] [Equation]
Furthermore, this type of composition is associative:
[Equation]
We are particularly interested in the case where F is a linear transformation,
i
[Equation]
e. F ∈ GL( R d ) .
Then we have
[Equation]
(note that in this case, we can pull out the determinant because it is just a constant).
For translations, we get
[Equation]
D. T HE D IRAC DELTA DISTRIBUTION
A very simple but important distribution is the following:
Deﬁnition 10.
For any x ∈ R d , the Dirac delta distribution δ x ∈ D ( R d ) is deﬁned by
[Equation]
It is clear from the deﬁnition that the distributional derivatives of the delta distribution are given by
[Equation]
D. C ONVERGENCE OF DISTRIBUTIONS
Deﬁnition 11.
We say that a sequence T n of distributions converges to T , written T n → T , if it converges pointwise, i.e.
[Equation]
for n → ∞
[Equation]
61) for all ϕ ∈ D ( U ) .
Abusing notation a bit, we also write f n → T for functions f n if T f n → T .
One important type of function sequences are Dirac sequences , which converge to the Delta distribution:
Lemma 7.
Let f n be a sequence in L 1 ( R d ) such that
(i) f n ≥ 0 , (ii) (cid:107) f n (cid:107) 1 = 1 , and (iii) (cid:82) R
d \ B ε (0) f n dλ d → 0 for all ε > 0 .
Then f n → δ 0 in the sense of distributions.
We also note that convergence plays together nicely with derivatives and with convolutions:
Lemma 8.
If T n → T , then ∂ α T n → ∂ α T for all α .
Proof.
[Equation]
Lemma 9.
If T n → T , then T n ∗ f → T ∗ f pointwise for all f ∈ D ( U ) .
Proof.
[Equation]
D. T EMPERED DISTRIBUTIONS AND THE F OURIER TRANSFORM
The Schwartz space S ( R d ) is the space of functions for which all derivatives decay very quickly as | x | → ∞ .
More precisely, a smooth function f : R d → R is in S ( R d ) iff
[Equation]
Intuitively, all derivatives of f must decay more quickly than any polynomial.
Examples are compactly supported functions or Gaussians.
Its dual space S (cid:48)
( R d ) is called the space of tempered distributions and can be continuously embedded into D (cid:48)
( R d ) .
Tempered distributions are still very general; in particular the delta distribution, distributions induced by functions, and derivatives of tempered distributions are all tempered.
The reason we’re interested in tempered distributions is that there is a Fourier transform deﬁned by
[Equation]
for tempered distributions T .
This is an automorphism on S (cid:48)
( R d ) .
We use the following convention for the Fourier transform on functions:
[Equation]
[Equation]
which means the inverse is given by
[Equation]
We will later need the Fourier transform of derivatives of the Dirac delta distribution:
[Equation]
so F { ∂ α δ 0 } ∝ x α (or more precisely, the distribution induced by the function x (cid:55)→ x α ).
D. M ATRICES OF DISTRIBUTIONS
Analogously to how we deﬁned matrices of PDOs or of polynomials, we will need matrices of distributions.
We will write D (cid:48)
( U, R c out × c in ) for the space of c out × c in dimensional matrices with entries in D (cid:48) ( U ) .
We get a pairing (cid:104)· , ·(cid:105) :
D (cid:48)
( U, R c out × c in ) × D ( U [Equation] R c in ) → R c out (69) deﬁned by
[Equation]
The convolution of T ∈ D (cid:48)
( U, R c out × c in ) and f ∈ D ( U, R c in ) is deﬁned analogously to the scalar case, i.e. ( T ∗ f )( x ) :
= (cid:104) T, τ x ˇ f (cid:105) , where we now use the more general duality pairing just deﬁned.
We can also deﬁne the multiplication of matrices of distributions by matrices over R .
For a matrix
A ∈ R c out × c out and a matrix of distributions T ∈ D (cid:48)
( R
[Equation]
R c out ×
[Equation]
) , we write
[Equation]
(multiplying a distribution by a scalar obviously deﬁnes another distribution).
Analogously, we deﬁne
TB for B ∈ R c in × c in .
It immediately follows that
[Equation]
holds.
Composition with diffeomorphisms of R d can simply be deﬁned component-wise for matrices of distributions.
This commutes with multiplication by constant matrices, meaning that
[Equation]
Finally, we also deﬁne the Fourier transform component-wise, and this commutes with multiplication by matrices in the same way.
E D ISTRIBUTIONAL FRAMEWORK FOR EQUIVARIANT MAPS
In this section, we develop two main results: ﬁrst that all linear continuous translation equivariant maps between feature spaces are convolutions with some distribution, and then the equivariance constraint for such convolutions.
See Appendix D for background on distributions.
We equip D ( U ) with the canonical LF topology throughout this section.
E. T RANSLATION EQUIVARIANT MAPS ARE CONVOLUTIONS WITH DISTRIBUTIONS
We begin by showing that the framework using convolutions with Schwartz distributions encompasses all translation equivariant continuous linear maps between feature spaces.
As preparation, we prove a simple Lemma on the reﬂection map s :
Lemma 10.
The map s : D ( U ) → D ( U ) given by s ( f ) :=
ˇ f is linear and continuous.
Proof.
Linearity is clear:
[Equation]
For continuity, we use the fact that a linear map from D ( U ) to itself is continuous if and only if it is sequentially continuous (Treves, 1967, Proposition 14.7).
So take any sequence f n → 0 in D ( U ) .
This means that
(i) there is a compact subset K ⊂ U such that the support of each f n is contained in K (ii) ∂ α f n → 0 uniformly for all multi-indices α
We set − K :
= {− x | x ∈ K } , then the support of s ( f n ) (which is just the mirror of the support of f n ) is contained in − K , and − K is compact.
Additionally,
[Equation]
Since ∂ α f n → 0 uniformly, the same is true for s ( ∂ α f n ) .
It follows that s is sequentially continuous, and thus continuous, which concludes the proof.
We ﬁrst prove our main generality result for the special case of one-dimensional ﬁbers:
Lemma 11.
Let Φ : D ( U ) → D ( U ) be a translation equivariant continuous linear map.
Then there is a distribution T ∈ D (cid:48)
( U ) such that Φ( f ) =
T ∗ f .
Proof.
Let Φ : D ( U ) → D ( U ) be any continuous linear map, where continuity is understood with respect to the canonical LF topology.
We then deﬁne
[Equation]
Equivalently, we can write T =
δ 0 ◦ Φ ◦ s , understood as normal composition of functions.
δ 0 and Φ are linear and continuous, as is s by Lemma 10.
Therefore, T is also linear and continuous, and thus a Schwartz distribution T ∈ D (cid:48)
( U ) .
We will also need that
[Equation]
Now using the assumption that Φ is translation equivariant, i.e.
[Equation]
78) it follows that convolution with T is given by
[Equation]
This shows that convolution with T is Φ , which concludes the proof.
Finally, we generalize to multi-dimensional feature ﬁelds:
Theorem 12.
Let Φ : D ( U, R c in ) → D ( U, R c out ) be a translation equivariant continuous linear map.
Then there is a distribution T ∈ D (cid:48)
( U, R c out × c in ) such that Φ( f )
=
T ∗ f .
Proof.
We write e j for the j -th canonical basis vector of R c in .
For f ∈ D ( U, R c in ) , we write f j for its j -th component.
Then we have
[Equation]
We now deﬁne the components Φ ij : D ( U ) → D ( U ) by
[Equation]
Here, ge j ∈ D ( U, R c in ) is the map with g in its j -th output component and 0 in the other ones.
We can then write
[Equation]
It is clear that the components Φ ij of Φ are still translation equivariant, continuous and linear.
Therefore, by the previous Lemma, there are distributions T ij ∈ D (cid:48)
( U ) such that Φ ij ( g ) =
T ij ∗ g .
We then get
[Equation]
where T ∈ D (cid:48)
( U, R c out × c in ) is the matrix of distributions with entries T ij .
E. E QUIVARIANCE CONSTRAINT FOR DISTRIBUTIONS
We will now characterize the distributions T ∈ D (cid:48)
( R d , R c out × c in ) for which the operator
[Equation]
given by f (cid:55)→ T ∗ f is H -equivariant.
The codomain of T ∗ may contain functions that are not compactly supported for some T .
We are mostly interested in distributions T for which the codomain can be restricted to D ( R d , R c out ) but for the discussion of equivariance this does not make any difference, so we keep the derivation general by not restricting the distribution T .
First, we can note that convolution with a distribution is always translation equivariant:
Proposition 13.
The map f (cid:55)→ T ∗ f is translation equivariant for any distribution T .
Proof.
The general equivariance condition is
[Equation]
) for all f ∈ D ( R d , R c in ) , h ∈ H, x ∈ R d .
More explicitly, this means
[Equation]
where g is the linear component of h .
Let h = t ∈ R d , i.e. a pure translation.
The condition then becomes
[Equation]
which always holds (since τ ( x − t )
=
τ x τ − t ).
So T ∗ is translation equivariant by construction.
Therefore, it sufﬁces to consider pure linear transformations g ∈ GL( R d ) .
For those, we have the following result, which generalizes the steerability constraint for PDOs and the one for kernels:
Theorem 14.
Let T ∈ D (cid:48)
( R d , R c out × c in ) .
Then the map f
(cid:55)→ T ∗ f is G -equivariant
if and only if
[Equation]
As is shown in Appendix D, the deﬁnitions of convolution with distributions and of composition with a diffeomorphism are compatible with those for classical functions.
So if T is a classical kernel, then the constraint on T is given by the same equation, which already gives us the steerability constraint for kernels.
Proof.
We start with the constraint Eq.
(86) from above with h = g ∈ G and will transform this into the desired form Eq.
First, to simplify notation, we can remove all the reﬂections ˇ · in Eq.
This is now possible because ­ f ◦ g − 1 =
ˇ f ◦ g − 1 , which was not true for translations.
Furthermore, we can pull the translations to the other side of the duality pairing:
[Equation]
Now we’ll use
[Equation]
90) since
[Equation]
Plugging this into Eq.
(89), we get
[Equation]
We want to have only f on the right side of the duality pairing, so we use the notation we introduced for multiplying distributions by matrices and get
[Equation]
This holds for all f iff we have equality of distributions,
[Equation]
Finally, multiplication with matrices and composition with diffeomorphisms commute, so we can cancel the τ g − 1 x .
This means our ﬁnal constraint on T is
[Equation]
95) which we can slightly rewrite as
[Equation]
This is exactly Eq.
All steps of the derivation work equally well in the other direction, which proves the “if and only if”.
E. D IFFERENTIAL OPERATORS AS CONVOLUTIONS
It is clear that convolutions with classical kernels are a special case of convolutions with distributions (see Appendix D).
But we have claimed that convolutions with distributions also cover PDOs, which is what we show now.
We have already seen in Appendix D that derivatives of the delta distribution are closely related to
PDOs.
So we calculate the convolution with such derivatives:
[Equation]
Noting that the map T (cid:55)→ T ∗ is R -linear, we see that D ( p ) f
= p ( ∂ ) δ 0 ∗ f for polynomials p .
It then also immediately follows that this is true for matrices of polynomials P because
[Equation]
This shows that PDOs can be interpreted as convolutions with distributions, as claimed.
E. T HE F OURIER DUALITY BETWEEN KERNELS AND PDO S
Interpreting PDOs as convolutions with distributions allows us to relate them to classical convolutional kernels via a Fourier transform.
We have already seen in Appendix D that
[Equation]
which by linearity of the Fourier transform immediately implies
[Equation]
for P ij = (cid:80) α c ijα x α .
So the Fourier transform of derivatives of the delta distribution are polynomials, and of course vice versa via the inverse Fourier transform.
Since PDOs are convolutions with such delta distribution derivatives, we can also interpret them as convolutions with the (inverse) Fourier transform of polynomials .
We will use this interpretation to give a derivation of the PDO steerability constraint that sheds some light on the similarities and differences to the kernel steerability constraint.
We begin by proving a basic fact about the Fourier transform of a composition of functions:
Lemma 15.
Let g ∈ GL
( R d ) and ϕ ∈ D ( U ) .
Then
[Equation]
where we use the shorthand g − T :
= (cid:0) g − 1 (cid:1) T .
Proof.
First we apply the transformation theorem for integrals:
[Equation]
Now we just rewrite g − 1 x · ξ = x · g − T ξ , which gives the desired result.
This result holds more generally for tempered distributions (a subset of distributions for which the
Fourier transform can be deﬁned, see Appendix D):
Proposition 16.
For a tempered distribution T and g ∈ GL( R d ) ,
[Equation]
Proof.
[Equation]
where we used Lemma 15 for (1).
Now note that the equivariance constraint for distributions, Eq.
(88), is equivalent to the constraint we get when we take the Fourier transform on both sides.
That’s because the Fourier transform is an automorphism on the space of tempered distributions.
By applying the result we just proved, we then get the constraint
[Equation]
We can cancel the determinants, which gives the equivariance constraint in Fourier space:
[Equation]
Now let T =
P ( ∂ ) δ 0 , so that T ∗ f = D ( P ) f for some matrix of polynomials P .
We’ve seen above that in this case F { T } =
P ◦ m − i (up to a constant coefﬁcient), where m − i is multiplication by the negative imaginary unit − i .
So D ( P ) is equivariant iff
[Equation]
but since m − i is invertible and commutes with the other maps, we can cancel it and get the equivariance condition
[Equation]
[Equation]
which is precisely the PDO steerability constraint.
The reason that it differs slightly from the kernel steerability constraint can now be traced back to Lemma 15.
Intuitively speaking, since PDOs are in a sense the Fourier transform of convolutional kernels, they transform differently under GL( R d ) , which leads to superﬁcial differences in the steerability constraints.
However, Fourier transforms commute with rotations
E. PDO S AS THE INFINITESIMAL LIMIT OF KERNELS
Let ψ n be any sequence of functions such that ψ n → δ 0 , for example a Dirac sequence.
Then for a polynomial p = (cid:80)
α c α x α , we also have
[Equation]
because of Lemma 8.
Then Lemma 9 implies that
[Equation]
where the convergence is understood pointwise.
Therefore, any sequence of kernels that approximates the delta distribution (by becoming “increasingly narrow”) can be used to approximate arbitrary
PDOs by convolving with the derivatives of the kernels.
One example is a sequence of Gaussians
[Equation] [Equation]
(now indexed by ε > 0 instead of natural numbers).
For ε → 0 , we have ψ ε → δ 0 , as is easy to check with Lemma 7.
This naturally leads to the “derivative of Gaussian” discretization used in our experiments: we can approximate the PDO as convolution with a derivative of a Gaussian kernel and then simply discretize this Gaussian derivative by sampling it on the grid points.
The discussion in this subsection focused on 1 × 1 PDOs and kernels, i.e. c in = c out = 1 , but since convergence for multi-dimensional PDOs and kernels works component-wise, everything generalizes immediately to that setting.
F T HE RELATION BETWEEN KERNELS AND PDO S
Convolutional kernels and PDOs are closely related but also differ in some important ways.
This appendix is meant to brieﬂy summarize various aspects of their relation that would otherwise only be scattered throughout this paper.
First and foremost, we would like to emphasize that, in a continuous setting, PDOs are not just a special case of convolutions with classical kernels.
For example, the gradient operator is not represented by convolution with any kernel.
In fact, even the identity operator (a zeroth-order PDO) is not the convolution with any function.
However, there are two caveats to the above statement.
First, if we allow convolutions with Schwartz distributions , rather than the usual kernels, then this generalizes both PDOs and convolutions with functions.
In this broader framework, we can thus interpret PDOs as convolutions.
But to the best of our knowledge, convolutions with Schwartz distributions have not been previously discussed in a deep learning context, so for the common usage of “convolution”, PDOs are distinct operators.
The second caveat is that when we discretize (translation-equivariant) PDOs on a regular grid, they become convolutions in this discrete setting.
Even so, the differences in the continuum can matter in practice.
First, we might want to discretize on point clouds or meshes.
Second, even on a regular grid, there are different methods for discretizing PDOs.
While all of them lead to convolutions, they lead to convolutions with slightly different kernels.
This shows that there is no clear one-to-one correspondence between kernels and PDOs even on discrete regular grids.
Finally, the space of discretizations of equivariant PDOs is not necessarily the same as the space of discretizations of equivariant kernels.
Finally, we would like to mention a connection that is outside the scope of this paper: inﬁnite series of differential operators and convolutions can be the same even in the continuous setting.
As an example, consider the diffusion equation
[Equation]
Its solution can be written as
[Equation]
The time evolution operator exp( t ∆) is an inﬁnite series of differential operators, but can also be written as a convolution with a heat kernel.
However, these inﬁnite series are not covered by our work—we restrict ourselves to PDOs of ﬁnite order.
We mention this connection between inﬁnite
PDOs and convolutions only to avoid confusion for readers familiar with this correspondence.
G P ROOF OF THE PDO EQUIVARIANCE CONSTRAINT
In this section, we prove Proposition 1 and Theorem 2, the characterizations of translation- and
G -equivariance for PDOs.
Appendix E already contains a proof of Theorem 2 that is arguably more insightful, but in this section we present an elementary proof that does not rely on Schwartz distributions.
G. T RANSLATION EQUIVARIANCE
We begin by proving that translation equivariance of a PDO is equivalent to spatially constant coefﬁcients, Proposition 1.
Let Φ ij = (cid:80) α c ijα ∂ α , where c ijα ∈ C ∞ ( R d ) and i, j index the c out × c in matrix describing the PDO.
The equivariance condition Eq.
(2) for the special case of translations is
[Equation]
Using the deﬁnition of the (cid:66) action in Eq.
(1), this becomes
[Equation]
Here, c α are matrix-valued, f is vector valued, and we have a matrix-vector product between the two.
Explicitly, this means
[Equation]
for all indices i .
If f is zero in all but one component, the sum over j reduces to one summand.
We therefore get a simpler scalar constraint
[Equation]
that has to hold for all indices i, j , where g ∈ C ∞ ( R d ) is now scalar-valued.
Since this must hold for all functions g , we get an equality of differential operators,
[Equation]
This implies c ijα ( x − t ) =
c ijα ( x ) for all x , t and α , which in turn means that c ijα must be constants.
From Eq.
(115), it is also apparent that constant coefﬁcients are sufﬁcient for translation equivariance, which proves the converse direction.
G. G - EQUIVARIANCE
Secondly we prove Theorem 2, the G -steerability constraint for PDOs.
Recall that the G -equivariance condition for D ( P ) is [Equation]
[Equation]
where f ∈ F in and g ∈ G .
If we write out the deﬁnition of the induced action (cid:66) , this becomes
[Equation]
We write P =
(cid:80)
α c α x α , where c α are constant matrix-valued coefﬁcients c α ∈ R c out × c in .
The LHS of Eq.
(120) is then
[Equation]
[Equation]
In Lemma 17, we will show that the last term in Eq.
(121) is given by
[Equation]
where g − T := (cid:0) g − 1 (cid:1) T and ∂ is understood as a vector that g − T acts on.
Plugging this into Eq.
(121), we get
[Equation]
for the LHS of Eq.
For the matrix of polynomials P = (cid:80) α c α x α , we now deﬁne
[Equation]
which is again a matrix of polynomials, each one rotated by g .
Then Eq.
(123) can be written more compactly as
[Equation]
Plugging this back into Eq.
(120), canceling the g − 1 and using the fact that this has to hold for all f , we get
[Equation]
as an equality of differential operators.
We move the ρ in ( g ) to the other side and use the fact that the map D is bijective, which yields our ﬁnal constraint
[Equation]
This concludes the proof of the G -steerability constraint for PDOs.
Finally, we prove the Lemma we just made use of, a higher-dimensional chain rule for the special case we need:
Lemma 17.
Let f ∈ C ∞ ( R d , R c ) and g ∈ GL( R d ) .
Then for any multi-index α ,
[Equation]
128) where g − T := (cid:0) g − 1 (cid:1) T .
Proof.
In general, for a linear map A ∈ GL( R d ) , we have
[Equation]
Therefore,
[Equation]
We can apply this iteratively to show that
[Equation]
H T RANSFERRING STEERABLE KERNEL BASES TO STEERABLE PDO BASES
In this section, we develop the method presented in Section 3.2 in more detail and with proofs.
We ﬁx a group G ≤ O( d ) and representations ρ in and ρ out .
Then we write K for the space of
G -steerable kernels.
Because the steerability constraints for kernels and PDOs are identical in this setting, the space of equivariant PDOs is the image under the isomorphism D of the intersection
[Equation]
In words, the space of equivariant PDOs is isomorphic to the space of polynomial steerable kernels.
Both spaces are inﬁnite-dimensional real vector spaces.
The question we tackle now is how we can ﬁnd a basis of K pol given a basis of K under certain conditions.
It will vastly simplify our discussion to treat K and K pol as modules over invariant kernels instead of as real vector spaces, at least for now.
A module is a generalization of a vector space, where the scalars for scalar multiplication can form a ring instead of a ﬁeld.
Because the radial parts of steerable kernels are unrestricted, it makes sense to think of K as a module over the ring of radial functions.
Formally:
Lemma 18.
K is a C ∞ ( R ≥ 0 )
-module, with scalar multiplication deﬁned by
[Equation]
for κ ∈ K and f ∈ C ∞ ( R ≥ 0 ) .
Similarly, K pol is an R [ r 2 ] -module, where r 2 := x 21 + . . .
+ x 2 d and multiplication is simply multiplication of polynomials.
Proof.
The steerability constraint
[Equation]
is clearly R -linear and in particular, K is closed under addition and 0 ∈ K .
Furthermore, for κ ∈ K ,
[Equation]
so K is also closed under the given scalar multiplication.
The proof for K pol is exactly analogous.
In Lemma 18 and in the following, we write r 2 instead of | x | 2 simply to emphasize its role as a polynomial; so when reading r 2 , think of it as a polynomial, and when reading | x | 2 simply as a function of x .
A basis of a module is deﬁned analogously to a basis of a vector space, as a set of linearly independent vectors that span the entire module.
However, linear combinations now allow coefﬁcients in the ring of radial functions, instead of only real numbers.
This means that fewer vectors are needed to span the entire space, because the coefﬁcients “do more work”.
In contrast to vector spaces, not every module has a basis.
K and K pol do have a basis in the cases we consider in the paper but for this section that doesn’t matter to us: we will simply assume that K has a basis with certain properties and then transfer this basis to K pol .
That the method developed here is indeed applicable to subgroups of O(2) and O(3) will be the topic of Appendix J.
We roughly proceed in two steps:
We show that a basis of K (over C ∞ ( R ≥ 0 ) ) that consists only of polynomials (and fulﬁlls a few other technical conditions) is also a basis of K pol , but this time of course over R [ r 2 ] .
We then show how to turn this module basis into a vector space basis of K pol .
The ﬁrst step is formalized as follows:
Proposition 19.
Let B ⊂ R [ x 1 , . . .
, x d ] c out × c in be a basis of K such that no matrix of polynomials in
B is divisible by r 2 and each one is homogeneous.
Then B is also a basis of K pol as an R [ r 2 ] -module.
Here, we say that b ∈ R [ x 1 , . . .
, x d ] c out × c in is divisible by r 2 if every component is divisible by r 2 (as a polynomial).
We call it homogeneous if all its entries are homogeneous polynomials of the same degree.
Proof.
Linear independence is obvious: R [ r 2 ] ⊂ C ∞ ( R ≥ 0 ) , so if B is linearly independent over
C ∞ ( R ≥ 0 ) , then it is also linearly independent over R [ r 2 ] .
To show that B generates K pol , ﬁrst let p ∈ K pol be homogeneous of degree l .
Because p ∈ K , there are f i ∈ C ∞ ( R ≥ 0 ) and κ i ∈ B such that
[Equation]
We want to show that f i ∈ R [ z ] , i.e. that f i is a polynomial.
Since each κ i is homogeneous of degree
l i , we get (cid:88)
[Equation]
because of p ( λx ) =
λ l p ( x ) for any λ ≥ 0 .
Because the κ i are linearly independent, we must have
[Equation]
Thus, f i is homogeneous of degree l − l i 2 and as we show in Lemma 20 below, this implies
[Equation]
or alternatively
[Equation]
What remains to show is that l − l i 2 is a natural number, i.e. that l − l i is even and non-negative.
To prove this, we divide all the i into two groups: those for which l − l i is even and those for which it is odd.
Then we get an expression of the form
[Equation]
Each summand is a rational function (the quotient of two polynomials), so both sums are rational functions.
p ( x ) is also rational, so the ﬁrst term on the RHS has to be rational.
This is only possible if a k = 0 for all k , otherwise we could divide by the (rational) sum and should get a rational function, but | x | is not rational.
This shows that l − l i is even for all i .
Now let l max := max i l i .
Then
[Equation]
It is not possible to cancel any terms: because κ i is not divisible by r 2 and for one i , l max − l i = 0 , the enumerator is not divisible by r 2 (as a polynomial).
Since the denominator is a power of r 2 , the fraction can’t be simpliﬁed.
But we know that p ( x ) is a polynomial.
Therefore, l max ≤ l .
In summary, we’ve shown that
[Equation]
where l − l i is even and non-negative.
Therefore, all f i are polynomials.
Now recall that we assumed p to be homogeneous.
But this is no signiﬁcant restriction: we can write any polynomial as a sum of homogeneous polynomials, each of which can be written as a linear combination of the κ i , as we just showed.
Adding those up leads to a linear combination of the κ i for arbitrary polynomials.
This complete the proof.
Lemma 20.
Let f : R ≥
→ R be homogeneous of degree l ∈ R , meaning that f ( λx ) =
λ l x for all λ ≥ 0 .
Then f ( z ) =
cz l for some c ∈ R (and if l ∈ N , then f is a polynomial).
Note that this Lemma does not hold in higher dimensions – in general there are many more homogeneous functions than polynomials!
Proof.
For any z ≥ 0 ,
[Equation]
[Equation]
(146) which proves the claim by setting c := f (1) .
In Appendix J, we will show that the construction of angular basis elements described in Section 3.
leads to a basis B of K with the properties required by Proposition 19.
It then follows that this also deﬁnes a basis of K pol as a module.
We now come to the second step, turning this module basis into a vector space basis.
Proposition 21.
Let B be a basis of K pol as an R [ r 2 ] -module.
Then
[Equation]
is a basis of K pol as a real vector space.
Proof.
Let v ∈ K pol .
By assumption, there are then basis vectors b 1 , . . .
, b n ∈ B and coefﬁcients p 1 , . .
. , p n ∈ R [ r 2 ] such that
[Equation]
For each i = 1 , . . . , n there are also real coefﬁcients a ( i ) 0 , . .
, a ( i ) m i such that
[Equation]
Combining these equations, we get
[Equation]
This is a real linear combination of elements of the form r 2 k b for b ∈ B .
So since v was arbitrary, the set of such elements does indeed span K pol .
To prove linear independence, let
[Equation]
for some choice of real coefﬁcients a i .
Now we only need to note that a i r 2 k i ∈
R
[ r 2 ] .
So we can interpret this expression as a linear combination over R [ r 2 ] .
Because B is a basis, it follows that a i r 2 k i = 0 for all i , and thus a i = 0 .
That proves linear independence.
As a ﬁnal note, we show that the condition in Proposition 19 that the basis elements are not divisible by r 2 is purely technical; any basis can easily be transformed into one that fulﬁlls it in a canonical way, as formalized by the following lemma.
We do not formally need this result anywhere but it may prove useful if this approach is extended to other groups G because it clariﬁes which parts of the conditions in Proposition 19 are actually important.
Lemma 22.
Let B ⊂ R [ x 1 , . . .
, x d ] c out × c in be a basis of K .
For b ∈ B , we write b = r 2 k b (cid:48) , where k is chosen maximally such that there is an (automatically unique) polynomial matrix b (cid:48) ∈ R [ x 1 , . . .
, x d ] c out × c in .
Then B (cid:48) :
= { b (cid:48) | b ∈ B } is a basis of K and no b (cid:48) is divisible by r 2 .
Furthermore, if b is homogeneous, then so is b (cid:48) .
Proof.
First, note that b (cid:48) is in fact well-deﬁned:
For k = 0 , b (cid:48) =
b works, while for k > deg( b ), no ﬁtting b (cid:48) exists 7 .
So there is some maximal k ≥ 0 with the desired property.
b (cid:48) is clearly unique, namely b (cid:48)
=
| x | − 2 k b as functions on R d .
Furthermore, b (cid:48) is not divisible by r 2 because k was chosen maximally.
B (cid:48) is also clearly a generating set: κ = (cid:80)
i f i b i =
(cid:80) i
( f i r 2 k i ) b (cid:48) i .
To prove linear independence, assume that
[Equation]
(cid:48) exists).
We can write k := max i k i and then get {+
+}
= r 2 k (cid:88) i f i b (cid:48) i = (cid:88) i f i r 2( k − k i ) b i .
This means that f i r 2(
[Equation]
− k i ) = 0 ∀ i, (154) which implies f i = {+
+} for all i , since r 2( k − k i ) is non-zero everywhere except in the origin and f i is continuous.
Therefore, B (cid:48) is linearly independent and hence a basis of K .
Now we can formulate Proposition 19 more generally: for any basis B of K consisting only of homogeneous matrices of polynomials, the corresponding B (cid:48) is a basis of K pol .
I S OLUTIONS FOR IMPORTANT GROUPS
This and the next two appendices describe the solutions of the PDO equivariance constraint for subgroups of O(2) and O(3) .
In this appendix, we describe the general form of these solutions; we recommend readers who are not interested in all the details focus on this one.
Appendix J contains proofs for some claims we make in this appendix where these proofs do not provide as much insight.
Finally, Appendix K contains tables with concrete solutions, it is mainly relevant to the implementation of steerable PDOs.
I. S OLUTIONS FOR SUBGROUPS OF O(2)
To solve the steerability constraint for all (compact) subgroups G ≤ O(2) and for arbitrary representations ρ in and ρ out , Weiler & Cesa (2019) derive explicit bases only for irreducible representations, since the general case can then easily be computed (see Appendix B for details).
This works exactly the same for PDOs as well.
The angular basis elements χ β that they describe for irreps are all matrices with entries of the form cos( kϕ ) and sin( kϕ ) , where k differs between basis elements but is the same for all entries of one matrix.
For example, for G = SO(2) , ρ in the frequency n irrep, i.e. ρ in ( g ) =
g n , and ρ out trivial, the angular basis elements are
[Equation]
We will show how to ﬁnd the steerable PDO basis using this example, but the method works exactly the same in all cases.
Tables with all explicit solutions can be found in Appendix K.
As described in Section 3.2, we now need to multiply these matrices with the smallest power of | x | such that all entries become polynomials.
We show in Appendix J that the necessary coefﬁcient for cos( nϕ ) and sin( nϕ ) is | x | n , so in the example above, we get
[Equation]
and
[Equation]
The entries are written in polar coordinates here, but they are in fact polynomials in the Cartesian coordinates x 1 and x 2 .
More precisely, we show in Appendix J that they are closely related to
Chebyshev polynomials, based on which we derive the following explicit expressions:
[Equation]
This Cartesian form then allows us to interpret the polynomial as a differential operator, by applying the ring isomorphism D , i.e. plugging in ∂ 1 and ∂ 2 for x 1 and x 2 .
If we set n = 1 in our example above, which corresponds to a vector ﬁeld, we get simply | x | cos( ϕ ) =
x 1 and | x | sin( ϕ ) =
x 2 , so we recover the angular PDO basis
[Equation]
div and
[Equation]
that we already derived in Section 2.3.
To get a complete basis, we combine these PDOs with powers of the Laplacian.
I. S OLUTIONS FOR O(3) AND SO(3)
We now turn to the other cases for which steerable kernel solutions have been published, namely O(3) and SO(3) .
Like for O(2) , we only need to consider pairs of irreducible representations.
As described in (Weiler et al., 2018a; Lang & Weiler, 2021), we can build the corresponding angular parts χ β out of real spherical harmonics
Y lm using Clebsch-Gordan coefﬁcients.
We show in Appendix J that we can apply this procedure to | x | l Y lm instead of Y lm to obtain the corresponding ˜ χ β .
We then build the basis by combining with powers of | x | 2 , as we described in Section 3.2 and already did for O(2) .
To ﬁnd the corresponding differential operators D (˜ χ β ) , we only need a Cartesian representation of the polynomials | x | l Y lm , which is fortunately well-known (Varshalovich et al., 1988).
This then again leads to a complete basis of the space of steerable PDOs, with the same general form containing powers of the Laplacian.
J P ROOFS FOR SOLUTIONS FOR SUBGROUPS OF O(2) AND O(3)
In this section, we apply the method from Appendix H to subgroups of O(2) and O(3) , by describing bases for the space of steerable kernels that satisfy the conditions of Proposition
. J. O(2)
As described in Appendix I.1, it is sufﬁcient to consider irreducible representations ρ in and ρ out and for these representations, the angular part of the kernel has a basis consisting of matrices with entries of the form cos( nϕ ) and sin( nϕ ) (where n ∈ Z ).
Crucially, n is the same for all entries of one matrix (though it may differ between basis elements).
We claim that multiplying each angular basis element with r | n | gives a basis of the space of polynomial steerable kernels K pol (as a R [ r 2 ] -module), as described in Appendix H. This follows from
Proposition 19 if we can prove that r | n | cos( nϕ ) and r | n | sin( nϕ ) are
(i) polynomials (in x, y ), (ii) homogeneous (of degree n ), (iii) and not divisible by r 2 .
To show that r | n | cos( nϕ ) and r | n | sin( nϕ ) are polynomials, we use Chebyshev polynomials , which can among other things be seen as a generalization of the addition theorems for sin(2 ϕ ) and cos(2 ϕ ) .
Speciﬁcally, they are families of polynomials T n ( ﬁrst kind ) and U n ( second kind ) deﬁned for n ≥with the property that
[Equation]
We extend the deﬁnition to negative n by setting
[Equation]
Then Eq.
(159) holds for all n ∈ Z , as follows immediately from the parity of sin and cos .
Motivated by the close relation to Chebyshev polynomials, we then deﬁne the following notation for the matrix entries we are considering:
[Equation]
[Equation] We now prove claim (i), that ˜ T n and ˜ U n are polynomials in the Cartesian coordinates x, y , by using a few well-known facts about Chebyshev polynomials: First, T n has degree | n | , so the highest order term in ˜ T n is r | n | (cos ϕ ) | n | = x | n | .
Similarly, U n − 1 has degree | n | − 1 and the highest order term in ˜ U n is thus r | n | (cos ϕ ) | n |− 1 sin ϕ
= x | n |− 1 y .
Lower order terms have additional powers of r that aren’t “matched” by a cosine or sine.
But the second fact about Chebyshev polynomials is that they are either even or odd, so all of these powers of r are even (i.e. powers of r 2 ) and thus themselves polynomials.
That ˜ T n and ˜ U n are homogeneous of degree n – claim (ii) – is immediately clear from their deﬁnition.
It remains to show claim (iii), that they are not divisible by r 2 .
For that, we use the following Lemma:
Lemma 23.
Let p be a non-zero, harmonic, homogeneous polynomial.
Then p is not divisible by r 2 .
Here, p is harmonic if its Laplacian vanishes.
Proof.
For any homogeneous polynomial p , there is a unique decomposition
[Equation]
such that h is harmonic and homogeneous.
Since q = 0 , h = p is one such decomposition for p harmonic, there is no solution with h =
(unless p = 0 ).
So non-zero homogeneous harmonic polynomials are not divisible by r 2 .
Now we only need to show that ˜ T n and ˜ U n are in fact harmonic.
This can be done with a brief calculation in polar coordinates:
[Equation]
The same holds for ˜ U n :
[Equation]
In summary, we have shown that ˜ T n and ˜ U n are homogeneous polynomials not divisible by r 2 , which makes Proposition 19 applicable.
In order to make the basis practically applicable, we also give explicit Cartesian expressions for ˜ T n and ˜ U n .
We restrict ourselves to non-negative n to keep the notation less cluttered.
Equation (161) immediately gives the case for n < 0 .
An explicit formula for the Chebyshev polynomials of the ﬁrst kind is
[Equation]
It follows that
[Equation]
Similarly, the Chebyshev polynomials of the second kind are given by
[Equation]
which means that
[Equation]
J. O(3)
As already mentioned in Appendix I.2, for O(3) and SO(3) , the irreps angular basis between irreducible representation can be built by combining real spherical harmonics
Y lm using ClebschGordan coefﬁcients.
We refer to Weiler et al.
(2018a)
; Lang & Weiler (2021) for details on how this works since it is exactly the same procedure whether one uses kernels or PDOs.
The only fact we need to know here is that each basis element is built from spherical harmonics with the same l .
The necessary steps are now very similar to those for O(2) and its subgroups.
We will use r l Y lm where we had ˜ T n and ˜ U n for O(2) .
Here it becomes important that inside one basis element, only one l appears, because this means we can multiply the entire matrix of polynomials by r l .
We would then need to show that these are homogeneous polynomials not divisible by r 2 but for spherical harmonics, it is already very well known that r l Y lm are homogeneous harmonic polynomials of degree l ; Lemma 23 then implies that they are not divisible by r 2 .
For explicit Cartesian expressions, we refer to e.g. Varshalovich et al.
K S OLUTION TABLES
Using the results from Appendices H and J, we now very easily get complete bases for all subgroups of O(2) and for all irreducible representations by transferring the solutions by Weiler & Cesa (2019).
The appendix of (Weiler & Cesa, 2019) contains tables with all kernel solutions; we simply replace every term of the form cos( kϕ ) with ˜ T k and sin( kϕ ) with ˜
U k to get the following tables.
For the sake of readability, we only write the polynomials ˜ T k and ˜
U
k inside the tables, though the
PDOs themselves should of course be D ( ˜ T k ) and D ( ˜ U k ) .
Explicit formulas for ˜ T k and ˜
U k were given in Eq.
All bases described here are module bases for the space of steerable PDOs as an R [∆] -module.
See
Section 3.2 and Appendix H for details.
K. S PECIAL ORTHOGONAL GROUP SO(2)
The irreducible representations of SO(2) are the trivial representation ψ 0 and those of the form
[Equation]
The bases for all the combinations of these irreps are:
Out
In
[Equation]
K. O RTHOGONAL GROUP O(2)
Elements of O(2) can be written as a tuple ( r, s ) consisting of a rotation r ∈ SO(2) and a ﬂip s ∈ {± 1 } .
The irreducible representations with frequency k > 0 are similar to those for SO(2) , only with the additional ﬂip:
[Equation]
Here, s is understood as a map acting on R 2 either as the identity or by ﬂipping along a certain axis.
In contrast to SO(2) , there are now two irreducible representations for k = 0 , namely the trivial representation ψ 0 ,
( r, s ) = 1 and the representation ψ 1 , 0 ( r, s ) =
s .
The solutions for all possible combinations of these irreducible representations are as follows:
Out
In ψ 0 ,
[Equation]
K. R EFLECTION GROUP ±
The reﬂection group has only two irreducible representations: the trivial representation ψ 0 and the representation ψ 1 ( s )
= s .
Both are one-dimensional, so all the PDOs are only 1 × 1 matrices:
Out
In
[Equation]
To get the full basis, µ needs to take on all natural numbers (in practice, we use all µ up to some maximum value).
Note that we assume that the reﬂection is with respect to the x 1 -axis, both here and later for D N .
To get other reﬂection axes, the PDOs simply need to be rotated.
K. C YCLIC GROUP C N
The irreducible representations of C N are the same as those of SO(2) but only up to a frequency of k = (cid:98) N − 1 2 (cid:99) .
If N is even, there is an additional one-dimensional irreducible representation, namely
[Equation]
We then get the following solutions, where t ranges over Z and ˆ t over N :
[Table] [Equation]
K. D IHEDRAL GROUP D N
Similarly to C N , the irreducible representations of D N are the same as those of O(2) up to a frequency of k = (cid:98) N − 1 2 (cid:99) .
If N is even, there are two additional one-dimensional irreducible representations, namely
[Equation]
The solutions are (again with t ∈ Z and ˆ t ∈ N ): Out
[Equation]
(cid:0)
[Equation]
T
[Equation]
ψ
[Equation] (cid:0) [Equation] tN (cid:1) (cid:0) ˜
T ˆ tN (cid:1) (cid:16) ˜
U (ˆ t +1 / 2) N (cid:17) (cid:16) ˜ T (ˆ t +1 / 2) N (cid:17) (cid:0) ˜ T n + tN ˜ U n
[Table]
L D ISCRETIZATION METHODS FOR PDO S L. F INITE DIFFERENCES
Finite difference methods are common in machine learning; for example, the discretization of ddx as [ − 1 1] or [ − 1 0 1] , or of d 2 dx 2 as [1 − 2 1] all use ﬁnite difference methods.
To understand where these ﬁlters come from, we need the following well-known result:
Proposition 24.
Let x 1 , . . .
, x N ∈ R be arbitrary but distinct grid points.
Then for m ≤ N − 1 , there are unique coefﬁcients w ( m ) n such that the approximation
[Equation]
has an error
O ( h N − m ) for any f ∈ C N ( R ) .
The coefﬁcients w ( m ) n are called ﬁnite difference coefﬁcients and approximating derivatives using
Eq.
(177) is the ﬁnite difference method.
We will soon describe how to generalize this to higher dimensions as well.
We remark that O ( h N − m ) is an asymptotic upper bound on the error, and it can sometimes be lower, even for all f .
For example, the central difference discretization of d 2 dx 2 as [1 − 2 1] uses
N = 3 grid points but still achieves an error of O ( h 2 ) , rather than O ( h ) .
For details on when such a “boosted” order of accuracy occurs, see (Sadiq & Viswanath, 2014).
Note that Eq.
(177) can be generalized to
[Equation]
This follow immediately because the coefﬁcients w ( m ) n don’t depend on the function f , so we can apply Proposition 24 to τ − x f .
Particularly interesting for us is the case of a regular grid.
We can use inﬁnitely many grid points x n ∈ Z as long as we demand that w ( m ) n is zero for almost all n .
Then we get
[Equation]
Fixing h = 1 , this is exactly the cross correlation
[Equation]
if we interpret w ( m ) as a function n (cid:55)→ w ( m ) n .
This is why, in the end, we discretize a derivative by convolving with some stencil, such as [1 − 2 1] , at least on a regular 1D grid.
Generalizing Proposition 24 to higher dimensions does not work in a straightforward way.
However, if we restrict ourselves to regular grids, then ﬁnite difference methods can be easily applied to PDOs.
The idea is very simple: a PDO such as ∂ x ∂ 2 y can be interpreted as ﬁrst applying ∂ 2 y and then ∂ x (or the other way around).
So we discretize each of these with the one-dimensional ﬁnite difference method described before, and then we convolve with both ﬁlters one after the other.
8 We can also combine the two one-dimensional ﬁlter into one two-dimensional ﬁlter, the outer product of the two.
The asymptotic error of this discretization will simply be the highest asymptotic error along all the dimensions, so we get similar guarantees.
L. RBF-FD
As mentioned, Proposition 24 does not directly generalize to higher dimensions.
So to discretize a
PDO on arbitrary point clouds in higher dimensions, a somewhat different approach is needed.
RBF-FD is one such method and works as follows: we still want to approximate a derivative using
[Equation]
similar to ﬁnite difference methods.
Here, x n ∈ R
d are arbitrary (but again distinct) points.
The idea is now that we require this approximation to be exact if f ( x ) = ϕ
( (cid:107) x − x n (cid:107) ) , where ϕ is an arbitrary but ﬁxed radial basis function.
In words, the approximation should become exact for a certain radial basis function centered on any of the points x n .
This leads to a linear system, which is solved for the coefﬁcients w αn .
For more details on both ﬁnite differences and RBF-FD, see for example Fornberg & Flyer (2015).
L. G AUSSIAN DERIVATIVES
Discretizing PDOs using derivatives of Gaussians is very simple to describe: given grid points x n ∈ R d , we approximate using
[Equation]
where G ( x ; σ ) is a Gaussian kernel with standard deviation σ centered around 0.
σ is a free parameter; larger σ will lead to a stronger denoising effect.
On regular grids, this again turns into a cross-correlation, with the ﬁlter being the derivative ∂ α G ( x n ; σ ) evaluated on the grid coordinates.
In Appendix E we brieﬂy touch on a possible interpretation of this discretization method using the distributional framework for PDOs.
M R ELATION TO PDO- E C ONVS
In this section, we describe how PDO-eConvs (Shen et al., 2020)
ﬁt into the framework of steerable
PDOs.
We mostly follow the original notation from Shen et al.
(2020) when describing PDO-eConvs but do make some minor changes to avoid clashes and confusion with our own notation.
As in our presentation, Shen et al.
(2020) use polynomials to describe PDOs.
One difference is that they never explicitly use matrices of polynomials, because they model the feature space somewhat differently (which we will discuss in a moment).
They write H for the polynomial describing a PDO (where we would write e.g. p ) and write χ ( A ) for the corresponding PDO transformed by A ∈ O( d ) .
In our notation,
[Equation]
PDO-eConvs use two types of PDO layers.
The ﬁrst one, Ψ , can be interpreted as a steerable PDO with ρ in trivial and ρ out regular.
It maps the scalar input to the network to the internally used regular representation.
The second layer type, Φ maps between regular representations and is used for hidden layers.
At the end, pooling is performed to obtain a scalar output again.
The ﬁrst layer type is deﬁned as
[Equation]
Here, ˜ E ( d ) := R d (cid:111) S with S
≤ O( d ) ; in practice, S needs to be a ﬁnite subgroup, i.e. C N or D N .
Elements of ˜ E ( d ) can be uniquely written as ( x, A ) with x ∈ R d and A ∈ S .
There is an obvious bijection C ∞ ( ˜ E ( d ))
∼ = C ∞
( R d , R c ) , where c := | S | is the order of S , i.e. the number of group elements.
Concretely, we deﬁne
[Equation]
where A 1 , . .
, A c is an enumeration of the group elements of S .
We will therefore interpret
C ∞ ( ˜ E ( d )) as a c -dimensional feature ﬁeld over R d , and we will show that using regular representations for this ﬁeld (and trivial ones for the input) makes the PDO-eConv layers equivariant and thus steerable PDOs.
First, note that under the Θ bijection, the ﬁrst PDO-eConv layer type Ψ becomes a c × 1 matrix of
PDOs, namely
[Equation]
What we mean by this is that the diagram
[Equation]
commutes.
Concretely, we have
[Equation]
So we need to check whether H Ψ satisﬁes the PDO steerability constraint for trivial to regular PDOs:
[Equation]
Using the deﬁnition of ρ regular ( A ) , we can rewrite the RHS as
[Equation]
[Equation]
Here, we use basis vectors e A 1 , . . .
, e A c for R c , with the same enumeration
A 1 , . .
, A c of S used to deﬁne Θ .
For the ﬁnal step, we reparameterized the sum with A l := AA k .
The LHS of Eq.
(188) can be written as
[Equation]
This is the same as the ﬁnal term in Eq.
(189), which proves that the PDO steerability constraint is satisﬁed.
The second PDO-eConv layer type, mapping between regular representations, is deﬁned as
[Equation]
χ ( A )
A j are c different PDOs and they act on e ∈ C ∞ ( ˜ E ( d )) by acting on each of the c components separately.
Under the Θ bijection, Φ becomes
[Equation]
We can therefore represent it as a c × c PDO D ( H Φ ) with
[Equation]
[Equation]
where H A − 1 i
A j is the polynomial that induces χ ( I )
A − 1 i A j .
This makes the diagram
[Equation]
commute, similar to the case discussed above.
So again, we need to check that H Φ satisﬁes the PDO steerability constraint, this time for ρ in and ρ out both regular:
[Equation]
Writing out H Ψ in its components, this becomes
[Equation]
The ﬁrst and the last term are the same, just written out in components on the RHS, so the steerability constraint is again satisﬁed.
For (1), we used that ρ regular ( A ) is orthogonal, and thus
[Equation]
(2) was again a reparameterization of the sum, with AA
i (cid:55)→
A i and AA j (cid:55)→ A j .
The other steps are only simpliﬁcations and plugging in deﬁnitions.
In conclusion, we have shown that there is a simple bijection between the feature spaces used for
PDO-eConv hidden layers and the feature ﬁelds we use, and that under this bijection, PDO-eConvs correspond to steerable PDOs with regular representations (and trivial representations for the input).
It is relatively easy to adapt the argument we present for the converse direction: every equivariant
PDO between two regular feature ﬁelds (or from a scalar to a regular one) can be interpreted as a
PDO-eConv layer.
N A DDITIONAL EXPERIMENTAL RESULTS N. F LUID FLOW PREDICTION
While our classiﬁcation experiments on MNIST-rot and STL-10 make use of various representations inside the networks, the input and output are always trivial. To showcase the ﬂexibility of the steerable PDO framework, we also conduct a small experiment in ﬂuid ﬂow prediction, where the output contains a vector ﬁeld. We use the dataset provided by Ribeiro et al.
It contains simulated laminar ﬂuid ﬂows around various objects.
The inputs are all scalar ﬁelds describing the geometry of the problem.
The outputs are the velocity and pressure ﬁeld, the former of which is a vector ﬁeld, and thus requires vector representations to get the correct equivariance (rather than only trivial and regular ones like
PDO-eConvs support).
In the original dataset, the ﬂuid always ﬂows into a tube from the same direction.
To better demonstrate the effects of equivariance, we randomly rotate each sample by a multiple of π 2 , making the task somewhat more challenging.
During training, the rotations are
[Table]
different in each epoch to avoid disadvantaging the non-equivariant networks (i.e. we effectively use data augmentation).
To ensure all inputs have the same shape, we pad the original samples with zeros to make them square before rotating them.
Our architecture and hyperparameters follow those of Ribeiro et al.
The only exception is that we use a single decoder for the vector ﬁeld (since we treat it as a single vector-valued output for the purposes of equivariance), whereas Ribeiro et al.
(2020) used separate decoders for the two vector components in some of their experiments.
Just as in the STL-10 experiments, we perform no hyperparameter tuning and use the hyperparameters that Ribeiro et al.
(2020) optimized for the non-equivariant network.
To make the network equivariant, we replace the usual convolutions with
C 8 -equivariant steerable kernels or PDOs.
We chose the channel sizes such that all networks had approximately the same number of parameters (slightly over 800k).
Table 3 shows a clear advantage of the equivariant networks over the non-equivariant ones.
11 Steerable
PDOs perform slightly worse than steerable kernels, though the difference is within the error intervals.
They still perform clearly better than any non-equivariant method.
The PDO results are based on
Gaussian discretization, since that performed best in our other experiments.
N. E QUIVARIANCE ERRORS
To check the equivariance error—and indirectly the discretization error, since at least equivariant layers have zero equivariance error in the continuous setting—, we checked how much rotating an input image changes the output of a layer, compared to what the output should be under perfect equivariance.
The challenge here is that rotating a discrete image itself introduces some errors.
To minimize those, we used a large high-dimensional image, rotated it, and then scaled it down before passing it into the layer, and scaled down again after that.
We compared the result of this procedure to what we get by ﬁrst downscaling, then applying a convolutional or PDO layer, then rotating, and then downscaling again.
Effectively, all rotations thus happen at large resolutions, which should minimize artifacts.
Table 4 shows the relative equivariance errors (as multiples of 1e-6).
These errors are for randomly initialized layers (averaged over 10 initializations).
As discussed in the main text, the asymptotic error bound for ﬁnite difference discretization does not lead to a particularly low error in practice.
N. R ESTRICTION EXPERIMENTS
Table 5 shows additional results on MNIST-rot.
The general architecture and hyperparameters are the same as in the experiments in Section 4 with regular representations, using our basis.
However, in the experiments in this section, the ﬁrst ﬁve layers are D 16 -equivariant, while the ﬁnal PDO/convolutional layer is C 16 -equivariant.
The motivation for this is that while the input images do not have global reﬂectional symmetry, such symmetry occurs on smaller scales, so that stronger equivariance in earlier layers might be helpful.
However, we don’t observe clear improvements over pure C 16 -equivariance.
A reason could be that even the C 16 -equivariant networks are already very parameter efﬁcient compared to classical CNNs, so that parameter efﬁciency and equivariance are not a bottleneck anymore.
It is also possible that the architecture would need to be adapted slightly to proﬁt from the D 16 -equivariant layers. [Table] [Table] [Figure]
N. S TENCIL IMAGES
Figure 4 contains examples of stencils used during the MNIST-rot experiments.
For the 3 × 3 stencil, the different methods yield qualitatively similar results (though FD and RBF-FD have fewer stencils that make use of the four corners).
But for 5 × 5 stencils, kernels and Gauss discretization make signiﬁcantly more use of the outer stencil points than FD and RBF-FD.
[Table]
O D ETAILS ON EXPERIMENTS O. MNIST- ROT EXPERIMENTS
For the MNIST-rot experiments, we use an architecture similar to one from (Weiler & Cesa, 2019).
Table 6 contains a listing of all the layers.
Each conv block consists of a 3 × 3 or 5 × 5 steerable layer, either convolutional or a PDO, followed by batch-normalization and an ELU nonlinearity.
The output ﬁelds are the number of C 16 -regular feature ﬁelds that is used; in the case of the Vanilla CNN and for quotient representations, the number of ﬁelds is adjusted so that the parameter count is approximately preserved.
For the quotient experiments, we use 5 ρ regular ⊕ 2 ρ C 16 /C 2 quot ⊕
2 ρ C 16 /C 4 quot ⊕ 4 ρ trivial as the representation, where the numbers are scaled to reach the same parameter count as the model with only regular representations.
This combination of representations is the same one used by Weiler & Cesa (2019) and we refer to their appendix for motivation on why we need to combine different representations when using quotients.
We trained all MNIST-rot models for 30 epochs with Adam (Kingma & Ba, 2015) and a batch size of 64.
The training data was normalized and augmented using random rotations.
For the ﬁnal training runs, we used the entire set of 12k training plus validation images, as is common practice on
MNIST-rot.
The initial learning rate was 0.05, which was decayed exponentially after a burn-in of 5 epochs at a rate of 0.7 per epoch.
We used a dropout of 0.5 after the fully connected layer, and a weight decay of 1e-7.
These hyperparameters are based on those used in (Weiler & Cesa, 2019); the main difference is that we use another learning rate schedule, which works better than the original one for all models.
For the Gaussian discretization models, we use a standard deviation of σ = 1 for 3 × 3 stencils and σ = 1 . 3 for 5 × 5 stencils; we chose these values by visual inspection of the stencils, with the aim that full use is made of the stencil (see Appendix N).
The RBF-FD discretization uses third-order polyharmonic basis functions, i.e. ϕ ( r ) =
r 3 .
O. STL-10 EXPERIMENTS
For the STL-10 experiments, we used exactly the same architecture and hyperparameters as Weiler &
Cesa (2019), which in turn are essentially those of DeVries & Taylor (2017).
This means we train a Wide-ResNet-16-8 (Zagoruyko & Komodakis, 2016) for 1000 epochs, with SGD and Nesterov momentum of 0.9, a batch size of 128 and weight decay of 5e-4.
We begin with a learning rate of 0.1 and divide it by 5 after 300, 400, 600 and 800 epochs.
For data augmentation, we pad the image by 12 pixels, then randomly crop a 96 × 96 pixel patch, randomly ﬂip horizontally, and apply
Cutout (DeVries & Taylor, 2017) with a cutout size of 60 × 60 pixels.
O. C OMPUTATIONAL REQUIREMENTS
We performed our experiments on an internal cluster with a GeForce RTX 2080 Ti and 6 CPU cores.
A single run of an MNIST-rot model took about 12 minutes and a run of the STL-10 model about 5.5h.
Training the ﬂuid ﬂow prediction model took about 45 minutes.
Multiplying this by the number of experiments we did and by six runs with different seeds, the MNIST-rot results took abouthours to produce, the STL-10 results about 264 hours, and the ﬂuid ﬂow results about 18 hours.
The initial tests we did to debug and ﬁnd a good learning rate schedule (on MNIST-rot) took much less time than that.
So we estimate that producing this paper took around 350 GPU-hours on the GeForce RTX 2080 Ti.