On Non-Linear operators for Geometric Deep Learning
Anonymous Author(s)
AffiliationAddress
Abstract
This work studies operators mapping vector and scalar fields defined over a manifold M , and which commute with its group of diffeomorphisms Diff ( M ) .
Weprove that in the case of scalar fields L pω ( M , R ) , those operators correspond topoint-wise non-linearities, recovering and extending known results on R d .
Inthe context of Neural Networks defined over M , it indicates that point-wise nonlinear operators are the only universal family that commutes with any group ofsymmetries, and justifies their systematic use in combination with dedicated linear operators commuting with specific symmetries.
In the case of vector fields
L pω ( M , T M ) , we show that those operators are solely the scalar multiplication.
Itindicates that Diff ( M ) is too rich and that there is no universal class of non-linearoperators to motivate the design of Neural Networks over the symmetries of M .
Introduction
Given a physical domain M and measurements f : M → Y observed over it, one is often interestedin processing intrinsic information from f , i.e. consistent with the symmetries of the domain. Let
M denote an operator, it can be seen as a non-linear operator acting on measurements.
In words,if two measurements f , ˜ f
= g.f are related by a symmetry g of the domain, like a rigid motionon an observed molecular compound, we would like our processed data M ( f ) and M ( ˜ f ) to berelated by the same symmetry — thus that M ( g.f ) = g.M ( f ) or equivalently that M commuteswith the symmetry transformation of the domain.
The study of operators that satisfy such symmetryconstraints has played a long and central role in the history of physics and mathematics, motivatedby the inherent symmetries of physical laws.
More recently, such importance has also extended tothe design of machine learning systems, where symmetries improve the sample complexity [25, 3].
For instance, Convolutional Neural Networks build translation symmetry, whereas Graph Neural
Networks build permutation symmetry, amongst other examples coined under the ‘Geometric Deep
Learning’ umbrella [5, 4].
Lie groups of transformations are of particular interest, because there exists a precise and systematicframework to build such intrinsic operators.
Indeed, for a locally compact group G , it is possible todefine a Haar measure which is invariant to the action of G [2]; then a simple filtering along the orbitof G allows to define a class of linear operators that commute with the group action.
Examples oflocally compact groups are given by specific Lie groups acting on R d , such as the translations or therotations O d ( R ) .
Often these Lie groups G only act on a manifold M , and one tries to average alongthe orbit induced by G .
Note that it is possible, beyond invariance, to linearize more complex groupsof variability like diffeomorphisms Diff ( M ) [7].
While the description of such linear intrinsic structures is of central mathematical importance andforms the basis of Representation theory [28], in itself is not sufficient to bear fruit in the contextof Representation learning using Neural Networks [12].
Indeed, linear operators do not have thecapacity to extract rich information needed to solve challenging high-dimensional learning problems.
It is therefore necessary to extend the systematic construction and classification of intrinsic operatorsto the non-linear case.
With that purpose in mind, our work aims at studying the class of ( non-linear )
operators M whichcommute with the action of the group Diff ( M ) , the diffeomorphisms over M .
This approach willlead to a natural class of non-linear intrinsic operators.
Indeed, any group G of symmetries is, bydefinition, a subgroup of Diff ( M ) , and thus commutes with such M [24].
Consequently, obtaining anon-linear invariant to a symmetry group G could be done by using a cascade of interlacing non-linearoperators which commute with Diff ( M ) and linear operators which commute with G .
A notable example of linear operators that are covariant to the Lie group of translations is a given bythe convolutions along the orbit of the group.
These can be constructed thanks to the canonical Haarmeasure [30].
However, such an approach fails for infinite dimensional groups, like our object ofinterest: contrary to Lie groups, Diff ( M ) is not locally compact and it is thus not possible to define
a
Haar measure on this group.
Our first contribution is to demonstrate that the non-linear operators which act on vector fields(elements of L pω ( M , T M ) ) and which commute with the group of diffeomorphisms, are actuallyjust scalar multiplications.
This implies that Diff ( M ) is too rich to obtain non-trivial operators.
Our second contribution is to demonstrate that non-linear operators acting on signals in L pω ( M , R )are pointwise non-linearities.
This fills a gap in the results of [7], and a fortiori justifies the use ofpoint-wise non-linearities in geometric Deep Learning [4].
Let us remark that the study of equivariant operators that take as input vector fields is motivated bythe use of Neural Networks in physics, in particular for dynamical systems such as fluid dynamics [8].
For example, one subject of interest in hydrodynamics is how a vector field of velocities evolves; thetime evolution of such field is described by a partial differential equation (PDE), the Navier-Stokesequations, in which Neural Networks found recent applications and it is more generally the case ofother PDE [29].
Our paper is structured as follows:
Sec. 2 introduces the necessary formalism, that we use throughthis paper: in particular, we formally define the action of diffeomorphism.
Then, we state and discussour theorems in Sec. 3.1 and sketch their proofs in Sec. 3.2.
Rigorous proofs of each statement canbe found in the Appendix.
Problem Setup
2.1 Related work and motivation
In this section, we discuss the notion of intrinsic operators, invariant and covariant
non-linear operatorsand linear representation over standard symetry groups.
Then, we formally state our objective.
Intrinsic Operators As discussed above, in this work we are interested in intrinsic operators
M : L p ( M , E ) → L p ( M , E ) , where M is a Riemannian manifold, and E = R or E = T M ,capturing respectively the setting of scalar signals and vector fields over M .
L p ( M , R ) is the spaceof scalar function f : M → R which p -th power is integrable, similarly L p ( M , T M ) is the space ofsections of the tangent bundle of M (denoted T M ), f : M → T M , which norm ∥ f ∥ :
M → R isin L p ( M , R ) . Here the notion of ‘intrinsic’ means that M is consistent with an equivalence classinduced by a symmetry group G in L p ( M , E ) : if f, ˜
f ∈
L p ( M , E ) are related by a transformationg ∈ G (in which case we write f = g. ˜ f ), then M ( f ) =
g.M ( ˜ f ) .
Naturally, a stronger equivalence class imposes a stronger requirement towards M , and consequently restrains the complexity of M .
We now describe the plausible techniques used to design such operators M .
GM-Convolutions
The notion of GM -convolutions [32] is an example of linear covariant operatorswhich commute with the reparametrization of a manifold.
In practice, this implies that the weightsof a GM -convolution are shared and the action of GM -convolutions is local – two properties thatfacilitate implementation and point out the similarity with Lie groups.
Another example of symmetrygroup corresponds to the isometry group of a Riemaniann manifold, whose pushforward preserves thetensor metric.
In this case, it is well known that isometries [31] are the only diffeomomorphism whichcommute with a manifold Laplacian.
Thus, any linear operators which commute with isometries isstabilized by Laplacian’s eigenspaces.
However, little is known on the non-linear counterpart of thesymmetry-covariant operators.
In this work, we characterize non-linear operators which commutewith Diff ( M ) .
We will see that such operators are intrinsically defined by Diff ( M ) and could becombined with any linear operators covariant with a symmetry group G .
Non-linear operators It has been shown that Convolutional Neural Networks are dense in theset of non-linear covariant operators [33].
The recipe of the corresponding proof is an extensionof the proof of the universal approximation theorem [14].
The Scattering Transform [6, 23] is alsoan example of a well-understood non-linear operator which corresponds to a cascade of complexwavelet transforms followed by a point-wise modulus non-linearity.
This representation provablylinearizes small deformations.
Compact Lie Groups
In the context of geometric Machine Learning [5], there are several relevantnotions of equivalence.
For instance, we can consider a compact Lie Group G acting on M , and anassociated representation in F = { f : M → R } : Given g ∈ G and f ∈ F , then g.f ( x ) ≜
f ( g − 1 .x )for x ∈ M .
We then consider f ∼ ˜ f , related by this group action:
˜ f
= g.f for some g ∈ G .
Theoperators M which are compatible with such group action are referred as being G -equivariant (orcovariant to the action of G ) in the ML literature [13, 4].
Such groups are typically of finite andsmall dimension, e.g. the Euclidean transformations of M = R d , with d = 2 for computer visionapplications, or d = 3 for computational biology/chemistry applications.
In this case, it is possible tocharacterize all linear intrinsic operators M as group convolutions [20], leading to a rich family ofnon-linear intrinsic operators by composing such group convolutions with element-wise non-linearoperators, as implemented in modern Neural Networks.
We highlight that stability to symetries vianon-linear operators finds useful application, in particular for flat manifolds [7].
Isometries Riemanian manifolds M come with a default equivalence class, which is given byisometries.
T u M denotes the tangent vector space of M at point u ∈ M . If m u : T u M× T u M → Rdenotes the Riemannian metric tensor at point u ∈ M , a diffeomorphism ψ : M → M is an isometryif g u ( v, w ) =
g ψ ( u ) ( dψ u ( v ) , dψ u ( w )) for any u ∈ M and v, w ∈ T u M .
In words, isometries arechanges of variables that preserve the local distances in the domain.
The ensemble of all isometriesforms a Lie Group which is locally compact [26].
In this case, one can also build a rich classof intrinsic operators by following the previously explained ‘blueprint’, namely composing linearintrinsic operators with element-wise non-linearities.
As a representative example, the LaplaceBeltrami operator of M only depends on intrinsic metric properties [31]: as said above, isometriespreserve the invariant subspaces of a Laplacian.
Beyond Isometries While isometries are the ‘natural’ transformations of the geometric domain,they cannot express high-dimensional sources of variability; indeed, if M is a d -dimensional completeconnected Riemannian manifold, its isometry group has dimension at most d ( d + 1) / 2 [10].
Thisraises the question whether one can characterize intrinsic operators relative to a broader class oftransformations.
Another class of important symmetries corresponds to the ones which are gaugeinvariant, i.e. which leads to transformations which preserve the change of parametrization and whichare used in [11, 32] through the notion of G -structure.
In this work, we consider the class of transformations given by Diff ( M ) , the diffeomorphismsover M .
As shown in the Appendix, compactly supported deformations ψ : M → M definebounded linear operators L ψ acting on L p ( M , E )
→ L p ( M , E ) , and constitute a far broader class of transformations than isometries.
Our proof is mainly based on the use of compactly supporteddiffeomorphisms.
Our objective is to characterize the (non-linear) operators M such that
[Equation]
In other words, we aim to understand continuous operators M that commute with deformations.
We will show that such operators are act locally and that they can be descriped explicitly, with simple formula.
The commutation condition is visualized in the following diagram:
[Equation]
Notations
We will now formally introduce the mathematical objects of interest in this document.
Let ( M , g )be an orientable, connected, Riemannian manifold, of finite dimension d ∈ N ∗ . Let T M denotethe tangent bundle of M , i.e. the union of tangent spaces at points u ∈ M .
T ∗ M is the cotangentbundle of M . g ∈ Γ( T ∗ M ⊗ T ∗ M ) is a section of symmetric definite positive bilinear forms on thetangent bundle of M .
It is common to denote Γ B the collection of sections of a bundle B ; (cid:86) n T ∗ Mfor n ≤ d is the bundle of n -linear alternated forms of M , and Γ( (cid:86) n T ∗ M ) is the space of sectionof this vector bundle over M .
For A ⊆ M , we denote A its closure; 1 A is the indicator function of A , i.e. which takes value 1if x ∈ A and 0 otherwise.
B ( u, r ) denotes the ball of radius r around u ∈ M .
Any two vectorsv, v 1 ∈ V in a pre-Hilbert space (with a scalar product ⟨ , ⟩ ) are orthogonal, denoted v ⊥ v 1 , when⟨ v, v 1 ⟩ = 0 .
Fix p ∈ [1 , + ∞ [ .
Any volume form ω ∈ Γ( (cid:86) d T ∗ M ) defines a (positive) measure on the orientable
Riemannian manifold M ; the total volume of M is ω ( M ) : = (cid:82) M 1 dω . Let us define L pω ( M , T M ) ,the space of L p vector fields, defined as the subspace of measurable functions f : M → T M suchthat f ( u ) ∈ T u M almost everywhere and
[Equation]
We will also consider L pω ( M , R ) the space of measurable scalar functions (fields) f : M → R thatfulfill
[Equation]
We may write ∥ · ∥ instead of ∥ · ∥ p when there is no ambiguity.
For a C ∞ diffeomorphismϕ ∈ Diff ( M ) , we will consider the action of L ϕ : L pω ( M , T M )
→ L pω ( M , T M ) which we definefor for any f ∈ L pω ( M , R ) as
[Equation]
Note that this action is contravariant:
[Equation]
For scalar function f ∈ L pω ( M , R ) , we define the action of ϕ via L ϕ f ( u ) ≜
f ( ϕ ( u )) .
Let A be a measurable set of M and f ∈ L p ( M , E ) , f 1 A is the product of f with 1 A , i.e. f 1 A isequal to f on A and 0 elsewhere.
In what follows we introduce ’constant’ fields over an open set,they are denoted c 1 U with U an open subset of M . For scalar fields, a ’constant’ scalar field f ( u )is equal to the same constant c ∈ R for any u ∈ U . On the other hand, ’constant’ vector fields f 1 Uare vector fields over U for which there is a chart from U to an open subset of R d , in which for anyu ∈ U f ( u ) is equal to a constant vector c ∈ R d ; in the vector case we say that the vector field f 1 Ucan be straightened.
This latter operator is also contravariant.
If there is no ambiguity, we will use the same notation L ϕ , whether we apply it to L pω ( M , R ) or L pω ( M , T M ) .
We might sometimes refer to L pω ( M , R )or L pω ( M , T M ) as L p ( M , R ) or L p ( M , T M ) .
Throughout the article we restrict ourselves to ϕsuch that L ϕ is a bounded operator.
Write supp ( ϕ ) =
{ u, ϕ ( u ) ̸ = u } for the support of ϕ and saythat ϕ has a compact support if supp ( ϕ ) is compact.
We denote by Diff c ( M ) ⊂
Diff ( M ) the set ofcompactly supported diffeomorphisms.
Recall that since M is second-countable, C ∞ c ( M ) is dense in
L pω ( M , R ) and C ∞ c ( M , T M ) is dense in L pω ( M , T M ) .
Finally, denote by O d ( R ) the set of unitaryoperators on R d .
Throughout the article, we might not write explicitly that equalities hold almosteverywhere, since this is the default in L p spaces.
As mentioned earlier, compactly supported diffeomorphisms lead to continuous operators, which ismade rigorous by the following lemma whose proof is in the appendix.
Lemma 1.
If supp ( ϕ ) is compact, then L ϕ is bounded.
Main theorems
In this section we present our main results.
We first show that any (non-linear) deformationequivariant operator acting on scalar fields must be point-wise (Theorem 1), and then establishthat any deformation-equivariant operator acting on vector fields corresponds to a multiplication by ascalar (Theorem 2).
3.1 Theorem statements
Now, we are ready to state our two main theorems:
Theorem 1 (Scalar case) .
Let M be a connected and orientable manifold of dimension d ≥ 1 .
We consider a Lipschitz continuous operator M : L pω ( M , R )
→ L pω ( M , R ) , where 1 ≤ p < ∞ .
Then, ∀ ϕ ∈ [Equation] ( M [Equation]
ML ϕ
[Equation] L ϕ M is equivalent to the existence of a Lipschitz continuous function ρ : R → R that fulfills
[Equation]
In that case, we have ρ (0) =
if ω ( M ) = ∞ .
Theorem 2 (Vector case) .
Let M be a connected and orientable manifold of dimension d ≥ 1 .
We consider a continuous operator M :
L pω ( M , T M )
→ L pω ( M , T M ) , where 1 ≤ p < ∞ .
Then, ∀ ϕ ∈ Diff ( M ) : ML ϕ
=
L ϕ M is equivalent to the existence of a scalar λ ∈ R such that [Equation]
We highlight that our theorems are quite generic in the sense that they apply to the manifolds usuallyused in applications or theory, R d in particular.
Remark 1.
The scalar case allows to recover standard operators which are exploited for Deep
Neural Networks architectures.
However, Theorem 2 indicates that the group of diffeomorphism istoo rich to obtain non-trivial non-linear operators.
Remark 2.
The case p = ∞ leads to different results.
For instance,
Remark 3.
The condition “ ω ( M ) =
∞
=
⇒ ρ (0) = 0 ” in Theorem 1 is necessary, since in thecase M = R , the operator Mf ( x ) ≜ e if ( x ) is not in L pω ( M , R ) .
Remark 4.
The Lipschitz condition in Theorem 1 is crucial, otherwise, Mf ( x ) =
ρ ( f ( x )) might notbe an operator of L pω ( M , R ) .
For instance, if p = 2 , M = [0 , 1] and Mf ( x ) = (cid:112) f
( x ) , we see thatin this case, let f ( x ) = x , then f ∈ L pω ( M , R ) and Mf ̸∈ L pω ( M , R )
Remark 5.
If M is not Lipschitz, we can find an example which is not even continuous.
The following example holds in both cases, the scalar case and the vector case.
In both cases f ∈ L p ( M, R ) , the only thing that changes is the action of L ϕ on f .
M = R , let for all f ∈ L p ( M, R ) :
[Equation]
It is a measurable function.
Let us show that this M is a counterexample to the vector case: for anyϕ ∈ Diff ( M ) and x ∈ R , one has
[Equation]
However, M is not continuous as changing any function to 0 on Q does not change its norm butchanges the set where the limits exists.
More precisely let c > 0 be a strictly positive scalar, M [ c ]
= c ;let f = c 1[ x / ∈ Q ] , M [ f ] = 0 as { z, ∃ lim y
→ z f ( ϕ ( y )) } = ∅ .
However c = f almost everywherebut M [ c ] ̸ = M [ f ] therefore M is not continuous.
3.2 Proof Sketch
We now describe the main ideas for proving the Theorems 1 and 2.
The appendix contains completeformal arguments and technical lemmata which we omit here due to lack of space.
The two proofsshare quite some similarities despite substantially different final results.
Three ideas guide our proofs:
First, we prove that it is possible to localize M on a certain class of open sets which behaves nicelywith the manifold structure, the strongly convex sets which we denote as O 1 .
This is closely relatedto the notion of pre-sheaf [15].
Secondly, we characterize M on small open-sets.
In the scalar case,we will study the representation of locally constant functions.
In the vector case, we will show thatlocally, the image M (1 U c ) of a vector field c is co-linear to c provided that U is small enough.
Wewill also show that those local properties are independent of the position on the manifold M via aconnectedness argument.
Thirdly and finally, we combine a compacity and a density argument toextend this characterization to M , which is developed in Sec. 3.3.
Throughout the presentation, wewill use the following definitions and theorems obtained from other works:
Definition 1 (Strong convexity, from [18]) .
Let O 1 be the collection of open sets which are boundedand strongly convex, i.e. such that any points p, q in such a set can be joined by a geodesic containedin the set.
Furthermore let ˙ O
= { A ∈ O 1 : ∃ B ∈ O 1 , ¯ A ⊂ B and ω ( ¯ A \ A ) = 0 } .
The intuition behind the definition of ˙ O 1 is that all of its elements are contained in a ‘security’ openset,which avoids degenerated effects on the manifold.
In particular, this allows to control the boundaryof a given open set.
Theorem 3 (theorem adapted from [17, 18]) .
(1) ˙
O 1 is a system of neighborhoods.
Any elementof O 1 is diffeomorph to R d .
Both O 1 and ˙ O 1 are stable by intersection.
Theorem 4 (Flowbox theorem, as stated in [9]) .
Let f, g ∈ C ∞ c ( M , T M ) .
For any m ∈ M withf ( m ) ̸ = [-
-]0 and g ( m ) ̸ = 0 , there exists an open set U ⊂ M and ϕ ∈ Diff ( M ) such that ϕ ( m ) =
mand L ϕ (1 U f ) = 1 ϕ ( U ) g .
We will now present some lemmata that are necessary for the proofs of theorems 1 and 2.
As a firststep, we argue that one may assume M (0) = 0 where 0 denotes the constant 0 -function.
This isbecause in the appendix we show that M (0) is a constant function C , with C = {+
+}
if ω ( M ) = ∞ .
Therefore, we may substract C from ρ and λ , leaving us with having to show the theorems only for M (0) = 0 .
Next, a key idea of the proof is to exploit the flexibility of the deformation equivariance to localisethe input, i.e. to show that the image of compactly supported functions is also compactly supported.
To do so, the following lemma provides a way of collapsing an open ball into a singleton whilemaintaining a good control on the support of the diffeomorphism.
Lemma 2 (Key lemma) .
Let ϵ > 0 .
There exists a sequence of diffeomorphisms ϕ n : R d → R d , compactly supported in B (0 , 1 + ϵ ) such that:
[Equation]
and
[Equation]
Proof.
Set ϕ n ( u ) =
f n ( ∥ u ∥ ) u , where f n (
[Equation]
, if | r | ≤ , if | r | ≥ 1 + ϵ , and f n is smoothly interpolated for | r | ∈ [1 , 1 + ϵ ] in a way that it remains nondecreasing.
It is thenclear that ϕ n fulfills the desired properties.
We will often use that if the support of ϕ ∈ Diff ( M ) is such that supp ( ϕ ) ∩ U = ∅ , then for anyf ∈ L pω ( M , R ) one has 1 U f
= L ϕ (1 U f ) .
This implies the following important lemma, for which arigorous proof can be found in the appendix:
Lemma 3.
Let U ∈ ˙
O 1 and M as in Theorem 1 or Theorem 2.
Then, for any f ∈ E , where
E =
L pω ( M , R ) or E =
L pω ( M , T M ) respectively, we have:
[Equation]
Furthermore, if U is any closed set, the same conclusion applies.
Equipped with this result, our proof will characterize the image of functions of the type c 1 U whereeither c ∈ R , or c is a vector field which can be straightened (isomorphic to a constant vector), viathe following Lemma.
In the Vector case:
Lemma 4 (Image of localized vector field) .
For M as in Theorem 2 there is U ∈ ˙
O 1 , and λ ( U ) suchthat for any f ∈ L pω ( M, TM ) :
[Equation]
Here is the scalar case:
Lemma 5 (Image of constant functions, scalar case) .
Let M as in Theorem 1.
For any U ∈ ˙
O 1 andc ∈ R , then: M ( c 1 U )
=
h ( c, U )1 U .
Furthermore, c → h ( c, U ) is Lipschitz for any U ∈ ˙
O 1 .
At this stage, we note that both representations are point-wise, and the next steps of the proofs will beidentical both for the scalar and vector cases.
The extension to L pω ( M , R ) or L pω ( M , T M ) will bedone thanks to:
Lemma 6 (Image of a disjoint union of opensets) .
Let U 1 , ..., U n ∈ O 1 and M as in Theorem 2 or
Theorem 1, s.t. ∀ i ̸
= j, U i ∩ U j = ∅ .
Then for any f ∈ L pω ( M , T M ) :
[Equation]
This lemma states that we can completely characterize M on disjoint union of simple sets.
We willthen need an argument similar to Vitali covering Lemma in order to "glue" those open sets together,which shows that simple functions with disjoint support can approximate any elements of L pω ( M , R )or L pω ( M , T M ) (we only state the lemma for L pω ( M , R ) as our proof on L pω ( M , T M ) does notnecessarily need this result):
Lemma 7 (Local Vitali) .
For f ∈ C ∞ c ( M ) and m ∈ M , there exists U ∈ ˙
O 1 with m ∈ U , such that for any ϵ > 0 , there exist subsets U 1 , ..., U n ∈ ˙
O 1 with U
i ⊂ U and numbers c 1 , ..., c n ∈ R such that: ∥
[Equation]
Note that this type of covering is not possible on any open set without further assumptions on themanifold, such as bounds on its Ricci curvature [22].
Fortunately, we will only need a local versionwhich is true because charts are locally bi-Lipschitz.
Both Lemma 6 and Lemma 7 imply that:
Proposition 1.
Consider M from either Theorem 1 or 2.
Assume that there exists U ∈ ˙
O 1 such that M ( c 1 V )
=
h ( c, V )1 V for any V ⊂ U , with V ∈ ˙
O 1 , where c is either a vector field in the case E
=
L pω ( M , T M ) or a constant scalar in the case E
=
L pω ( M , R ) .
If we further assume that c → h ( c, U ) is L -Lipschitz, then
[Equation]
Furthermore, it does not depend on U , meaning that for any other such ˜ U , we have:
[Equation]
We briefly discuss the intuition behind Theorem 2.
It is linked to the idea that the operators M athand have to commute with local rotations, and this even for locally constant vector fields.
We reducethe characterisation of deformation-equivariant vector operators using an invariance to symmetryargument: functions which are invariant to rotations are multiples of a scalar.
The intuition iscontained in the following lemma, which is commonly used in physics:
Lemma 8 (Invariance to rotation) .
Let f : R d → R d such that for any W ∈ O d ( R ) and x ∈ R d , onehas f ( Wx ) =
Wf ( x ) .
Then, there is λ : R d → R , f ( x ) = λ ( ∥ x ∥ ) x .
Proof.
We write f ( x ) =
λ ( x ) x + x ⊥ , with x ⊥ ( m ) ̸ = 0 and x ⊥ ⊥ x .
Then, we introduce W ∈ O d ( R )such that Wx ⊥ ( m ) =
− x ⊥
( m ) and Wx ( m ) =
x ( m ) .
From f ( x ) =
f ( Wx ) =
Wf ( x ) we deducethat
x ⊥ = 0 .
Next, λ ( Wx ) =
λ ( x ) thus λ ( x ) = λ ( x ′ ) for any ∥ x ∥ = ∥ x ′ ∥ .
Distinction between scalar and vector case The scalar case is simpler to handle than the vectorcase: there are several more steps for the proof of Theorem 2, one needs to show that the point-wisenon-linearity is actually a scalar multiplication.
We also highlight that the non-linearity is fullydefined by its image on locally constant functions.
Finally, we conclude the proof of the theorem by appealing to a common density argument of thefunctions smooth with compact support, combing all the lemmata we have just presented in Sec.
3.3.3.3 Proofs conclusions (common to the scalar and vector case)
In this section, we prove that the local properties of M can be extended globally on M .
The main idea is to exploit the well-known Poincaré’s formula, which states that:
[Equation]
and to localize the action of M on each U i
∩ U i 2 ∩ ...
∩ U i k ∈ ˙
O 1 thanks to Lemma 3.
Proof of Theorem 1 and Theorem 2.
Let f be a smooth and compactly supported function.
Furtherconsider ∪ i ≤ n U i a finite covering of its support with U i ∈ ˙
O 1 .
Using an inclusion-exclusion formulatogether with Lemma 3, we obtain
[Equation]
where we used that U i 1 ∩ U i 2 ∩ ...
∩ U i k ∈ ˙
O 1 .
Now, the support of f is closed and included in ∪ i
U i .
Thus using Lemma 3:
[Equation]
Note that if ρ is a pointwise operator with ρ (0) = 0 , then ρ (1 U f )
=
1 U ρ ( f ) andn (cid:88)
[Equation]
Thus, Mf = ρ ( f ) where ρ is obtained from Lemma 4 or 5 combined with Prop 1.
We conclude bydensity in L pω ( M , R ) or L pω ( M , T M ) respectively.
This ends the proof.
Remarks and conclusion
In this work, we have fully characterized non-linear operators which commute under the action ofsmooth deformations.
In some sense, it settles the intuitive fact that commutation with the wholediffeomorphism group is too strong a property, leading to a small, nearly trivial family of non-linearintrinsic operators.
While on their own they have limited interest for geometric deep representationlearning, they can ‘upgrade’ any family of linear operators associated with any group G ⊂ Diff ( M )into a powerful non-linear class — the so-called GDL Blueprint in [4].
Also, this result is a first steptowards characterizing the non-linear operators which commute with Gauge transformations andcould give useful insights for specifying novel Gauge invariant architectures.
We now state a coupleof unsolved questions and future work.
On the commutativity assumption: Several examples and approximation results [21][33] existfor operators that commute with Lie groups and discrete groups [19].
In this case, it is possible todefine a measure on the group that is invariant by the group action (called the Haar measure), whichmakes it possible to define convolutions.
Roughly, non-linear operators covariant with some actionsof those groups can be thought of as an approximation by a Group Convolution Neural Networks.
It is important to note that the inputs of the operators described in these articles are functions thattake real values; the much more general class of inputs that take values in vector bundles is, to ourknowledge, not covered in the literature.
To our knowledge, we are the first work to study the designof equivariant Neural Networks that process vector fields defined over a manifold.
In this settingeven for M = R d , it is unclear which type of non-linear operators commute with smaller groups ofsymmetry such as the Euclidean group.
In fact, a generic question holds for manifolds: for a givensymmetry group G , what is elementary non-linear building block of a Neural Network?
This couldbe, for instance, useful to design Neural Networks which are Gauge invariant.
It is an open questionfor future work which would be relevant many applications in physics [16].
Furthermore, the factthat the characterization of diffeomorphism invariant operators we exhibited in this paper is veryrestrictive opens the way for the study of other non-locally ’smaller’ compact groups; we believe thatany results in that direction are completely novel.
Example of vector operators for L ∞
It is slightly unclear how the vector case p = ∞ can behandled in our framework, yet [1] seems to have interesting insights toward this direction.
Linearization of Diff ( M )
In this work, we considered an exact commutation between operatorsand a symmetries: however, it is unclear which operators approximatively commute with a givensymmetry group.
Such operators would be better to linearize a high-dimensional symmetry grouplike Diff ( M ) .
An important instance of
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
) If your work uses existing assets, did you cite the creators?
[N/A]
Did you mention the license of the assets?
[N/A]
Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
you describe any participant risks, with links to Institutional Review
Board ( ) approvals, if applicable?
[N/A]
Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]