O N THE U NIVERSALITY OF R OTATION E QUIVARIANT P OINT C LOUD N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning functions on point clouds has applications in many ﬁelds, including computer vision, computer graphics, physics, and chemistry.
Recently, there has been a growing interest in neural architectures that are invariant or equivariant to all three shape-preserving transformations of point clouds: translation, rotation, and permutation.
In this paper, we present a ﬁrst study of the approximation power of these architectures.
We ﬁrst derive two sufﬁcient conditions for an equivariant architecture to have the universal approximation property, based on a novel characterization of the space of equivariant polynomials.
We then use these conditions to show that two recently suggested models (Thomas et al., 2018; Fuchs et al., 2020) are universal, and for devising two other novel universal architectures.
I NTRODUCTION
Designing neural networks that respect data symmetry is a powerful approach for obtaining efﬁcient deep models.
Prominent examples being convolutional networks which respect the translational invariance of images, graph neural networks which respect the permutation invariance of graphs (Gilmer et al., 2017; Maron et al., 2019b), networks such as (Zaheer et al., 2017; Qi et al., 2017a) which respect the permutation invariance of sets, and networks which respect 3D rotational symmetries (Cohen et al., 2018; Weiler et al., 2018; Esteves et al., 2018; Worrall & Brostow, 2018; Kondor et al., 2018a).
While the expressive power of equivariant models is reduced by design to include only equivariant functions, a desirable property of equivariant networks is universality: the ability to approximate any continuous equivariant function.
This is not always the case: while convolutional networks and networks for sets are universal (Yarotsky, 2018; Segol & Lipman, 2019), popular graph neural networks are not (Xu et al., 2019; Morris et al., 2018).
In this paper, we consider the universality of networks that respect the symmetries of 3D point clouds: translations, rotations, and permutations.
Designing such networks is a popular paradigm in recent years (Thomas et al., 2018; Fuchs et al., 2020;
Poulenard et al., 2019; Zhao et al., 2019).
While there have been many works on the universality of permutation invariant networks (Zaheer et al., 2017; Maron et al., 2019c; Keriven & Peyr´e, 2019), and a recent work discussing the universality of rotation equivariant networks (Bogatskiy et al., 2020), this is a ﬁrst paper which discusses the universality of networks which combine rotations, permutations and translations.
We start the paper with a general, architecture-agnostic, discussion, and derive two sufﬁcient conditions for universality.
These conditions are a result of a novel characterization of equivariant polynomials for the symmetry group of interest.
We use these conditions in order to prove universality of the prominent Tensor Field Networks (TFN) architecture (Thomas et al., 2018; Fuchs et al., 2020).
The following is a weakened and simpliﬁed statement of Theorem 2 stated later on in the paper:
Theorem (Simpliﬁcation of Theorem 2) .
Any continuous equivariant function on point clouds can be approximated uniformly on compact sets by a composition of TFN layers.
We use our general discussion to prove the universality of two additional equivariant models: the ﬁrst is a simple modiﬁcation of the TFN architecture which allows for universality using only low dimensional ﬁlters.
The second is a minimal architecture which is based on tensor product representations, rather than the more commonly used irreducible representations of SO(3) .
We discuss the advantages and disadvantages of both approaches.
To summarize, the contributions of this paper are: (1) A general approach for proving the universality of rotation equivariant models for point clouds; (2) A proof that two recent equivariant models (Thomas et al., 2018; Fuchs et al., 2020) are universal; (3) Two additional simple and novel universal architectures.
P REVIOUS WORK
Deep learning on point clouds.
(Qi et al., 2017a; Zaheer et al., 2017) were the ﬁrst to apply neural networks directly to the raw point cloud data, by using pointwise functions and pooling operations.
Many subsequent works used local neighborhood information (Qi et al., 2017b; Wang et al., 2019b;
Atzmon et al., 2018).
We refer the reader to a recent survey for more details (Guo et al., 2020).
In contrast with the aforementioned works which focused solely on permutation invariance, more related to this paper are works that additionally incorporated invariance to rigid motions.
(Thomas et al., 2018) proposed Tensor Field Networks (TFN) and showed their efﬁcacy on physics and chemistry tasks.(Kondor et al., 2018b) also suggested an equivariant model for continuous rotations.
(Li et al., 2019) suggested models that are equivariant to discrete subgroups of SO(3) .
(Poulenard et al., 2019) suggested an invariant model based on spherical harmonics.
(Fuchs et al., 2020) followed TFN and added an attention mechanism.
Recently, (Zhao et al., 2019) proposed a quaternion equivariant point capsule network that also achieves rotation and translation invariance.
Universal approximation for invariant networks.
Understanding the approximation power of invariant models is a popular research goal.
Most of the current results assume that the symmetry group is a permutation group.
(Zaheer et al., 2017; Qi et al., 2017a; Segol & Lipman, 2019; Maron et al., 2020; Serviansky et al., 2020) proved universality for several S n -invariant and equivariant models.
(Maron et al., 2019b;a; Keriven & Peyr ´ e, 2019; Maehara & NT, 2019) studied the approximation power of high-order graph neural networks.
(Maron et al., 2019c; Ravanbakhsh, 2020) targeted universality of networks that use high-order representations for permutation groups(Yarotsky, 2018) provided several theoretical constructions of universal equivariant neural network models based on polynomial invariants, including an SE (2) equivariant model.
In a recent work (Bogatskiy et al., 2020) presented a universal approximation theorem for networks that are equivariant to several Lie groups including SO (3) .
The main difference from our paper is that we prove a universality theorem for a more complex group that besides rotations also includes translations and permutations.
A FRAMEWORK FOR PROVING UNIVERSALITY
In this section, we describe a framework for proving the universality of equivariant networks.
We begin with some mathematical preliminaries:
3. M ATHEMATICAL SETUP
An action of a group G on a real vector space W is a collection of maps ρ ( g ) :
W → W deﬁned for any g ∈ G , such that ρ ( g 1 ) ◦ ρ ( g 2 )
=
ρ ( g 1 g 2 ) for all g 1 , g 2 ∈ G , and the identity element of G is mapped to the identity mapping on W .
We say ρ is a representation of G if ρ ( g ) is a linear map for every g ∈ G .
As is customary, when it does not cause confusion we often say that W itself is a representation of G .
In this paper, we are interested in functions on point clouds.
Point clouds are sets of vectors in Rarranged as matrices:
[Equation]
Many machine learning tasks on point clouds, such as classiﬁcation, aim to learn a function which is invariant to rigid motions and relabeling of the points.
Put differently, such functions are required to be invariant to the action of G = R 3 (cid:111)
SO
[Equation]
3) × S n on R 3 × n via
[Equation]
where t ∈ R 3 deﬁnes a translation, R is a rotation and P is a permutation matrix.
Equivariant functions are generalizations of invariant functions: If G acts on W 1 via some action ρ 1 ( g ) , and on W 2 via some other group action ρ 2 ( g ) , we say that a function f : W 1 → W 2 is equivariant if
[Equation]
Invariant functions correspond to the special case where ρ 2 ( g ) is the identity mapping for all g ∈ G .
In some machine learning tasks on point clouds, the functions learned are not invariant but rather equivariant.
For example, segmentation tasks assign a discrete label to each point.
They are invariant to translations and rotations but equivariant to permutations – in the sense that permuting the input causes a corresponding permutation of the output.
Another example is predicting a normal for each point of a point cloud.
This task is invariant to translations but equivariant to both rotations and permutations.
In this paper, we are interested in learning equivariant functions from point clouds into W nT , where
W T is some representation of SO(3) .
The equivariance of these functions is with respect to the action ρ G on point clouds deﬁned in equation 1, and the action of G on W nT deﬁned by applying the rotation action from the left and permutation action from the right as in 1, but ‘ignoring’ the translation component.
Thus, G -equivariant functions will be translation invariant.
This formulation of equivariance includes the normals prediction example by taking W T
= R 3 , as well as the segmentation case by setting W T
=
R with the trivial identity representation.
We focus on the harder case of functions into W nT which are equivariant to permutations, since it easily implies the easier case of permutation invariant functions to W T .
Notation.
We use the notation N + = N ∪ { 0 } and N ∗ + = (cid:83) r ∈ N N r
+ .
We set [ D ] = { 1 , . . .
, D } and [ D ] 0 = { 0 , . . .
, D } .
Proofs.
Proofs appear in the appendices, arranged according to sections.
C ONDITIONS FOR UNIVERSALITY
The semi-lifted approach In general, highly expressive equivariant neural networks can be achieved by using a ‘lifted approach’, where intermediate features in the network belong to high dimensional representations of the group.
In the context of point clouds where typically n (cid:29) 3 , many papers, e.g., (Thomas et al., 2018; Kondor, 2018; Bogatskiy et al., 2020) use a ‘semi-lifted’ approach, where hidden layers hold only higher dimensional representations of SO(3) , but not high order permutation representations.
In this subsection, we propose a strategy for achieving universality with the semi-lifted approach.
We begin by an axiomatic formulation of the semi-lifted approach (see illustration in inset): we assume that our neural networks are composed of two main components: the ﬁrst component is a family F feat of parametric continuous G -equivariant functions
f
feat which map the original point cloud R 3 × n to a semi-lifted point cloud
W n feat =
(cid:76) ni =
W feat , where W feat is a lifted (i.e., high-order) representation of SO(3) .
The second component is a family of parametric linear SO(3) -equivariant functions F pool , which map from the high order representation W feat down to the target representation
W T .
Each such SO(3) –equivariant function Λ :
W feat → W T can be extended to a SO(3) × S n equivariant function ˆΛ : W n feat → W nT by applying Λ elementwise.
For every positive integer C , these two families of functions induce a family of functions F C obtained by summing
C different compositions of these functions:
[Equation]
Conditions for universality We now describe two conditions that guarantee universality using the semi-lifted approach.
The ﬁrst step is showing, as in (Yarotsky, 2018), that continuous G -equivariant functions C G ( R 3 × n , W nT ) can be approximated by G -equivariant polynomials P G ( R 3 × n , W nT ) .
Lemma 1.
Any continuous G -equivariant function in C G ( R 3 × n , W nT ) can be approximated uniformly on compact sets by G -equivariant polynomials in P G ( R 3 × n , W nT ) .
Universality is now reduced to the approximation of G -equivariant polynomials.
Next, we provide two conditions which guarantee that G -equivariant polynomials of degree D can be expressed by function spaces F C ( F feat , F pool ) as deﬁned in equation 2.
The idea behind these conditions is that explicit characterizations of polynomials equivariant to the joint action of translations, rotations and permutations is challenging.
However, it is possible to explicitly characterize polynomials equivariant to translations and permutations (but not rotations).
The key observation is then that this characterization can be rewritten as a sum of functions to W n feat , a high dimensional representations of SO(3) which is equivariant to translations, permutations and rotations , composed with a linear map which is permutation equivariant (but does not respect rotations).
Accordingly, our ﬁrst condition is that F feat contains a spanning set of such functions to W n feat .
We call this condition
D -spanning:
Deﬁnition 1 ( D -spanning) .
For D ∈ N + , let F feat be a subset of C G ( R 3 × n , W n feat ) .
We say that
F feat is D -spanning , if there exist f 1 , . . .
, f K ∈ F feat , such that every polynomial p : R 3 × n → R n of degree D which is invariant to translations and equivariant to permutations, can be written as
[Equation]
where Λ k : W feat → R are all linear functionals, and ˆΛ k : W n feat → R n are the functions deﬁned by elementwise applications of Λ k .
In Lemma 4 we explicitly construct a D -spanning family of functions.
This provides us with a concrete condition which implies D -spanning for other function families as well.
The second condition is that F pool contains all SO(3) linear equivariant layers.
We call this condition Linear universality .
Intuitively, taking linear rotation equivariant Λ k in equation 3 ensures that the resulting function p will be rotation equivariant and thus fully G -equivariant, and linear universality guarantees the ability to express all such G invariant functions.
Deﬁnition 2 (Linear universality) .
We say that a collection F pool of equivariant linear functionals between two representations W feat and W T of SO(3) is linearly universal, if it contains all linear
SO(3) -equivariant mappings between the two representations.
When these two conditions apply, a rather simple symmetrization arguments leads to the following theorem:
Theorem 1.
If F feat is D -spanning and F pool is linearly universal, then there exists some C ( D ) ∈ N such that for all C ≥ C ( D ) the function space F C ( F feat , F pool ) contains all G -equivariant polynomials of degree ≤ D .
Proof idea.
By the D -spanning assumption, there exist f 1 , . . .
, f K ∈ F feat such that any vector valued polynomial invariant to translations and equivariant to permutations is of the form
[Equation]
While by deﬁnition this holds for functions p whose image is R n , this is easily extended to functions to W nT as well.
It remains to show that when p is also SO(3) -equivariant, we can choose Λ k to be SO(3) equivariant.
This is accomplished by averaging over SO(3) .
As a result of Theorem 1 and Lemma 1 we obtain our universality result (see inset for illustration)
Corollary 1.
For all C, D ∈ N + , let F C,D denote function spaces generated by a pair of functions spaces which are D -spanning and linearly universal as in equation 2.
Then any continuous G -equivariant function in
C G ( R 3 × n , W nT ) can be approximated uniformly on compact sets by equivariant functions in
F = (cid:91)
D ∈ N
F C ( D ) ,D .
UNIVERSALITY CONDITIONS IN ACTION
In the remainder of the paper, we prove the universality of several G -equivariant architectures, based on the framework we discussed in the previous subsection.
We discuss two different strategies for achieving universality, which differ mainly in the type of lifted representations of SO(3) they use: (i)
The ﬁrst strategy uses (direct sums of) tensor-product representations; (ii) The second uses (direct sums of) irreducible representations.
The main advantage of the ﬁrst strategy from the perspective of our methodology is that achieving the D -spanning property is more straightforward.
The advantage of irreducible representations is that they almost automatically guarantees the linear universality property.
In Section 4 we discuss universality through tensor product representations, and give an example of a minimal tensor representation network architecture that would satisfy universality.
In sectionwe discuss universality through irreducible representations, which is currently the more common strategy.
We show that the TFN architecture (Thomas et al., 2018; Fuchs et al., 2020) which follows this strategy is universal, and describe a simple tweak that achieves universality using only low order ﬁlters, though the representations throughout the network are high dimensional.
U NIVERSALITY WITH TENSOR REPRESENTATIONS
In this section, we prove universality for models that are based on tensor product representations, as deﬁned below.
The main advantage of this approach is that D -spanning is achieved rather easily.
The main drawbacks are that its data representation is somewhat redundant and that characterizing the linear equivariant layers is more laborious.
Tensor representations We begin by deﬁning tensor representations.
For k ∈ N + denote T k
= R 3 k . SO(3) acts on T k by the tensor product representation, i.e., by applying the matrix Kronecker product k times: ρ k ( R ) :
=
R ⊗ k .
The inset illustrates the vector spaces and action for k = 1 , 2 , 3 .
With this action, for any i 1 , . . .
, i k ∈ [ n ] , the map from R 3 × n to T k deﬁned by
[Equation]
is SO(3) equivariant.
A D -spanning family We now show that tensor representations can be used to deﬁne a ﬁnite set of
D -spanning functions.
The lifted representation W feat will be given by
[Equation]
The D -spanning functions are indexed by vectors (cid:126)r = ( r 1 , . .
, r K ) , where each r k is a non-negative integer.
Denoting T
= (cid:107) r (cid:107) 1 , the functions Q ( (cid:126)r ) :
R 3 × n → T nT , Q ( (cid:126)r ) =
( Q ( (cid:126)r ) j ) nj =1 are deﬁned for ﬁxed j ∈ [ n ] by
[Equation]
[Equation]
The functions Q ( (cid:126)r ) j are SO(3) equivariant as they are a sum of equivariant functions from equation 5.
Thus Q ( (cid:126)r ) is SO(3) × S n equivariant.
The motivation behind the deﬁnition of these functions is that known characterizations of permutation equivariant polynomials Segol & Lipman (2019) tell us that the entries of these tensor valued functions span all permutation equivariant polynomials (see the proof of Lemma 2 for more details).
To account for translation invariance, we compose the functions Q ( (cid:126)r ) with the centralization operation and deﬁne the set of functions
[Equation]
where ι is the natural embedding that takes each T T into W T feat =
(cid:76) DT =0 T T .
In the following lemma, we prove that this set is D -spanning.
Lemma 2.
For every D ∈ N + , the set Q D is D -spanning.
Proof idea.
It is known (Segol & Lipman, 2019)
(Theorem 2) that polynomials p : R 3 × n → R n which are S n -equivariant, are spanned by polynomials of the form p (cid:126)α = ( p j(cid:126)α ) n j =1 , deﬁned as
[Equation]
where (cid:126)α = ( α 1 , . .
. , α K ) and each α k ∈ N 3+ is a multi-index.
We ﬁrst show that these polynomials can be extracted from Q ( (cid:126)r ) and then use them to represent p .
A minimal universal architecture Once we have shown that Q D is D -spanning, we can design
D -spanning architectures, by devising architectures that are able to span all elements of Q D .
As we will now show, the compositional nature of neural networks allows us to do this in a very clean manner.
We deﬁne a parametric function f
( X, V | θ 1 , θ 2 ) which maps R 3 × n ⊕T nk to R 3 × n ⊕T
nk +1 as follows:
For all j ∈ [ n ] , we have f j ( X, V ) =
( x j , ˜ V
j
( X, V )) , where
[Equation]
We denote the set of functions ( X, V )
(cid:55)→ f
( X, V | θ 1 , θ 2 ) obtained by choosing the parameters θ 1 , θ 2 ∈ R , by F min .
While in the hidden layers of our network the data is represented using both coordinates ( X, V ) , the input to the network only contains an X coordinate and the output only contains a V coordinate.
To this end, we deﬁne the functions
[Equation]
We can achieve D -spanning by composition of functions in F min with these functions and centralizing:
Lemma 3.
The function set Q D is contained in
[Equation]
Thus F feat is D -spanning.
Proof idea.
The proof is technical and follows by induction on D .
To complete the construction of a universal network, we now need to characterize all linear equivariant functions from W T feat to the target representation W T .
In Appendix G we show how this can be done for the trivial representation W T
= R .
This characterization gives us a set of linear functions
F pool , which combined with F feat deﬁned in equation 11 (corresponds to SO(3) invariant functions) gives us a universal architecture as in Theorem 1.
However, the disadvantage of this approach is that implementation of the linear functions in F pool is somewhat cumbersome.
In the next section we discuss irreducible representations, which give us a systematic way to address linear equivariant mappings into any W T .
Proving D -spanning for these networks is accomplished via the D -spanning property of tensor representations, through the following lemma
Lemma 4.
If all functions in Q D can be written as
[Equation]
where f k ∈ F feat , A k : W feat → W T feat and ˆ A k : W n feat → ( W T feat ) n is deﬁned by elementwise application of A k , then F feat is D -spanning.
We note that as before, A k are not necessarily SO(3) - equivariant.
Proof idea.
The lemma follows directly from the assumptions.
U NIVERSALITY WITH IRREDUCIBLE REPRESENTATIONS
In this section, we discuss how to achieve universality when using irreducible representations of
SO(3) .
We will begin by deﬁning irreducible representations, and explaining how linear universality is easily achieved by them, while the D -spanning properties of tensor representations can be preserved.
This discussion can be seen as an interpretation of the choices made in the construction of TFN and similar networks in the literature.
We then show that these architectures are indeed universal.
5. I RREDUCIBLE REPRESENTATIONS OF SO (3)
In general, any ﬁnite-dimensional representation W of a compact group H can be decomposed into irreducible representations: a subspace W 0 ⊂ W is H -invariant if hw ∈ W 0 for all h ∈ H, w ∈ W 0 .
A representation W is irreducible if it has no non-trivial invariant subspaces.
In the case of SO(3) , all irreducible real representations are deﬁned by matrices D ( (cid:96) )
( R ) , called the real Wigner D-matrices, acting on W (cid:96) :
= R 2 (cid:96) +1 by matrix multiplication.
In particular, the representation for (cid:96) = 0 , 1 are
D (0) ( R ) = 1 and D (1) ( R ) = R .
Linear maps between irreducible representations As mentioned above, one of the main advantages of using irreducible representations is that there is a very simple characterization of all linear equivariant maps between two direct sums of irreducible representations.
We use the notation W l for direct sums of irreducible representations, where l = ( (cid:96) 1 , . . .
, (cid:96) K ) ∈ N K + and W l = (cid:76) K k =1 W (cid:96) k .
Lemma 5.
Let l (1) =
( (cid:96) (1)1 , . . .
, (cid:96) (1) K 1 ) and l (2) =
( (cid:96) (2)1 , . . .
, (cid:96) (2) K 2 ) .
A function Λ = (Λ 1 , . . .
, Λ K 2 ) is a linear equivariant mapping between W l (1) and W l (2) , if and only if there exists a K 1 × K 2 matrix
M with M ij = 0 whenever (cid:96) (1) i (cid:54) = (cid:96) (2) j , such that
[Equation]
where V =
( V i ) K
i
=1 and V i ∈ W (cid:96) (1) i for all
i = 1 , . .
, K 1 .
Proof idea.
This lemma is a simple generalization of Schur’s lemma, a classical tool in representation theory, which asserts that a non-zero linear map between irreducible representations is a scaled multiply of the identity mapping.
Lemma 5 was stated in the complex setting in Kondor (2018).
While Schur’s lemma, and thus Lemma 5, does not always hold for representations over the reals, we observe here that it holds for real irreducible representations of SO(3) since their dimension is always odd.
Clebsch-Gordan decomposition of tensor products As any ﬁnite-dimensional representation of
SO(3) can be decomposed into a direct sum of irreducible representations, this is true for tensor representations as well.
In particular, the Clebsch-Gordan coefﬁcients provide an explicit formula for decomposing the tensor product of two irreducible representations W (cid:96) 1 and W (cid:96) 2 into a direct sum of irreducible representations.
This decomposition can be easily extended to decompose the tensor product W l 1 ⊗ W l 2 into a direct sum of irreducible representations, where l 1 , l 2 are now vectors.
In matrix notation, this means there is a unitary linear equivariant U ( l 1 , l 2 ) mapping of W l 1 ⊗ W lonto W l , where the explicit values of l = l ( l 1 , l 2 ) and the matrix U ( l 1 , l 2 ) can be inferred directly from the case where (cid:96) 1 and (cid:96) 2 are scalars.
By repeatedly taking tensor products and applying Clebsch-Gordan decompositions to the result,
TFN and similar architectures can achieve the D -spanning property in a manner analogous to tensor representations, and also enjoy linear universality since they maintain irreducible representations throughout the network.
5. T ENSOR FIELD NETWORKS
We now describe the basic layers of the TFN architecture (Thomas et al., 2018), which are based on irreducible representations, and suggest an architecture based on these layers which can approximate
G -equivariant maps into any representation W n l T , l T ∈ N ∗ + .
There are some superﬁcial differences between our description of TFN and the description in the original paper, for more details see Appendix F.
We note that the universality of TFN also implies the universality of Fuchs et al.
(2020), which is a generalization of TFN that enables adding an attention mechanism.
Assuming the attention mechanism is not restricted to local neighborhoods, this method is at least as expressive as TFN.
TFNs are composed of three types of layers: (i) Convolution (ii) Self-interaction and (iii) Nonlinearities.
In our architecture, we only use the ﬁrst two layers types, which we will now describe: 1 .
Convolution.
Convolutional layers involve taking tensor products of a ﬁlter and a feature vector to create a new feature vector, and then decomposing into irreducible representations.
Unlike in standard CNN, a ﬁlter here depends on the input, and is a function F : R 3 → W l D , where
l D =
, D ] T .
The (cid:96) -th component of the ﬁlter F ( x )
= (cid:2) F (0) ( x ) , . .
, F ( D ) ( x ) (cid:3) will be given by
[Equation]
where ˆ x = x/ (cid:107) x (cid:107) if x (cid:54) = 0 and ˆ x = 0 otherwise, Y (cid:96)m are spherical harmonics, and R ( (cid:96) )
any polynomial of degree ≤ D .
In Appendix F we show that these polynomial functions can be replaced by fully connected networks, since the latter can approximate all polynomials uniformly.
The convolution of an input feature V ∈ W n l i and a ﬁlter F as deﬁned above, will give an output feature ˜ V
=
( ˜ V a ) na =1 ∈ W n l 0 , where l o = l
( l f , l i ) , which is given by
[Equation]
More formally we will think of convolutional layer as functions of the form f
( X, V ) =
( X, ˜ V
( X, V )) .
These functions are deﬁned by a choice of D , a choice of a scalar polynomial
R ( (cid:96) ) , (cid:96) = 0 , . . .
, D , and a choice of the parameter θ 0 ∈ R in equation 14.
We denote the set of all such functions f by F D .
Self Interaction layers.
Self interaction layers are linear functions from ˆΛ : W n l → W n l T , which are obtained from elementwise application of equivariant linear functions Λ : W l → W l T .
These linear functions can be speciﬁed by a choice of matrix M with the sparsity pattern described in Lemma 5.
Activation functions.
TFN, as well as other papers, proposed several activation functions.
We ﬁnd that these layers are not necessary for universality and thus we do not deﬁne them here.
Network architecture.
For our universality proof, we suggest a simple architecture which depends on two positive integer parameters ( C, D ) :
For given D , we will deﬁne F feat ( D ) as the set of function obtained by 2 D recursive convolutions
[Equation]
where ext and π V are deﬁned as in equation 10.
The output of a function in F feat ( D ) is in W n l ( D ) , for some l ( D ) which depends on D .
We then deﬁne F pool ( D ) to be the self-interaction layers which map W n l ( D ) to W n l T .
This choice of F feat ( D ) and F pool ( D ) , together with a choice of the number of channels C , deﬁnes the ﬁnal network architecture F TFN C,D = F C ( F feat ( D ) , F pool ( D )) as in equation 2.
In the appendix we prove the universality of TFN:
Theorem 2.
For all n ∈ N , l T ∈ N ∗ + ,
For D ∈ N + , every G -equivariant polynomial p : R 3 × n → W nT of degree D is in F TFN C ( D ) ,D . 2.
Every continuous G -equivariant function can be approximated uniformly on compact sets by functions in ∪ D ∈ N + F TFN C ( D ) ,D
As discussed previously, the linear universality of F pool is guaranteed.
Thus proving Theoremamounts to showing that F feat ( D ) is D -spanning. This is done
using the sufﬁcient condition for
D -spanning deﬁned in Lemma
Proof idea.
The proof is rather technical and involved.
A useful observation (see Dai & Xu (2013)) used in the proof is that the ﬁlters of orders (cid:96) = 0 , 1 , . . .
, D , deﬁned in equation 13, span all polynomial functions of degree D on R 3 .
This observation is used to show that all functions in Q D can be expressed by F feat ( D ) and so F feat is D -spanning, as stated in Lemma 2.
Alternative architecture The complexity of the TFN network used to construct G -equivariant polynomials of degree D , can be reduced using a simple modiﬁcations of the convolutional layer in equation 14: We add two parameters θ 1 , θ 2 ∈ R to the convolutional layer, which is now deﬁned as:
[Equation]
With this simple change, we can show that F feat ( D ) is D -spanning even if we only take ﬁlters of order 0 and 1 throughout the network.
This is shown in Appendix E.
C ONCLUSION
In this paper, we have presented a new framework for proving the universality of G -equivariant point cloud networks.
We used this framework for proving the universality of the TFN model Thomas et al.
(2018); Fuchs et al.
(2020), and for devising two additional novel simple universal architectures.
In the future we hope to extend these simple constructions to operational G -equivariant networks with universality guarantees and competitive practical performance.
Our universal architectures do not require activation functions, and use a single self-interaction layer.
In Appendix H we present an experiment indicating that the performance of TFN is not signiﬁcantly altered by these simpliﬁcations.
Our architectures also require high order representations, and our experiments show that using increasingly high order representations does indeed improve performance.
In the same time, we note that to date, practical TFN implementation included a relatively small amount of layers, and did not use very high order representations.
We believe our theoretical results will inspire interest in stable implementation of larger architectures.
On the other hand, an interesting open problem is understanding whether universality can be achieved using only lowdimensional representations.
Finally, we believe that the framework we developed here will be useful for proving the universality of other G -equivariant models for point cloud networks, and other related equivariant models.
We note that while the discussion in the paper was limited to point clouds in R 3 and to the action of
SO(3) , large parts of it are relevant to many other setups involving d -dimensional point clouds with symmetry groups of the form G = R d (cid:111)
H × S n on R d × n , where H can be any compact topological group.
A N OTATION
We introduce some notation for the proofs in the appendices.
We use the shortened notation ¯ X =
X − 1 n X 1 n 1 Tn and denote the columns of ¯ X by (¯ x 1 , . .
, ¯ x n ) .
We denote
[Equation]
B P ROOFS FOR S ECTIONB. G - EQUIVARIANT POLYNOMIALS ARE DENSE
A ﬁrst step in proving denseness of G -equivariance polynomials, and in the proof used in the next subsection is the following simple lemma, which shows that translation invariance can be dealt with simply by centralizing the point cloud.
In the following, ρ W T is some representation of SO(3) on a ﬁnite dimensional real vector space W T .
this induces an action ρ W T × S n of SO(3) × S n on W nT by
[Equation]
This is also the action of G which we consider, ρ G = ρ W T × S n , where we have invariance with respect to the translation coordinate.
The action of G on R 3 × n is deﬁned in equation 1.
Lemma B.1.
A function f : R 3 × n → W nT is G -equivariant, if and only if there exists a function h which is equivariant with respect to the action of SO(3) × S n on R 3 × n , and
[Equation]
Proof.
Recall that G -equivariance means SO(3) × S n equivariance and translation invariance.
Thus if f is G -equivariant then equation 16 holds with h
= f .
On the other hand, if f satisﬁes equation 16 then we claim it is G -equivariant.
Indeed, for all ( t, R, P ) ∈
R d (cid:111) SO(3) × S n , since P T 1 n 1 Tn = 1 n 1 Tn = 1 n 1 Tn P T ,
[Equation]
We now prove denseness of G -equivariant polynomials in the space of G -invariant continuous functions (Lemma 1).
Lemma 1.
Any continuous G -equivariant function in C G ( R 3 × n , W nT ) can be approximated uniformly on compact sets by G -equivariant polynomials in P G ( R 3 × n , W nT ) .
Proof of Lemma 1.
Let K ⊆ R 3 × n be a compact set.
We need to show that continuous Gequivariant functions can be approximated uniformly in K by G -equivariant polynomials.
Let Kdenote the compact set which is the image of K under the centralizing map
X
(cid:55)→ X − 1 n X 1 n 1 Tn .
By Lemma B.1, it is sufﬁcient to show that every SO(3) × S n equivariant continuous function f can be approximated uniformly on K 0 by a sequence of SO(3) × S n equivariant polynomials p k .
The argument is now concluded by the following general lemma:
Lemma B.2.
Let G be a compact group, Let ρ 1 and ρ 2 be continuous 2 representations of G on the
Euclidean spaces W 1 and W 2 .
Let K ⊆ W 1 be a compact set.
Then every equivariant function f : W 1 → W 2 can be approximated uniformly on K by a sequence of equivariant polynomials p k : W 1 (cid:55)→ W 2 .
Let µ be the Haar probability measure associated with the compact group G .
Let K 1 denote the compact set obtained as an image of the compact set G × K under the continuous mapping ( g, X ) (cid:55)→ ρ 1 ( g ) X.
Using the Stone-Weierstrass theorem, let p k be a sequence of (not necessarily equivariant) polynomials which approximate f uniformly on K 1 .
Every degree D polynomial p : W 1 → W 2 induces a
G -equivariant function
[Equation]
This function (cid:104) p (cid:105) is a degree D polynomial as well: This is because (cid:104) p (cid:105) can be approximated uniformly on K 1 by “Riemann Sums” of the form (cid:80) Nj =1 w j ρ
( g − 1 j ) p ( ρ 1 ( g j ) X ) which are degree D polynomials, and because degree D polynomials are closed in C ( K 1 ) .
Now for all X ∈ K 1 , continuity of the function g
(cid:55)→ ρ 2 ( g − 1 ) implies that the operator norm of ρ 2 ( g − 1 ) is bounded uniformly by some constant N > 0 , and so
[Equation]
B. P ROOF OF T HEOREM
Theorem 1.
If F feat is D -spanning and F pool is linearly universal, then there exists some C ( D ) ∈ N such that for all C ≥ C ( D ) the function space F C ( F feat , F pool ) contains all G -equivariant polynomials of degree ≤ D .
Proof.
By the D -spanning assumption, there exist f 1 , . . .
, f K ∈ F feat such that any vector valued polynomial p : R 3 × n → R n invariant to translations and equivariant to permutations is of the form p ( X [Equation] =
K (cid:88) k
= ˆΛ k
( f k ( X )
[Equation]
, (17) where Λ k are linear functions to R .
If p is a matrix valued polynomial mapping R 3 × n to
W nT =
R t × n , which is invariant to translations and equivariant to permutations, then it is of the form p
= ( p ij ) i ∈
[ t ] ,j ∈ [ n ] , and each p i =
( p ij ) j ∈ [ n ] is itself invariant to translations and permutation equivariant.
It follows that matrix valued p can also be written in the form equation 17, the only difference being that the image of the linear functions Λ k is now R t .
Now let p : R 3 × n → W nT be a G -equivariant polynomial of degree ≤ D .
It remains to show that we can choose Λ k to be SO(3) equivariant.
We do this by a symmetrization argument: denote the
Haar probability measure on SO(3) by ν , and the action of SO(3) on W feat and W T by ρ 1 and ρrespectively Denote p = ( p j ) nj =1 and f k = ( f jk ) nj =1 .
For every j = 1 , . . . , n , we use the SO(3) equivariance of p j and f jk to obtain
[Equation]
where ˜Λ k stands for the equivariant linear functional from W feat to W T , deﬁned for w ∈ W feat by
[Equation]
Thus we have shown that p is in F C ( F feat , F pool ) for C = K , as required.
C P ROOFS FOR S ECTION
We prove Lemma
Lemma 2.
For every D ∈ N + , the set Q D is D -spanning.
Proof.
It is known (Segol & Lipman, 2019)
(Theorem 2) that polynomials p : R 3 × n → R n which are S n -equivariant, are spanned by polynomials of the form p (cid:126)α = ( p j(cid:126)α ) nj =1 , deﬁned as
[Equation]
18) where (cid:126)α = ( α 1 , . .
. , α K ) and each α k ∈ N 3+ is a multi-index.
It follows that S n -equivariant polynomials of degree ≤ D are spanned by polynomials of the form p j(cid:126)α where (cid:80)
Kk =1 | α k | ≤ D .
Denoting r k
= | α k | , k = 1 , . . .
K , the sum of all r k by T , and (cid:126)r = ( r k )
Kk =1 , we see that there exists a linear functional Λ (cid:126)α,(cid:126)r : T T → R such that
[Equation]
where we recall that Q (cid:126)r = [Equation] Q
[Equation] 1 is deﬁned in equation 6 as
[Equation]
Thus polynomials p =
( p j ) nj =1 which are of degree ≤ D , and are S n equivariant, can be written as p
[Equation]
X )
[Equation]
[Equation]
where Λ (cid:126)r = (cid:80) (cid:126)α || α k
| = r k Λ
[Equation] [Equation] ◦ ι − 1 T , and ι − 1 T is the left inverse of the embedding ι .
If p is also translation invariant, then
[Equation]
Thus Q D is D -spanning.
We prove Lemma
Lemma 3.
The function set Q D is contained in
[Equation]
Thus F feat is D -spanning.
Proof.
In this proof we make the dependence of F feat on D explicit and denote F feat ( D ) .
We prove the claim by induction on D .
Assume D
= 0 .
Then Q 0 contains only the constant function
X (cid:55)→ 1 n ∈ T n 0 , and this is precisely the function π V ◦ ext ∈ F feat (0) .
Now assume the claim holds for all D (cid:48) with D − 1 ≥ D (cid:48) ≥ 0 , and prove the claim for D .
Choose (cid:126)r =
( r 1 , . .
, r k ) ∈ Σ T for some T ≤ D , we need to show that the function Q ( (cid:126)r ) is in F feat ( D ) .
Since F feat ( D − 1) ⊆
F feat ( D ) we know from the induction hypothesis that this is true if T < D .
Now assume T = D .
We consider two cases:
If r 1 > 0 , we set ˜ r
= ( r 1 − 1 , r 2 , . . .
, r K ) .
We know that ι ◦ Q (˜ r ) ( ¯ X ) ∈ F feat ( D − 1) by the induction hypothesis.
So there exist f 2 , . . .
, f D such that
[Equation]
Now choose f 1 ∈ F min to be the function whose V coordinate ˜ V
=
( ˜ V j ) nj =1 , is given by ˜ V j
( X, V ) =
x j ⊗ V j , obtained by setting θ 1 = 1 , θ 2 = 0 in equation 9.
Then , we have
[Equation]
[Equation]
and so
[Equation]
20) and ι ◦ Q
[Equation]
If r 1 = 0 .
We assume without loss of generality that r 2 > 0 .
Set ˜ r
= ( r 2 − 1 , r 3 , . . .
, r K ) .
As before by the induction hypothesis there exist f 2 , . . .
, f D which satisfy equation 19.
This time we choose f 1 ∈ F min to be the function whose V coordinate ˜ V
=
( ˜ V j ) nj =1 , is given by ˜ V j
( X, V ) = (cid:80) j x j ⊗ V j , obtained by setting θ 1 = 0 , θ 2 = 1 in equation 9.
Then we have
[Equation]
Thus equation 20 holds, and so again we have that ι ◦ Q ( (cid:126)r )
( X − 1 n X 1 n 1 Tn ) ∈
F feat ( D ) .
Finally we prove Lemma
Lemma 4.
If all functions in Q D can be written as
[Equation]
where f k ∈ F feat , A k : W feat → W T feat and ˆ A k : W n feat → ( W T feat ) n is deﬁned by elementwise application of A k , then F feat is D -spanning.
Proof.
If the conditions in Lemma 4 hold, then since Q D is D -spanning, every translation invariant and permutation equivariant polynomials p of degree D can be written as
[Equation]
where we denote Λ k,(cid:126)r = Λ (cid:126)r ◦ ι ◦ A k,(cid:126)r .
Thus we proved F feat is D -spanning.
D P ROOFS FOR S ECTION
We prove Lemma
Lemma 5.
Let l (1) =
( (cid:96) (1)1 , . . .
, (cid:96) (1) K 1 ) and l (2) =
( (cid:96) (2)1 , . . .
, (cid:96) (2) K 2 ) .
A function Λ = (Λ 1 , . . .
, Λ K 2 ) is a linear equivariant mapping between W l (1) and W l (2) , if and only if there exists a K 1 × K 2 matrix
M with M ij = 0 whenever (cid:96) (1) i (cid:54) = (cid:96) (2) j , such that
[Equation]
where V =
( V i ) K
i
=1 and V i ∈ W (cid:96) (1) i for all
i = 1 , . .
, K 1 .
Proof.
As mentioned in the main text, this lemma is based on Schur’s lemma.
This lemma is typically stated for complex representations, but holds for odd dimensional real representation as well.
We recount the lemma and its proof here for completeness (see also (Fulton & Harris, 2013)).
Lemma D.1 (Schur’s Lemma for SO(3) ) .
Let Λ : W (cid:96) 1 → W (cid:96) 2 be a linear equivariant map.
If (cid:96) 1 (cid:54) =
(cid:96) 2 then Λ = 0 .
Otherwise Λ is a scalar multiply of the identity.
Proof.
Let Λ : W (cid:96) 1 → W (cid:96) 2 be a linear equivariant map.
The image and kernel of Λ are invariant subspaces of W (cid:96) 1 and W (cid:96) 2 , respectively.
It follows that if Λ (cid:54) = 0 then Λ is a linear isomorphism so necessarily (cid:96)
= (cid:96) 2 .
Now assume (cid:96)
= (cid:96) 2 .
Since the dimension of W (cid:96) 1 is odd, Λ has a real eigenvalue λ .
The linear function Λ − λI is equivariant and has a non-trivial kernel, so Λ − λI = 0 .
We now return to the proof of Lemma 5.
Note that each Λ j : W l (1) → W (cid:96) (2) j is linear and SO(3) equivariant.
Next denote the restrictions of each Λ j to W (cid:96) (1) i , i = 1 , . .
, K 2 by Λ ij , and note that
[Equation]
[Equation]
By considering vectors in W l (1) of the form (0 , . . . , 0 , V i , 0 . . .
we see that each Λ ij : W (cid:96) (1) i →
W (cid:96) (2) j is linear and SO(3) -equivariant.
Thus by Schur’s lemma, if (cid:96) (1) i [Equation]
j then Λ ij [Equation] i )
[Equation] M ij [Equation] for some real M ij , and otherwise M ij = 0 .
Plugging this into equation 21 we obtain equation 12.
We prove Theorem 2 which shows that the TFN network described in the main text is universal:
Theorem 2.
For all n ∈ N , l T ∈ N ∗ + ,
For D ∈ N + , every G -equivariant polynomial p : R 3 × n → W nT of degree D is in F TFN C ( D ) ,D . 2.
Every continuous G -equivariant function can be approximated uniformly on compact sets by functions in ∪ D ∈ N + F TFN C ( D ) ,D
Proof.
As mentioned in the main text, we only need to show that the function space F feat ( D ) is
D -spanning.
Recall that F feat ( D ) is obtained by 2 D consecutive convolutions with D -ﬁlters.
In general, we denote the space of functions deﬁned by applying J consecutive convolutions by G J,D .
If Y is a space of functions from R 3 × n → Y n , we denote by (cid:104)Y , T T (cid:105) the space of all functions p : R 3 × n → T nT of the form
[Equation]
where A k : Y → T T are linear functions, ˆ A k : Y n → T nT are induced by elementwise application, and f k ∈ Y .
This notation is useful because: (i) by Lemma 4 it is sufﬁcient to show that Q ( (cid:126)r )
( ¯ X ) is in (cid:104)G 2 D,D , T T (cid:105) for all (cid:126)r ∈ Σ T and all T ≤ D , and because (ii) it enables comparison of the expressive power of function spaces Y 1 , Y 2 whose elements map to different spaces Y n 1 , Y n 2 , since the elements in (cid:104)Y i , T T (cid:105) , i = 1 , 2 both map to the same space.
In particular, note that if for every f ∈ Y 2 there is a g ∈ Y 1 and a linear map A : Y 1 → Y 2 such that f ( X )
=
ˆ A ◦ g
( X ) , then (cid:104)Y 2 , T T (cid:105) ⊆
(cid:104)Y 1 , T T (cid:105) .
We now use this abstract discussion to prove some useful results: the ﬁrst is that for the purpose of this lemma, we can ‘forget about’ the multiplication by a unitary matrix in equation 14, used for decomposition into irreducible representations: To see this, denote by ˜ G J,D the function space obtained by taking J consecutive convolutions with D -ﬁlters without multiplying by a unitary matrix in equation 14.
Since Kronecker products of unitary matrices are unitary matrices, we obtain that the elements in G J,D and ˜ G J,D differ only by multiplication by a unitary matrix, and
thus (cid:104) ˜
G J,D , T T (cid:105) ⊆
(cid:104)G J,D , T T (cid:105) and (cid:104)G J,D , T T (cid:105) ⊆
(cid:104)
˜
G J,D , T T (cid:105) , so both sets are equal.
Next, we prove that adding convolutional layers (enlarging J ) or taking higher order ﬁlters (enlarging D ) can only increase the expressive power of a network.
Lemma D.2.
For all J, D, T ∈ N + ,
(cid:104)G J,D , T T (cid:105) ⊆
(cid:104)G J +1 ,D , T T (cid:105) .
(cid:104)G J,D , T T (cid:105) ⊆
(cid:104)G J,D +1 , T T (cid:105) .
Proof.
The ﬁrst claim follows from the fact that every function f in (cid:104)G J,D , T T (cid:105) can be identiﬁed with a function in (cid:104)G J +1 ,D , T T (cid:105) by taking the J + 1 convolutional layer in equation 14 with θ
= 1 , F = 0 .
The second claim follows from the fact that D -ﬁlters can be identiﬁed with D + 1 -ﬁlters whose
D + 1 -th entry is 0 .
The last preliminary lemma we will need is
Lemma D.3.
For every J, D ∈ N + , and every t, s ∈ N + , if p ∈ (cid:104)G J,D , T t (cid:105) , then the function q deﬁned by
[Equation]
is in (cid:104)G J +1 ,D , T t + s (cid:105) .
Proof.
This lemma is based on the fact that the space of s homogeneous polynomial on Ris spanned by polynomials of the form (cid:107) x (cid:107) s − (cid:96) Y (cid:96) m ( x ) for (cid:96) =
s, s − 2 , s − 4 . . .
(Dai & Xu, 2013).
For each such (cid:96) , and s ≤ D , these polynomials can be realized by ﬁlters F ( (cid:96) ) by setting
R ( (cid:96) )
( (cid:107) x (cid:107) )
= (cid:107) x (cid:107) s
so that
[Equation]
For every D ∈ N and s ≤ D , we can construct a D -ﬁlter F s,D
=
( F (0) , . .
, F ( D ) )
where
F ( s ) , F ( s − 2) , . . .
are as deﬁned above and the other ﬁlters are zero.
Since both the entries of
F s,D ( x ) , and the entries of x ⊗ s , span the space of s -homogeneous polynomials on R 3 , it follows that there exists a linear mapping B s : W l D → T s so that
[Equation]
Thus, since p can be written as a sum of compositions of linear mappings with functions in G J,D as in equation 22, and similarly x ⊗ s is obtained as a linear image of functions in G 1 ,D as in equation 23, we deduce that
[Equation]
is in (cid:104)G + , T t + s (cid:105)
As a ﬁnal preliminary, we note that D -ﬁlters can perform an averaging operation by setting R (0) =
and θ 0 , R (1) , . . .
, R ( D ) =
in equation 13 and equation 14 .
We call this D -ﬁlter an averaging ﬁlter .
We are now ready to prove our claim: we need to show that for every D, T ∈ N + where T ≤ D , for every (cid:126)r ∈ Σ T , the function Q ( (cid:126)r ) is in (cid:104)G 2 D,D , T T (cid:105) .
Note that due to the inclusion relations in
Lemma D.2 it is sufﬁcient to prove this for the case T =
D .
We prove this by induction on D .
For
D = 0 , vectors (cid:126)r ∈ Σ 0 contains only zeros and so
[Equation]
We now assume the claim is true for all D (cid:48) with D > D (cid:48) ≥ 0 and prove the claim is true for D .
We need to show that for every (cid:126)r ∈ Σ D the function Q ( (cid:126)r ) is in (cid:104)G 2 D,D , T D (cid:105) .
We prove this yet again by induction, this time on the value of r 1 : assume that (cid:126)r ∈ Σ D and r 1 = 0 ..
Denote by ˜
r the vector in Σ D − 1 deﬁned by ˜ r
= ( r 2 − 1 , r 3 , . . .
, r K ) .
By the induction assumption on D , we know that Q (˜ r ) ( ¯ X ) ∈
G 2( D − 1) ,D − 1 ,D − 1 and so q a (
[Equation]
=
[Equation]
−
[Equation]
is in (cid:104)G 2 D − 1 ,D − 1 , T D (cid:105) by Lemma D.3, which is contained in (cid:104)G 2 D − 1 ,D , T D (cid:105) by Lemma D.2.
Since ¯ x a has zero mean, while Q ( (cid:126)r ) a ( ¯ X ) does not depend on a since r 1 = 0 , applying an averaging ﬁlter to q a gives us a constant value − Q ( (cid:126)r ) a ( ¯ X ) in each coordinate a ∈ [ n ] , and so Q ( (cid:126)r ) ( ¯ X ) is in (cid:104)G 2 D,D , T D (cid:105) .
Now assume the claim is true for all (cid:126)r ∈ Σ D which sum to D , and whose ﬁrst coordinate is smaller than some r (cid:48) 1 ≥ 1 , we now prove the claim is true when the ﬁrst coordinate of (cid:126)r is equal to r (cid:48) 1 .
The vector ˜ r = ( r 2 , . . .
, r K ) obtained from (cid:126)r by removing the ﬁrst coordinate, sums to D (cid:48)
=
D − r (cid:48) 1 <
D , and so by the induction hypothesis on D we know that Q (˜ r ) ∈ (cid:104)G 2 D (cid:48) ,D (cid:48) , T D (cid:48) (cid:105) .
By Lemma D. we obtain a function q a ∈ (cid:104)G 2 D (cid:48)
+1 ,D (cid:48) , T D (cid:105) ⊆
(cid:104)G 2 D,D , T D (cid:105) deﬁned by
[Equation]
where the additional terms are linear combinations of functions of the form P D Q ( r (cid:48) )
a ( ¯ X ) where r (cid:48) ∈ Σ D and their ﬁrst coordinate r 1 is smaller than r (cid:48) 1 , and P D : T D → T D is a permutation.
By the induction hypothesis on r 1 , each such Q ( r (cid:48) ) is in (cid:104)G 2 D,D , T D (cid:105) .
It follows that P D Q ( r (cid:48) ) a ( ¯ X ) , a = 1 , . . . , n , and thus Q ( (cid:126)r ) ( ¯ X ) , are in (cid:104)G 2 D,D , T D (cid:105) as well.
This concludes the proof of Theorem 2.
E A LTERNATIVE TFN ARCHITECTURE
In this appendix we show that replacing the standard TFN convolutional layer with the layer deﬁned in equation 15:
[Equation]
we can obtain D -spanning networks using 2 D consecutive convolutions with 1 -ﬁlters (that is, ﬁlters in W l 1 , where l 1 = [0 , 1] T ).
Our discussion here is somewhat informal, meant to provide the general ideas without delving into the details as we have done for the standard TFN architecture in the proof of Theorem 2.
In the end of our discussion we will explain what is necessary to make this argument completely rigorous.
We will only need two ﬁxed ﬁlters for our argument here: The ﬁrst is the 1 -ﬁlter F Id = ( F (0) , F (1) ) deﬁned by setting R (0) ( (cid:107) x (cid:107) )
= 0 and R (1) ( (cid:107)
x
(cid:107) )
= (cid:107) x (cid:107) to obtain
[Equation]
The second is the ﬁlter F 1 deﬁned by setting R (0) ( (cid:107) x (cid:107) )
= 1 and R (1) ( (cid:107) x
(cid:107) )
= 0 , so that
[Equation]
We prove our claim by showing that a pair of convolutions with 1 -ﬁlters can construct any convolutional layer deﬁned in equation 9 for the D -spanning architecture using tensor representations.
The claim then follows from the fact that D convolutions of the latter architecture sufﬁce for achieving
D -spanning, as shown in Lemma 3.
Convolutions for tensor representations, deﬁned in equation 9, are composed of two terms:
[Equation]
tensor ,
[Equation]
To obtain the ﬁrst term ˜ V tensor , 1 a , we set θ 1 = 0 , θ 2 = 1 /n, F = F Id in equation 15 we obtain (the decomposition into irreducibles of) ˜ V tensor , 1 a ( ¯ X, V ) =
¯ x a ⊗ V a .
Thus this term can in fact be expressed by a single convolution.
We can leave this outcome unchanged by a second convolution, deﬁned by setting θ 1 = 0 , θ 2 = 1 /n, F = F 1 .
To obtain the second term ˜ V tensor , 2 a , we apply a ﬁrst convolution with θ 1 = − 1 , F = F Id , θ 2 = 0 , to obtain n (cid:88)
[Equation]
By applying an additional averaging ﬁlter, deﬁned by setting θ 1 = 1 n , F = F 1 , θ 2 = 0 , we obtain ˜ V tensor , 2 a ( V, ¯ X ) .
This concludes our ‘informal proof’.
Our discussion here has been somewhat inaccurate, since in practice F Id ( x ) =
(0 , x ) ∈ W 0 ⊕ Wand F 1 ( x ) =
(1 , 0) ∈ W 0 ⊕ W 1 .
Moreover, in our proof we have glossed over the multiplication by the unitary matrix used to obtain decomposition into irreducible representations.
However the ideas discussed here can be used to show that 2 D convolutions with 1 -ﬁlters can satisfy the sufﬁcient condition for D -spanning deﬁned in Lemma 4.
See our treatment of Theorem 2 for more details.
F C OMPARISON WITH ORIGINAL TFN PAPER
In this Appendix we discuss three superﬁcial differences between the presentation of the TFN architecture in Thomas et al.
and our presentation here:
We deﬁne convolutional layers between features residing in direct sums of irreducible representations, while (Thomas et al., 2018) focuses on features which inhabit a single irreducible representation.
This difference is non-essential, as direct sums of irreducible representations can be represented as multiple channels where each feature inhabits a single irreducible representation.
The term θ 0 V a in equation 14 appears in (Fuchs et al., 2020), but does not appear explicitly in (Thomas et al., 2018).
However it can be obtained by concatenation of the input of a self-interaction layer to the output, and then applying a self-interaction layer.
We take the scalar functions R ( (cid:96) ) to be polynomials, while (Thomas et al., 2018) take them to be fully connected networks composed with radial basis functions.
Using polynomial scalar bases is convenient for our presentation here since it enables exact expression of
equivariant polynomials.
Replacing polynomial bases with fully connected networks, we obtain approximation of equivariant polynomials instead of exact expression.
It can be shown that if p is a G -equivariant polynomial which can be expressed by some network
F C,D deﬁned with ﬁlters coming from a polynomial scalar basis, then p can be approximated on a compact set K , up to an arbitrary (cid:15) error, by a similar network with scalar functions coming from a sufﬁciently large fully connected network.
G T ENSOR U NIVERSALITY
In this section we show how to construct the complete set F pool of linear SO(3) invariant functionals from W T feat =
(cid:76) DT =0 T T to R .
Since each such functional Λ is of the form
[Equation]
where each Λ T is SO(3) -invariant, it is sufﬁcient to characterize all linear SO(3) -invariant functionals Λ : T D → R .
It will be convenient to denote
[Equation]
We achieve our characterization using the bijective correspondence between linear functional Λ :
W ⊗ D → R and multi-linear functions ˜Λ : W D
→ R : each such Λ corresponds to a unique ˆΛ , such that
[Equation]
where e 1 , e 2 , e 3 denote the standard basis elements of R 3 .
We deﬁne a spanning set of equivariant linear functionals on W ⊗ D via a corresponding characterization for multi-linear functionals on W D .
Speciﬁcally, set
[Equation]
is even and non-negative. }
For k ∈ K D we deﬁne a multi-linear functional: ˜Λ
[Equation]
×
[Equation]
25) and for ( k, σ ) ∈ K D × S D we deﬁne
[Equation]
Proposition 1.
The space of linear invariant functions from T D to R is spanned by the set of linear invariant functionals λ D =
{ Λ k,σ | ( k, σ ) ∈ K D × S D } induced by the multi-linear functional ˜Λ k,σ described in equation 25 and equation
We note that (i) equation 24 provides a (cumbersome) way to compute all linear invariant functionals Λ k,σ explicitly by evaluating the corresponding ˜Λ k,σ on the 3 D elements of the standard basis and (ii) the set λ D is spanning, but is not linearly independent.
For example, since (cid:104) w 1 , w 2 (cid:105) =
(cid:104)
w 2 , w 1 (cid:105) , the space of SO(3) invariant functionals on T
= W ⊗ 2 is one dimensional while | λ 2 | = 2 .
Proof of Proposition 1.
We ﬁrst show that the bijective correspondence between linear functional Λ : W ⊗ D → R and multi-linear functions ˜Λ : W D → R , extends to a bijective correspondence between SO(3) -invariant linear/multi-linear functionals.
The action of SO(3) on W D is deﬁned by
[Equation]
The action ρ ( R ) =
R ⊗ D of SO(3) on W ⊗ D is such that the map
[Equation] [Equation]
w
[Equation]
w
[Equation]
w D is SO(3) - equivariant.
It follows that if ˜Λ and Λ satisfy equation 24, then for all R ∈ SO(3) , the same equation holds for the pair ˜Λ ◦ ˜ ρ
( R ) and Λ ◦ ρ ( R ) .
Thus SO(3) -invariance of ˜Λ is equivalent to SO(3) -invariance of Λ .
Multi-linear functionals on W D invariant to ˜ ρ are a subset of the set of polynomials on W D invariant to ˜ ρ .
It is known (see (Kraft & Procesi, 2000), page 114), that all such polynomials are algebraically generated by functions of the form det( w i 1 , w i 2 , w i 3 ) and (cid:104) w j 1 , w j 2 (cid:105) , where i 1 , i 2 , i 3 , j 1 , j 2 ∈
[ D ] .
Equivalently, SO(3) -invariant polynomials are spanned by linear combinations of polynomials of the form
[Equation]
When considering the subset of multi-linear invariant polynomials, we see that they must be spanned by polynomials as in equation 27, where each w 1 , . . .
, w D appears exactly once in each polynomial in the spanning set.
These precisely correspond to the functions in λ D .
H E XPERIMENTS
[Table]
This section provides an experimental evaluation of different design choices of the TFN architecture, inspired by our theoretical analysis.
We study the following questions:
The importance of non-linear activation.
Our proof shows that using non-linear activation functions is not necessary for proving universality.
Here, we empirically test the possibility of removing these layers.
The importance of high-dimensional irreducible representations.
Our theoretical analysis shows that in order to represent/approximate high degree polynomials, high-order representations should be used.
Here, we check whether using high-order representations has practical beneﬁts.
The effect of self-interaction layers.
Our proof suggests that it is enough to use self interaction linear layers at the end of the model.
We empirically compare this approach with the more common approach of using self-interaction layers after each convolutional layer.
Dataset.
We use the QM9 (Ramakrishnan et al., 2014) dataset for our experiments.
The dataset contains 134K molecules, with node 3D positions, 5 categorical node features and 4 categorical edge features.
The task is a molecule property prediction regression task.
Framework.
We used pytorch (Paszke et al.,
2017)as
the deep learning framework and the Deep
Graph Library (DGL) (Wang et al., 2019a) as the graph learning framework.
All experiments ran on
NVIDIA GV100 GPUs.
Experimental setup.
We use the the TFN implementation from Fuchs et al.
We trained each model variant for 50 epochs on the (cid:15) homo target variable using an (cid:96) 1 loss function and the
ADAM optimizer with learning rate 10 − 3 and report results on the test set on the ﬁnal epoch averaged over two runs.
We used the default parameters and data splits from Fuchs et al.
Architecture.
The architecture consists of 4 TFN convolutional layers, each followed by a linear self-interaction layer.
We used 16 copies of each irreducible representation used.
We used normbased non-linearity as in the original TFN paper (Thomas et al., 2018).
These convolutional layers are followed by a max-pooling layer and two fully connected layers with 16 d features in the hidden layer, where d is the maximal degree of irreducible representations used.
Results.
Table 1 and Figure 1 present the results.
The main conclusions are: (1) The experiments show that, at least for this task, using non-linear activations does not improve performance.
This result ﬁts our theoretical analysis which shows that these layers are not needed for universality.
(2) Figure 1 presents a plot of error vs representation degrees used.
The plot clearly shows that using high-dimensional representations (up to order 3) improves performance, which also ﬁts our analysis.
Using representation orders higher than 3 is signiﬁcantly more time consuming, and was found in Fuchs et al.
(2020) to have little effect on the results.
(3) Using self interaction layers only at the end of the model is shown to have marginal negative effect on the results.