Original: O N U NIVERSAL E QUIVARIANT S ET N ETWORKS
Revised: O N U NIVERSAL E QUIVARIANT S ET N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Using deep neural networks that are either invariant or equivariant to permutations in order to learn functions on unordered sets has become prevalent.
Revised: Using deep neural networks that are either invariant or equivariant to permutations in order to learn functions on unordered sets has become prevalent.

Original: The most popular, basic models are DeepSets (Zaheer et al., 2017) and PointNet (Qi et al., 2017).
Revised: The most popular, basic models are DeepSets (Zaheer et al., 2017) and PointNet (Qi et al., 2017).

Original: While known to be universal for approximating invariant functions, DeepSets and
Revised: While known to be universal for approximating invariant functions, DeepSets and

Original: PointNet are not known to be universal when approximating equivariant set functions.
Revised: PointNet are not known to be universal when approximating equivariant set functions.

Original: On the other hand, several recent equivariant set architectures have been proven equivariant universal (Sannai et al., 2019; Keriven & Peyr´e, 2019), however these models either use layers that are not permutation equivariant (in the standard sense) and/or use higher order tensor variables which are less practical.
Revised: On the other hand, several recent equivariant set architectures have been proven equivariant universal (Sannai et al., 2019; Keriven & Peyr´e, 2019), however these models either use layers that are not permutation equivariant (in the standard sense) and/or use higher order tensor variables which are less practical.

Original: There is, therefore, a gap in understanding the universality of popular equivariant set models versus theoretical ones.
Revised: There is, therefore, a gap in understanding the universality of popular equivariant set models versus theoretical ones.

Original: In this paper we close this gap by proving that:
Revised: In this paper we close this gap by proving that:

Original: (i) PointNet is not equivariant universal; and (ii) adding a single linear transmission layer makes PointNet universal.
Revised: (i) PointNet is not equivariant universal; and (ii) adding a single linear transmission layer makes PointNet universal.

Original: We call this architecture PointNetST and argue it is the simplest permutation equivariant universal model known to date.
Revised: We call this architecture PointNetST and argue it is the simplest permutation equivariant universal model known to date.

Original: Another consequence is that DeepSets is universal, and also PointNetSeg, a popular point cloud segmentation network (used e.g., in (Qi et al., 2017)) is universal.
Revised: Another consequence is that DeepSets is universal, and also PointNetSeg, a popular point cloud segmentation network (used e.g., in (Qi et al., 2017)) is universal.

Original: The key theoretical tool used to prove the above results is an explicit characterization of all permutation equivariant polynomial layers.
Revised: The key theoretical tool used to prove the above results is an explicit characterization of all permutation equivariant polynomial layers.

Original: Lastly, we provide numerical experiments validating the theoretical results and comparing different permutation equivariant models.
Revised: Lastly, we provide numerical experiments validating the theoretical results and comparing different permutation equivariant models.

Original:  INTRODUCTION
Revised:  I NTRODUCTION

Original: Many interesting tasks in machine learning can be described by functions F that take as input a set, X = ( x 1 , . . .
Revised: Many interesting tasks in machine learning can be described by functions F that take as input a set, X = ( x 1 , . . .

Original: , x n ) , and output some per-element features or values, F ( X ) =
Revised: , x n ) , and output some per-element features or values, F ( X ) =

Original: ( F ( X ) 1 , . .
Revised: ( F ( X ) 1 , . .

Original: , F ( X ) n ) .
Revised: , F ( X ) n ) .

Original: Permutation equivariance is the property required of F so it is welldeﬁned.
Revised: Permutation equivariance is the property required of F so it is welldeﬁned.

Original: Namely, it assures that reshufﬂing the elements in X and applying F results in the same output, reshufﬂed in the same manner.
Revised: Namely, it assures that reshufﬂing the elements in X and applying F results in the same output, reshufﬂed in the same manner.

Original: For example, if ˜ X
Revised: For example, if ˜ X

Original: =
Revised: =

Original: ( x 2 , x 1 , x 3 , . . .
Revised: ( x 2 , x 1 , x 3 , . . .

Original: , x n ) then
Revised: , x n ) then

Original: F ( ˜ X ) =
Revised: F ( ˜ X ) =

Original: ( F ( X ) 2 , F ( X ) 1 , F ( X ) 3 , . .
Revised: ( F ( X ) 2 , F ( X ) 1 , F ( X ) 3 , . .

Original: , F ( X ) n ) .
Revised: , F ( X ) n ) .

Original: Building neural networks that are permutation equivariant by construction proved extremely useful in practice where the most popular models, DeepSets Zaheer et al.
Revised: Building neural networks that are permutation equivariant by construction proved extremely useful in practice. Arguably the most popular models are DeepSets Zaheer et al.

Original: (2017) and PointNet Qi et al.
Revised: (2017) and PointNet Qi et al.

Original: (2017) enjoy small number of parameters, low memory footprint and computational efﬁciency along with high empirical expressiveness.
Revised: These models enjoy small number of parameters, low memory footprint and computational efﬁciency along with high empirical expressiveness.

Original: Although both DeepSets and PointNet are known to be invariant universal (i.e., can approximate arbitrary invariant continuous functions) they are not known to be equivariant universal (i.e., can approximate arbitrary equivariant continuous functions).
Revised: Although both DeepSets and PointNet are known to be invariant universal (i.e., can approximate arbitrary invariant continuous functions) they are not known to be equivariant universal (i.e., can approximate arbitrary equivariant continuous functions).

Original: On the other hand, several researchers have suggested theoretical permutation equivariant models and proved they are equivariant universal.
Revised: On the other hand, several researchers have suggested theoretical permutation equivariant models and proved they are equivariant universal.

Original: Sannai et al.
Revised: Sannai et al.

Original: (2019) builds a universal equivariant network by taking n copies of ( n − 1) -invariant networks and combines them with a layer that is not permutation invariant in the standard (above mentioned) sense.
Revised: (2019) builds a universal equivariant network by taking n copies of ( n − 1) -invariant networks and combines them with a layer that is not permutation invariant in the standard (above mentioned) sense.

Original: Keriven & Peyr´e (2019) solves a more general problem of building networks that are equivariant universal over arbitrary high order input tensors
Revised: Keriven & Peyr´e (2019) solves a more general problem of building networks that are equivariant universal over arbitrary high order input tensors

Original: R n d (including graphs); their construction, however, uses higher order tensors as hidden variables which is of less practical value.
Revised: R n d (including graphs); their construction, however, uses higher order tensors as hidden variables which is of less practical value.

Original: Yarotsky (2018) proves that neural networks constructed using a ﬁnite set of invariant and equivariant polynomial layers are also equivariant universal, however his network is not explicit (i.e., the polynomials are not characterized for the equivariant case) and also of less practical interest due to the high degree polynomial layers.
Revised: Yarotsky (2018) proves that neural networks constructed using a ﬁnite set of invariant and equivariant polynomial layers are also equivariant universal, however his network is not explicit (i.e., the polynomials are not characterized for the equivariant case) and also of less practical interest due to the high degree polynomial layers.

Original: In this paper we close the gap between the practical and theoretical permutation equivariant constructions and prove:
Revised: In this paper we close the gap between the practical and theoretical permutation equivariant constructions and prove:

Original: Theorem 1.
Revised: Theorem 1.

Original: (i) PointNet is not equivariant universal.
Revised: (i) PointNet is not equivariant universal.

Original: (ii) Adding a single linear transmission layer (i.e., X (cid:55)→ 11 T X ) to PointNet makes it equivariant universal.
Revised: (ii) Adding a single linear transmission layer (i.e., X (cid:55)→ 11 T X ) to PointNet makes it equivariant universal.

Original: (iii) Using ReLU activation the minimal width required for universal permutation equivariant network satisﬁes [Equation]
Revised: (iii) Using ReLU activation the minimal width required for universal permutation equivariant network satisﬁes ω ≤ k out + k in + (cid:0) n + k in k in (cid:1) .

Original: This theorem suggests that, arguably, PointNet with an addition of a single linear layer is the simplest universal equivariant network, able to learn arbitrary continuous equivariant functions of sets.
Revised: This theorem suggests that, arguably, PointNet with an addition of a single linear layer is the simplest universal equivariant network, able to learn arbitrary continuous equivariant functions of sets.

Original: An immediate corollary of this theorem is
Revised: An immediate corollary of this theorem is

Original: Corollary 1.
Revised: Corollary 1.

Original: DeepSets and PointNetSeg are universal.
Revised: DeepSets and PointNetSeg are universal.

Original: PointNetSeg is a network used often for point cloud segmentation (e.g., in Qi et al.
Revised: PointNetSeg is a network used often for point cloud segmentation (e.g., in Qi et al.

Original: One of the beneﬁt of our result is that it provides a simple characterization of universal equivariant architectures that can be used in the network design process to guarantee universality.
Revised: One of the beneﬁt of our result is that it provides a simple characterization of universal equivariant architectures that can be used in the network design process to guarantee universality.

Original: The theoretical tool used for the proof of Theorem 1 is an explicit characterization of the permutation equivariant polynomials over sets of vectors in R k using power-sum multi-symmetric polynomials.
Revised: The theoretical tool used for the proof of Theorem 1 is an explicit characterization of the permutation equivariant polynomials over sets of vectors in R k using power-sum multi-symmetric polynomials.

Original: We prove:
Revised: We prove:

Original: Theorem 2.
Revised: Theorem 2.

Original: Let P : R n × k
Revised: Let P : R n × k

Original: → R n × l be a permutation equivariant polynomial map.
Revised: → R n × l be a permutation equivariant polynomial map.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where b α =
Revised: where b α =

Original: ( x α 1 , . . .
Revised: ( x α 1 , . . .

Original: , x αn ) T , q α =
Revised: , x αn ) T , q α =

Original: ( q α, 1 , . . .
Revised: ( q α, 1 , . . .

Original: , q α,l ) T , where q α,j = q α,j ( s 1 , . . .
Revised: , q α,l ) T , where q α,j = q α,j ( s 1 , . . .

Original: , s t ) , t = (cid:0) n + kk (cid:1) , are invariant polynomials; s j ( X )
Revised: , s t ) , t = (cid:0) n + kk (cid:1) , are invariant polynomials; s j ( X )

Original: =
Revised: =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 x α j i are the power-sum multi-symmetric polynomials.
Revised: =1 x α j i are the power-sum multi-symmetric polynomials.

Original: On the other hand every polynomial map P satisfying Equation 1 is equivariant.
Revised: On the other hand every polynomial map P satisfying Equation 1 is equivariant.

Original: This theorem, which extends a result in Golubitsky & Stewart (2002) to sets of vectors using multivariate polynomials, lends itself to expressing arbitrary equivariant polynomials as a composition of entry-wise continuous functions and a single linear transmission, which in turn facilitates the proof of Theorem 1.
Revised: This theorem, which extends Proposition 2.27 in Golubitsky & Stewart (2002) to sets of vectors using multivariate polynomials, lends itself to expressing arbitrary equivariant polynomials as a composition of entry-wise continuous functions and a single linear transmission, which in turn facilitates the proof of Theorem 1.

Original: We conclude the paper by numerical experiments validating the theoretical results and testing several permutation equivariant networks for the tasks of set classiﬁcation and regression.
Revised: We conclude the paper by numerical experiments validating the theoretical results and testing several permutation equivariant networks for the tasks of set classiﬁcation and regression.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Equivariant maps.
Revised: Equivariant maps.

Original: Vectors x ∈ R k are by default column vectors; 0 , 1 are the all zero and all one vectors/tensors; e i is the i -th standard basis vector; I is the identity matrix; all dimensions are inferred from context or mentioned explicitly.
Revised: Vectors x ∈ R k are by default column vectors; 0 , 1 are the all zero and all one vectors/tensors; e i is the i -th standard basis vector; I is the identity matrix; all dimensions are inferred from context or mentioned explicitly.

Original: We represent a set of n vectors in R k as a matrix X ∈ R n × k and denote X
Revised: We represent a set of n vectors in R k as a matrix X ∈ R n × k and denote X

Original: =
Revised: =

Original: ( x 1 , x 2 , . . .
Revised: ( x 1 , x 2 , . . .

Original: , x n ) T , where x i ∈
Revised: , x n ) T , where x i ∈

Original: R
Revised: R

Original: k , i ∈ [ n ] , are the columns of X .
Revised: k , i ∈ [ n ] , are the columns of X .

Original: We denote by S n the permutation group of [ n ] ; its action on X is deﬁned by σ · X = ( x σ − 1 (1) , x σ − 1 (2) , . . .
Revised: We denote by S n the permutation group of [ n ] ; its action on X is deﬁned by σ · X = ( x σ − 1 (1) , x σ − 1 (2) , . . .

Original: , x σ − 1 ( n ) ) T , σ ∈ S n .
Revised: , x σ − 1 ( n ) ) T , σ ∈ S n .

Original: That is, σ is reshufﬂing the rows of X .
Revised: That is, σ is reshufﬂing the rows of X .

Original: The natural class of maps assigning a value or feature vector to every element in an input set is permutation equivariant maps:
Revised: The natural class of maps assigning a value or feature vector to every element in an input set is permutation equivariant maps:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A map F : R n × k
Revised: A map F : R n × k

Original: → R n × l satisfying F ( σ · X ) = σ · F ( X ) for all σ ∈ S n and
Revised: → R n × l satisfying F ( σ · X ) = σ · F ( X ) for all σ ∈ S n and

Original: X ∈ R n × d is called permutation equivariant.
Revised: X ∈ R n × d is called permutation equivariant.

Original: Power-sum multi-symmetric polynomials.
Revised: Power-sum multi-symmetric polynomials.

Original: For a vector x ∈ R k and a multi-index vector α = ( α 1 , . . .

Original: , α k ) ∈

Original: N k we deﬁne x α = x

Original: α 1 1 · · · x α k k , and | α | = (cid:80) i ∈

Original: [ k ] α i .

Original: Given a vector x = ( x 1 , . . .
Revised: Given a vector z = ( z 1 , . . .

Original: , x n ) ∈ R n the power-sum symmetric polynomials s j ( x ) = (cid:80) ni
Revised: , z n ) ∈ R n the powersum symmetric polynomials s j ( z ) = (cid:80) ni

Original: =1 x ji , with j ∈ [ n ] , uniquely characterize x up to permuting its entries.
Revised: =1 z ji , with j ∈ [ n ] , uniquely characterize z up to permuting its entries.

Original: In other words, for x , y ∈ R n we have y = σ · x for some σ ∈ S n if and only if s j ( y ) =
Revised: In other words, for z , y ∈ R n we have y = σ · z for some σ ∈ S n if and only if s j ( y ) =

Original: s j ( x ) for all j ∈ [ n ] .
Revised: s j ( z ) for all j ∈ [ n ] .

Original: An equivalent property is that every S n invariant polynomial p can be expressed as a polynomial in the power-sum symmetric polynomials, i.e., p ( x ) =
Revised: An equivalent property is that every S n invariant polynomial p can be expressed as a polynomial in the power-sum symmetric polynomials, i.e., p ( z ) =

Original: q ( s 1 ( x ) , . . .
Revised: q ( s 1 ( z ) , . . .

Original: , s n ( x )) , see Rydh (2007)
Revised: , s n ( z )) , see Rydh (2007)

Original: Corollary 8.4, Briand (2004) Theorem 3.
Revised: Corollary 8.4, Briand (2004) Theorem 3.

Revised: This fact was previously used in Zaheer et al.

Revised: (2017) to prove that DeepSets is universal for invariant functions

Revised: We extend this result to equivariant functions and the multi-feature (sets of vectors) case.

Revised: For a vector x ∈ R k and a multi-index vector α = ( α 1 , . . .

Revised: , α k ) ∈

Original:  A generalization of the power-sum symmetric polynomials to matrices exists and is called power-sum multi-symmetric polynomials, deﬁned with a bit of notation abuse: s α ( X ) =
Revised: N k we deﬁne x α = x α 1 1 · · · x α k k , and | α | = (cid:80) i ∈ [ k ] α i . A generalization of the power-sum symmetric polynomials to matrices exists and is called power-sum multi-symmetric polynomials, deﬁned with a bit of notation abuse: s α ( X ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 x αi , where α ∈ N k is a multi-index satisfying | α | ≤ n .
Revised: =1 x αi , where α ∈ N k is a multi-index satisfying | α | ≤ n .

Original: Note that the number of power-sum multi-symmetric polynomials acting on X ∈ R n × k is t = (cid:0) n + kk (cid:1) .
Revised: Note that the number of power-sum multi-symmetric polynomials acting on X ∈ R n × k is t = (cid:0) n + kk (cid:1) .

Original: For notation simplicity let α 1 , . . .
Revised: For notation simplicity let α 1 , . . .

Original: , α t be a list of all α ∈ N k with | α | ≤ n .
Revised: , α t be a list of all α ∈ N k with | α | ≤ n .

Original: Then we index the collection of power-sum multi-symmetric polynomials as s 1 , . . .
Revised: Then we index the collection of power-sum multi-symmetric polynomials as s 1 , . . .

Original: , s t .
Revised: , s t .

Original: Similarly to the vector case the numbers s j ( X ) , j ∈ [ t ] characterize X up to permutation of its rows.
Revised: Similarly to the vector case the numbers s j ( X ) , j ∈ [ t ] characterize X up to permutation of its rows.

Original: That is Y = σ · X for some σ ∈ S n iff s j ( Y ) =
Revised: That is Y = σ · X for some σ ∈ S n iff s j ( Y ) =

Original: s j ( Y ) for all j ∈ [ t ] .
Revised: s j ( Y ) for all j ∈ [ t ] .

Original: Furthermore, every S n invariant polynomial p : R n × k
Revised: Furthermore, every S n invariant polynomial p : R n × k

Original: → R can be expressed as a polynomial in the power-sum multi-symmetric polynomials (see (Rydh, 2007) corollary 8.4), i.e., 
Revised: → R can be expressed as a polynomial in the power-sum multi-symmetric polynomials (see (Rydh, 2007) corollary 8.4), i.e., [Equation]

Original: [Equation]
Revised: [Equation]

Original: These polynomials were recently used to encode multi-sets in Maron et al.
Revised: These polynomials were recently used to encode multi-sets in Maron et al.

Original:  E QUIVARIANT MULTI - SYMMETRIC POLYNOMIAL LAYERS
Revised:  E QUIVARIANT MULTI - SYMMETRIC POLYNOMIAL LAYERS

Original: In this section we develop the main theoretical tool of this paper, namely, a characterization of all permutation equivariant polynomial layers.
Revised: In this section we develop the main theoretical tool of this paper, namely, a characterization of all permutation equivariant polynomial layers.

Original: As far as we know, these layers were not fully characterized before.
Revised: As far as we know, these layers were not fully characterized before.

Original: Theorem 2 provides an explicit representation of arbitrary permutation equivariant polynomial maps
Revised: Theorem 2 provides an explicit representation of arbitrary permutation equivariant polynomial maps

Original: Let P : R n × k → R n × l using the basis of power-sum multi-symmetric polynomials, s i ( X ) .
Revised: P : R n × k → R n × l using the basis of power-sum multi-symmetric polynomials, s i ( X ) .

Original: The particular use of power-sum polynomials s i ( X ) has the advantage it can be encoded efﬁciently using a neural network: as we will show s i ( X ) can be approximated using a PointNet with a single linear transmission layer.
Revised: The particular use of power-sum polynomials s i ( X ) has the advantage it can be encoded efﬁciently using a neural network: as we will show s i ( X ) can be approximated using a PointNet with a single linear transmission layer.

Original: This allows approximating an arbitrary equivariant polynomial map using PointNet with a single linear transmission layer.
Revised: This allows approximating an arbitrary equivariant polynomial map using PointNet with a single linear transmission layer.

Original: In contrast, Yarotsky (2018) also provides a polynomial characterization of equivariant maps (see Lemma 2.1 and Proposition 2.4 in (Yarotsky, 2018)) however there is no formula given to the invariant/equivariant generating set of polynomials and their efﬁcient implementation/approximation is therefore questionable.

Original: A version of this theorem for vectors instead of matrices (i.e., the case of k = 1 ) appears  in Golubitsky & Stewart (2002); we extend their proof to matrices, which is the relevant scenario for ML applications as it allow working with sets of vectors. 
Revised: A version of this theorem for vectors instead of matrices (i.e., the case of k = 1 ) appears as Proposition 2.27 in Golubitsky & Stewart (2002); we extend their proof to matrices, which is the relevant scenario for ML applications as it allows working with sets of vectors. For k = 1 Theorem 2 reduces to the following form: p ( x ) i = (cid:80) a ≤ n p a ( s 1 ( x ) , . . . , s n ( x )) x ai with s j ( x ) = (cid:80) i x ji .

Revised: For matrices the monomial x ki is replaced by x αi for a multi-index α and the power-sum symmetric polynomials are replaced by the power-sum multi-symmetric polynomials.

Original: First, note that it is enough to prove Theorem 1 for l = 1 and apply it to every column of P .
Revised: First, note that it is enough to prove Theorem 1 for l = 1 and apply it to every column of P .

Original: Hence, we deal with a vector of polynomials p : R n × k
Revised: Hence, we deal with a vector of polynomials p : R n × k

Original: → R n and need to prove it can be expressed as p = (cid:80) | α |≤ n b α q α , for S n invariant polynomial q α .
Revised: → R n and need to prove it can be expressed as p = (cid:80) | α |≤ n b α q α , for S n invariant polynomial q α .

Original: Given a polynomial p ( X ) and the cyclic permutation σ − 1 = (123 · · · n ) the following operation, taking a polynomial to a vector of polynomials, is useful in characterizing equivariant polynomial maps:
Revised: Given a polynomial p ( X ) and the cyclic permutation σ − 1 = (123 · · · n ) the following operation, taking a polynomial to a vector of polynomials, is useful in characterizing equivariant polynomial maps:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 will be proved using the following two lemmas:
Revised: Theorem 2 will be proved using the following two lemmas:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let p : R n × k
Revised: Let p : R n × k

Original: → R n be an equivariant polynomial map.
Revised: → R n be an equivariant polynomial map.

Original: Then, there exists a polynomial p : R n × k
Revised: Then, there exists a polynomial p : R n × k

Original: → R , invariant to S n − 1 (permuting the last n − 1 rows of X ) so that p = (cid:100) p (cid:101) .
Revised: → R , invariant to S n − 1 (permuting the last n − 1 rows of X ) so that p = (cid:100) p (cid:101) .

Original: Proof.
Revised: Proof.

Original: Equivariance of p means that for all σ ∈ S n it holds that 
Revised: Equivariance of p means that for all σ ∈ S n it holds that σ · p ( X )

Original: [Equation]
Revised: = p ( σ · X ) σ · p ( X ) = p ( σ · X ) .

Original: Choosing an arbitrary permutation σ ∈ stab(1) <
Revised: Choosing an arbitrary permutation σ ∈ stab(1) <

Original: S n , namely a permutation satisfying σ (1) = 1 , and observing the ﬁrst row in Equation 4 we get p 1 ( X ) =
Revised: S n , namely a permutation satisfying σ (1) = 1 , and observing the ﬁrst row in Equation 4 we get p 1 ( X ) =

Original: p 1 ( σ · X )
Revised: p 1 ( σ · X )

Original: =
Revised: =

Original: p 1 ( x 1 , x σ − 1 (2) , . .
Revised: p 1 ( x 1 , x σ − 1 (2) , . .

Original: , x σ − 1 ( n ) ) .
Revised: , x σ − 1 ( n ) ) .

Original: Since this is true for all
Revised: Since this is true for all

Original: σ ∈ stab(1) p 1 is S n − 1 invariant.
Revised: σ ∈ stab(1) p 1 is S n − 1 invariant.

Original: Next, applying σ = (1 i ) to Equationand observing the ﬁrst row again we get p i ( X )
Revised: Next, applying σ = (1 i ) to Equationand observing the ﬁrst row again we get p i ( X )

Original: =
Revised: =

Original: p 1 ( x i , . .
Revised: p 1 ( x i , . .

Original: , x 1 , . . . ) .
Revised: , x 1 , . . . ) .

Original: Using the invariance of p 1 to
Revised: Using the invariance of p 1 to

Original: S n − 1 we get p =
Revised: S n − 1 we get p =

Original: (cid:100) p 1 (cid:101) .
Revised: (cid:100) p 1 (cid:101) .

Original: Lemma 2.
Revised: Lemma 2.

Original: Let p : R n × k
Revised: Let p : R n × k

Original: → R be a polynomial invariant to S n − 1 (permuting the last n − 1 rows of
Revised: → R be a polynomial invariant to S n − 1 (permuting the last n − 1 rows of

Original: X ) then
Revised: X ) then

Original: [Equation]
Revised: [Equation]

Original: where q α are S n invariant.
Revised: where q α are S n invariant.

Original: Proof.
Revised: Proof.

Original: Expanding p with respect to x 1 we get
Revised: Expanding p with respect to x 1 we get

Original: [Equation]
Revised: [Equation]

Original: for some m ∈ N .
Revised: for some m ∈ N .

Original: We ﬁrst claim p α are S n − 1 invariant.
Revised: We ﬁrst claim p α are S n − 1 invariant.

Original: Indeed, note that if p ( X ) =
Revised: Indeed, note that if p ( X ) =

Original: p ( x 1 , x 2 , . . .
Revised: p ( x 1 , x 2 , . . .

Original: , x n ) is S n − 1 invariant, i.e., invariant to permutations of x 2 , . . .
Revised: , x n ) is S n − 1 invariant, i.e., invariant to permutations of x 2 , . . .

Original: , x n , then also its derivatives ∂ | β | ∂ x β 1 p ( X ) are S n − 1 permutation invariant, for all β ∈ N k .
Revised: , x n , then also its derivatives ∂ | β | ∂ x β 1 p ( X ) are S n − 1 permutation invariant, for all β ∈ N k .

Original: Taking the derivative ∂ | β | /∂ x β 1 on both sides of Equation 6 we get that p β is S n − 1 equivariant.
Revised: Taking the derivative ∂ | β | /∂ x β 1 on both sides of Equation 6 we get that p β is S n − 1 equivariant.

Original: For brevity denote p = p α .
Revised: For brevity denote p = p α .

Original: Since p is S n − 1 invariant it can be expressed as a polynomial in the powersum multi-symmetric polynomials, i.e., p ( x 2 , . . .
Revised: Since p is S n − 1 invariant it can be expressed as a polynomial in the powersum multi-symmetric polynomials, i.e., p ( x 2 , . . .

Original: , x n )
Revised: , x n )

Original: =
Revised: =

Original: r ( s 1 ( x 2 , . . .
Revised: r ( s 1 ( x 2 , . . .

Original: , x n ) , . . .
Revised: , x n ) , . . .

Original: , s t ( x 2 , . . .
Revised: , s t ( x 2 , . . .

Original: , x n )) .
Revised: , x n )) .

Original: Note that s i ( x 2 , . . .
Revised: Note that s i ( x 2 , . . .

Original: , x n )
Revised: , x n )

Original: =
Revised: =

Original: s i ( X ) − x
Revised: s i ( X ) − x

Original: α
Revised: α

Original: i 1 and therefore
Revised: i 1 and therefore

Original: [Equation]
Revised: [Equation]

Original: Since r is a polynomial, expanding its monomials in s i ( X ) and x α 1 shows p can be expressed as p = (cid:80) | α |≤ m (cid:48)
Revised: Since r is a polynomial, expanding its monomials in s i ( X ) and x α 1 shows p can be expressed as p = (cid:80) | α |≤ m (cid:48)

Original: x
Revised: x

Original: α 1 ˜ p
Revised: α 1 ˜ p

Original: α , where m (cid:48) ∈ N , and ˜ p α are S n invariant (as multiplication of invariant S n polynomials s i ( X ) ).
Revised: α , where m (cid:48) ∈ N , and ˜ p α are S n invariant (as multiplication of invariant S n polynomials s i ( X ) ).

Original: Plugging this in Equation 6 we get Equation 5, possibly with the sum over some n’ > n .
Revised: Plugging this in Equation 6 we get Equation 5, possibly with the sum over some n’ > n .

Original: It remains to show n (cid:48) can be taken to be at-most n .
Revised: It remains to show n (cid:48) can be taken to be at-most n .

Original: This is proved in Corollary 5 in
Revised: This is proved in Corollary 5 in

Original: Briand (2004)
Revised: Briand (2004)

Original: Proof.
Revised: Proof.

Original: ( Theorem 2 ) Given an equivariant p as above, use Lemma 1 to write p = (cid:100) p (cid:101) where p ( X ) is invariant to permuting the last n − 1 rows of X .
Revised: ( Theorem 2 ) Given an equivariant p as above, use Lemma 1 to write p = (cid:100) p (cid:101) where p ( X ) is invariant to permuting the last n − 1 rows of X .

Original: Use Lemma 2 to write p ( X ) =
Revised: Use Lemma 2 to write p ( X ) =

Original: (cid:80)
Revised: (cid:80)

Original: | α |≤ n x α 1 q α ( X ) , where q α are S n invariant.
Revised: | α |≤ n x α 1 q α ( X ) , where q α are S n invariant.

Original: We get,
Revised: We get,

Original: [Equation]
Revised: [Equation]

Original: The converse direction is immediate after noting that b α are equivariant and q α are invariant.
Revised: The converse direction is immediate after noting that b α are equivariant and q α are invariant.

Original:  U NIVERSALITY OF SET EQUIVARIANT NEURAL NETWORKS
Revised:  U NIVERSALITY OF SET EQUIVARIANT NEURAL NETWORKS

Original: We consider equivariant deep neural networks f : R n × k in → R n × k out ,
Revised: We consider equivariant deep neural networks f : R n × k in → R n × k out ,

Original: [Equation]
Revised: [Equation]

Original: (7) where L i : R n × k i → R n × k i +1 are afﬁne equivariant transformations, and ν is an entry-wise nonlinearity (e.g., ReLU).
Revised: (7) where L i : R n × k i → R n × k i +1 are afﬁne equivariant transformations, and ν is an entry-wise nonlinearity (e.g., ReLU).

Original: We deﬁne the width of the network to be ω = max i k i ; note that this deﬁnition is different from the one used for standard MLP where the width would be nω , see e.g., Hanin &
Revised: We deﬁne the width of the network to be ω = max i k i ; note that this deﬁnition is different from the one used for standard MLP where the width would be nω , see e.g., Hanin &

Original: Sellke (2017).
Revised: Sellke (2017).

Original: Zaheer et al.
Revised: Zaheer et al.

Original: (2017) proved that afﬁne equivariant L i are of the form
Revised: (2017) proved that afﬁne equivariant L i are of the form

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where A , B ∈ R k i × k i +1 , and c ∈ R k i +1 are the layer’s trainable parameters; we call the linear transformation X (cid:55)→ 1 n 11 T XB a linear transmission layer.
Revised: where A , B ∈ R k i × k i +1 , and c ∈ R k i +1 are the layer’s trainable parameters; we call the linear transformation X (cid:55)→ 1 n 11 T XB a linear transmission layer.

Revised: We now deﬁne the equivariant models considered in this paper:

Original:  Equation 7 with the choice of layers as in Equation 8 is the DeepSets architecture (Zaheer et al., 2017).
Revised: The DeepSets (Zaheer et al., 2017) architecture is Equation 7 with the choice of layers as in Equation 8.

Original: Taking B = 0 in all layers is the PointNet architecture (Qi et al., 2017).
Revised: Taking B = 0 in all layers is the PointNet architecture (Qi et al., 2017).

Original: Another type of architecture of interest is PointNetSeg appearing in (Qi et al., 2017). Variations of this architecture are used for Point cloud segmentation, see e,g, Li et al.
Revised: PointNetST is an equivariant model of the form Equation 7 with layers as in Equation 8 where only a single layer L

Original: (2018) and Engelmann et al.
Revised: i has a non-zero

Original: PointNsetSeg uses an invariant version of PointNet (i.e., PointNet composed with an invariant max layer) and concatenates its output as a constant feature to an intermediate layer which is then inputed to another PointNet.
Revised: B .

Original: Lastly, we will also consider a model we call PointNetST that is PointNet  with a Single (linear) Transmission layer; in more details,
Revised: The PointNetSeg (Qi et al., 2017) architecture is PointNet composed with an invariant max layer, namely max ( F ( X ))

Original: PointNetST is an equivariant model of the form Equation 7 with layer as in Equation 8 where only a single layer L i has a non-zero B (see Equation 8).
Revised: j = max i ∈ [ n ] F ( X ) i,j and then concatenating it with the input X , i.e., [ X , 1 max ( F ( X ))] , and feeding is as input to another PointNet G , that is G ([ X , 1 max ( F ( X ))]) .

Original: We will prove PointNetST is permutation equivariant universal and therefore arguably the simplest permutation equivariant universal model known to date.
Revised: We will prove PointNetST is permutation equivariant universal and therefore arguably the simplest permutation equivariant universal model known to date.

Original: Universality of equivariant deep networks is deﬁned next.
Revised: Universality of equivariant deep networks is deﬁned next.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Permutation equivariant universality 1 of a model F : R n × k in → R n × k out means that for every permutation equivariant continuous function H : R n × k in → R n × k out deﬁned over the cube K =
Revised: Permutation equivariant universality 1 of a model F : R n × k in → R n × k out means that for every permutation equivariant continuous function H : R n × k in → R n × k out deﬁned over the cube K =

Original: [0 , 1] n × k in ⊂ R n × k in , and (cid:15) > 0 there exists a choice of m (i.e., network depth), k i (i.e., network width) and the trainable parameters of F so that (cid:107) H ( X ) − F
Revised: [0 , 1] n × k in ⊂ R n × k in , and (cid:15) > 0 there exists a choice of m (i.e., network depth), k i (i.e., network width) and the trainable parameters of F so that (cid:107) H ( X ) − F

Original: ( X ) (cid:107) ∞ <
Revised: ( X ) (cid:107) ∞ <

Original: (cid:15) for all X ∈ K .
Revised: (cid:15) for all X ∈ K .

Original: We prove our main theorem next.

Original: Proof.
Revised: Proof.

Original: (Theorem 1) Fact (i) , namely that PointNet is not equivariant universal is a consequence of the following simple lemma:
Revised: (Theorem 1) Fact (i) , namely that PointNet is not equivariant universal is a consequence of the following simple lemma:

Original: Lemma 3.
Revised: Lemma 3.

Original: Let h
Revised: Let h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: . , h n ) T : R n → R n be the equivariant linear function deﬁned by h ( x ) = 11 T x .
Revised: . , h n ) T : R n → R n be the equivariant linear function deﬁned by h ( x ) = 11 T x .

Original: There is no f : R → R so that | h i ( x ) − f ( x i ) | < 12 for all i ∈ [ n ] and x ∈ [0 , 1] n .
Revised: There is no f : R → R so that | h i ( x ) − f ( x i ) | < 12 for all i ∈ [ n ] and x ∈ [0 , 1] n .

Original: Proof.
Revised: Proof.

Original: Assume such f exists.
Revised: Assume such f exists.

Original: Let e 1 = (1 , 0 , . .
Revised: Let e 1 = (1 , 0 , . .

Original: . , 0) T ∈ R n .
Revised: . , 0) T ∈ R n .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: reaching a contradiction.
Revised: reaching a contradiction.

Original: To prove (ii) we ﬁrst reduce the problem from the class of all continuous equivariant functions to the class of equivariant polynomials.
Revised: To prove (ii) we ﬁrst reduce the problem from the class of all continuous equivariant functions to the class of equivariant polynomials.

Original: This is justiﬁed by the following lemma.
Revised: This is justiﬁed by the following lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: Equivariant polynomials P : R n × k in → R n × k out are dense in the space of continuous equivariant functions F : R n × k in → R n × k out over the cube K .
Revised: Equivariant polynomials P : R n × k in → R n × k out are dense in the space of continuous equivariant functions F : R n × k in → R n × k out over the cube K .

Original: Proof.
Revised: Proof.

Original: Take an arbitrary (cid:15) > 0 .
Revised: Take an arbitrary (cid:15) > 0 .

Original: Consider the function f ij : R n × k in → R , which denotes the ( i, j ) -th output entry of F .
Revised: Consider the function f ij : R n × k in → R , which denotes the ( i, j ) -th output entry of F .

Original: By the Stone-Weierstrass Theorem there exists a polynomial p ij : R n × k in → R such that (cid:107) f ij ( X ) − p ij ( X ) (cid:107) ∞ ≤ (cid:15) for all X ∈ K .
Revised: By the Stone-Weierstrass Theorem there exists a polynomial p ij : R n × k in → R such that (cid:107) f ij ( X ) − p ij ( X ) (cid:107) ∞ ≤ (cid:15) for all X ∈ K .

Original: Consider the polynomial map P : R n × k in →
Revised: Consider the polynomial map P : R n × k in →

Original: R n × k out deﬁned by ( P ) ij = p ij .
Revised: R n × k out deﬁned by ( P ) ij = p ij .

Original: P is in general not equivariant.
Revised: P is in general not equivariant.

Original: To ﬁnish the proof we will symmetrize P :
Revised: To ﬁnish the proof we will symmetrize P :

Revised: (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) F

Original: [Equation]
Revised: [Equation]

Revised: X ) −

Revised: [Equation]

Revised: σ ∈ S

Revised: [Equation]

Revised: [Equation]

Revised: ) (cid:13)(cid:13)(cid:13)

Original:  where in the ﬁrst equality we used the fact that F is equivariant.
Revised: [Equation] ∞ [Equation] where in the ﬁrst equality we used the fact that F is equivariant.

Original: This concludes the proof since (cid:80) σ ∈ S n σ · P ( σ − 1 · X ) is an equivariant polynomial map.
Revised: This concludes the proof since (cid:80) [Equation] P [Equation]

Original:  Now, according to Theorem 2 an arbitrary equivariant polynomial P : R n × k in → R n × k out can be written as P = (cid:80) | α |≤ n b α ( X ) q α ( X ) T , where b α ( X ) =
Revised: [Equation] ) is an equivariant polynomial map. Now, according to Theorem 2 an arbitrary equivariant polynomial P : R n × k in → R n × k out can be written as P = (cid:80) | α |≤ n b α ( X ) q α ( X ) T , where b α ( X ) =

Original: (cid:100) x
Revised: (cid:100) x

Original: α 1 (cid:101) ∈
Revised: α 1 (cid:101) ∈

Original: R n and q α =
Revised: R n and q α =

Original: ( q α, 1 , . . .
Revised: ( q α, 1 , . . .

Original: , q α,k out ) ∈
Revised: , q α,k out ) ∈

Original: R k out are invariant polynomials.
Revised: R k out are invariant polynomials.

Original: Remember that every S n invariant polynomial can be expressed as a polynomial in the t = (cid:0) n + k in k in (cid:1) power-sum multi-symmetric polynomials s j ( X ) = 1 n (cid:80) ni
Revised: Remember that every S n invariant polynomial can be expressed as a polynomial in the t = (cid:0) n + k in k in (cid:1) power-sum multi-symmetric polynomials s j ( X ) = 1 n (cid:80) ni

Original: =1 x α j i , j ∈ [ t ] (we use the 1 /n normalized version for a bit more simplicity later on).
Revised: =1 x α j i , j ∈ [ t ] (we use the 1 /n normalized version for a bit more simplicity later on).

Original: We can therefore write P as composition of three maps:
Revised: We can therefore write P as composition of three maps:

Original: [Equation]
Revised: [Equation]

Original: where B : R n × k in → R n × t is deﬁned by
Revised: where B : R n × k in → R n × t is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: b ( x ) =
Revised: b ( x ) =

Original: ( x α 1 , . . .
Revised: ( x [Equation]

Original: , x α t ) ; L is deﬁned as in Equation 8 with B = [ 0 , I ] and A =
Revised: , x α t ) ; L is deﬁned as in Equation 8 with B = [ 0 , I ] and A =

Original: [ e 1 , . .
Revised: [ e 1 , . .

Original: , e k in , 0 ] , where I ∈ R t × t the identity matrix and e
Revised: , e k in , 0 ] , where I ∈ R t × t the identity matrix and e

Original: i ∈
Revised: i ∈

Original: R t represents the standard basis (as-usual).
Revised: R t represents the standard basis (as-usual).

Original: We assume α j = e j ∈ R k in , for j ∈ [ k in ] .
Revised: We assume α j = e j ∈ R k in , for j ∈ [ k in ] .

Original: Note that the output of L is of the form
Revised: Note that the output of L is of the form

Original: [Equation]
Revised: [Equation]

Original:  Finally, Q : R n × ( k in + t ) → R n × k out is deﬁned by
Revised: X [Equation] Finally, Q : R n × ( k in + t ) → R n × k out is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The decomposition in Equation 9 of P suggests that replacing Q , B with Multi-Layer Perceptrons
Revised: x

Original: [Equation]
Revised: [Equation]

Original: MLPs) would lead to a universal permutation equivariant network consisting of PointNet with a single linear transmission layer, namely PointNetST.
Revised: The decomposition in Equation 9 of P suggests that replacing Q , B with Multi-Layer Perceptrons (MLPs) would lead to a universal permutation equivariant network consisting of PointNet with a single linear transmission layer, namely PointNetST.

Original: The F approximating P will be deﬁned as
Revised: The F approximating P will be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Φ : R n × k in → R n × t and Ψ : R n × ( t + k in ) →
Revised: where Φ : R n × k in → R n × t and Ψ : R n × ( t + k in ) →

Original: R n × k out are both of PointNet architecture, namely there exist MLPs φ : R k in → R t and ψ : R t + k in →
Revised: R n × k out are both of PointNet architecture, namely there exist MLPs φ : R k in → R t and ψ : R t + k in →

Original: R k out so that Φ ( X ) =
Revised: R k out so that Φ ( X ) =

Original: ( φ ( x 1 ) , . . .
Revised: ( φ ( x 1 ) , . . .

Original: , φ ( x n )) T and Ψ ( X ) =
Revised: , φ ( x n )) T and Ψ ( X ) =

Original: ( ψ ( x 1 ) , . . .
Revised: ( ψ ( x 1 ) , . . .

Original: , ψ ( x n )) T .
Revised: , ψ ( x n )) T .

Original: See Figurefor an illustration of F .
Revised: See Figurefor an illustration of F .

Original: To build the MLPs φ , ψ we will ﬁrst construct ψ to approximate q , that is, we use the universality of
Revised: To build the MLPs φ , ψ we will ﬁrst construct ψ to approximate q , that is, we use the universality of

Original: MLPS (see (Hornik, 1991; Sonoda & Murata, 2017; Hanin & Sellke, 2017)) to construct ψ so that (cid:107) ψ ( x , s 1 , . . .
Revised: MLPS (see (Hornik, 1991; Sonoda & Murata, 2017; Hanin & Sellke, 2017)) to construct ψ so that (cid:107) ψ ( x , s 1 , . . .

Original: , s t ) − q ( x , s 1 , . . .
Revised: , s t ) − q ( x , s 1 , . . .

Original: , s t ) (cid:107) ∞ < (cid:15) 2 for all ( x , s 1 , . . .
Revised: , s t ) (cid:107) ∞ < (cid:15) 2 for all ( x , s 1 , . . .

Original: , s t ) ∈ [0 , 1] k in + t .
Revised: , s t ) ∈ [0 , 1] k in + t .

Original: Furthermore, as ψ over [0 , 1] k in + t is uniformly continuous, let δ be such that if z , z (cid:48) ∈
Revised: Furthermore, as ψ over [0 , 1] k in + t is uniformly continuous, let δ be such that if z , z (cid:48) ∈

Original: [0 , 1] k in + t , (cid:107)
Revised: [0 , 1] k in + t , (cid:107)

Original: z − z (cid:48) (cid:107) ∞ <
Revised: z − z (cid:48) (cid:107) ∞ <

Original: δ then (cid:107) ψ ( z ) − ψ
Revised: δ then (cid:107) ψ ( z ) − ψ

Original: ( z (cid:48) )
Revised: ( z (cid:48) )

Original: (cid:107) ∞ <
Revised: (cid:107) ∞ <

Original: (cid:15) 2 .
Revised: (cid:15) 2 .

Original: Now, we use universality again to construct φ approximating b , that is we take φ so that (cid:107) φ ( x ) − b ( x ) (cid:107) ∞ <
Revised: Now, we use universality again to construct φ approximating b , that is we take φ so that (cid:107) φ ( x ) − b ( x ) (cid:107) ∞ <

Original: δ for all x ∈ [0 , 1] k in .
Revised: δ for all x ∈ [0 , 1] k in .

Original: (cid:107) F ( X ) − P
Revised: (cid:107) F ( X ) − P

Original: ( X ) (cid:107) ∞ ≤ (cid:107) Ψ
Revised: ( X ) (cid:107) ∞ ≤ (cid:107) Ψ

Original: ( L ( Φ ( X )))
Revised: ( L ( Φ ( X )))

Original: − Ψ ( L ( B ( X )))
Revised: − Ψ ( L ( B ( X )))

Original: (cid:107) ∞
Revised: (cid:107) ∞

Original: + (cid:107) Ψ
Revised: + (cid:107) Ψ

Original: ( L ( B ( X )))
Revised: ( L ( B ( X )))

Original: − Q ( L ( B ( X )))
Revised: − Q ( L ( B ( X )))

Original: (cid:107) ∞ =
Revised: (cid:107) ∞ =

Original: err 1 + err
Revised: err 1 + err

Original: First, (cid:107) L ( Φ ( X ))
Revised: First, (cid:107) L ( Φ ( X ))

Original: − L ( B ( X ))
Revised: − L ( B ( X ))

Original: (cid:107) ∞ <
Revised: (cid:107) ∞ <

Original: δ for all X ∈ K and therefore err 1 < (cid:15) 2 .
Revised: δ for all X ∈ K and therefore err 1 < (cid:15) 2 .

Original: Second, note that if
Revised: Second, note that if

Original: X ∈ K then B ( X ) ∈
Revised: X ∈ K then B ( X ) ∈

Original: [0 , 1] n × t and L ( B ( X )) ∈ [0 , 1] n × ( k in + t ) .
Revised: [0 , 1] n × t and L ( B ( X )) ∈ [0 , 1] n × ( k in + t ) .

Original: Therefore by construction of ψ we have err 2 < (cid:15) 2 .
Revised: Therefore by construction of ψ we have err 2 < (cid:15) 2 .

Original: To prove (iii) we use the result in Hanin & Sellke (2017) (see Theorem 1) bounding the width of an MLP approximating a function f : [0 , 1] d in → R d out by d in + d out .
Revised: To prove (iii) we use the result in Hanin & Sellke (2017) (see Theorem 1) bounding the width of an MLP approximating a function f : [0 , 1] d in → R d out by d in + d out .

Original: Therefore, the width of the
Revised: Therefore, the width of the

Original: MLP φ is bounded by k in + t , where the width of the MLP ψ is bounded by t + k in + k out , proving the bound.
Revised: MLP φ is bounded by k in + t , where the width of the MLP ψ is bounded by t + k in + k out , proving the bound.

Original: Before we prove Corollary 1 let us recall the PointNetSeg model from Qi et al.

Original: (2017) in detail.

Original: We write the model N as follows:

Original: First, we apply Ψ ( X ) =

Original: ( ψ ( x 1 ) , . . .

Original: , ψ ( x n )) T .

Original: Let ( η ( x 1 ) , . . .

Original: , η ( x n ))

Original: T ∈ R n × k 1 be the output of the ﬁrst layer of the mlp Ψ and let max ( Ψ ( X ))

Original: ∈

Original: R K be the vector where the i ’th coordinate is the maximal value over the i ’th column in Ψ ( X ) .

Original: Then N ( X ) =

Original: Φ (( η ( x 1 ) , max ( Ψ ( X ))) , . .

Original: , ( η ( x n ) , max ( Ψ ( X )))) T , with Φ ( X ) =

Original: ( φ ( x 1 ) , . . .

Original: , φ ( x n )) T .

Original: We can now prove Cororllary 1.
Revised: We can now prove Cororllary 1.

Original: Proof.
Revised: Proof.

Original: ( Corollary 1 )
Revised: ( Corollary 1 )

Original: The fact that the DeepSets model is equivariant universal is immediate.
Revised: The fact that the DeepSets model is equivariant universal is immediate.

Original: Indeed, The PointNetST model can be obtained from the DeepSets model by setting B = 0 in all but one layer, with B as in
Revised: Indeed, The PointNetST model can be obtained from the DeepSets model by setting B = 0 in all but one layer, with B as in

Original: Equation 8.
Revised: Equation 8.

Original: For the PointNetSeg model note that by Theorem 1 in Qi et al.
Revised: For the PointNetSeg model note that by Theorem 1 in Qi et al.

Original:  every invariant function f : R n × k in → R t can be approximated by a network of the form[Equation] Where ¯ Ψ ( x 1 , . . . , x n ) = ( ¯ ψ ( x 1 ) , . . .
Revised: (2017) every invariant function f : R n × k in → R t can be approximated by a network of the form Ψ ( max ( F ( X ))) , where ( F ( X ) is a PointNet model and Ψ is an MLP.

Original: , ¯ ψ ( x n ))

Original: T ∈ R n × K , with ¯ ψ : R k in → R K , ¯ φ : R K

Original: → R t multi layer perceptrons.

Original: In particular, for every ε > 0 there exists¯ φ , ¯ Ψ such that (cid:107)¯ φ ◦ max ◦ ¯ Ψ (  X ) − ( s 1 ( X ) , . . .
Revised: In particular, for every ε > 0 there exists such F , Φ for which (cid:107) Ψ ( max ( F ( X ))) − ( s 1 ( X ) , . . .

Original: , s t ( X ))
Revised: , s t ( X ))

Original: (cid:107) ∞ < ε for every X ∈ [0 , 1] n × k in where s 1 ( X ) , . . .
Revised: (cid:107) ∞ < ε for every X ∈ [0 , 1] n × k in where s 1 ( X ) , . . .

Original: , s t ( X ) are the power sum multi symmetric polynomials.
Revised: , s t ( X ) are the power-sum multi-symmetric polynomials.

Revised: It follows that we can use

Revised: PointNetSeg to approximate 1 ( s 1 ( X ) , . . .

Revised: , s t ( X )) .

Original: The rest of the prooffollows closely  the proof of Theorem 1.
Revised: The rest of the proof closely resembles the proof of Theorem 1.

Revised: Graph neural networks with constructed adjacency.

Revised: One approach sometimes applied to learning from sets of vectors is to deﬁne an adjacency matrix (e.g., by thresholding distances of node feature vectors) and apply a graph neural network to the resulting graph (e.g., Wang et al.

Revised: , Li et al.

Original: E XPERIMENTS
Revised: Using the common message passing paradigm (Gilmer et al., 2017) in this case boils to layers of the form: L ( X ) i = ψ ( x i , (cid:80) j ∈ N i φ ( x i , x j )) = ψ ( x i , (cid:80) j ∈ [ n ] N ( x i , x j ) φ ( x i , x j )) , where φ, ψ are MLPs, N i is the index set of neighbors of node i , and N ( x i , x j ) is the indicator function for the edge ( i, j ) .

Revised: If N can be approximated by a continuous function, which is the case for example if the data is ﬁnite, then since L is also equivariant it follows from Theorem 1 that such a network can be approximated arbitrarily well by any universal equivariant network such as

Revised: PointNetST or DeepSets.

Revised: We tested the ability of a DeepSets model with varying depth and width to approximate a single graph convolution layer.

Revised: We found that a DeepSets model with a small number of layers can approximate a graph convolution layer reasonably well.

Revised: For details see Appendix A.

Revised:  E XPERIMENTS

Original: We conducted experiments in order to validate our theoretical observations.
Revised: We conducted experiments in order to validate our theoretical observations.

Original: We compared the results of several equivariant models, as well as baseline (full) MLP, on three equivariant learning tasks: a classiﬁcation task (knapsack) and two regression tasks (squared norm and Fiedler eigen vector).
Revised: We compared the results of several equivariant models, as well as baseline (full) MLP, on three equivariant learning tasks: a classiﬁcation task (knapsack) and two regression tasks (squared norm and Fiedler eigen vector).

Original: For all tasks we compare results of 6 different models: DeepSets, PointNet, PointNetSeg, PointNetST,  and PointNetQT.
Revised: For all tasks we compare results of 7 different models: DeepSets, PointNet, PointNetSeg, PointNetST, PointNetQT and GraphNet.

Original: PointNetQT is PointNet with a single quadratic equivariant transmission layer as deﬁned in  the appendix.
Revised: PointNetQT is PointNet with a single quadratic equivariant transmission layer as deﬁned in Appendix B. GraphNet is similar to the graph convolution network in Kipf & Welling (2016) and is deﬁned explicitly in Appendix B. We generated the adjacency matrices for GraphNet by taking 10 nearest neighbors of each set element.

Original: In all experiments we used a network of the form Equation 7 with m =depth and varying width, ﬁxed across all layers.
Revised: In all experiments we used a network of the form Equation 7 with m = 6 depth and varying width, ﬁxed across all layers.

Original: Equivariant classiﬁcation.
Revised: Equivariant classiﬁcation.

Original: For classiﬁcation, we chose to learn the multidimensional knapsack problem, which is known to be NP-hard.
Revised: For classiﬁcation, we chose to learn the multidimensional knapsack problem, which is known to be NP-hard.

Original: We are given a set of 4 -vectors, represented by X ∈ R n × 4 , and our goal is to learn the equivariant classiﬁcation function f : R n × 4 → { 0 , 1 } n deﬁned by the following optimization problem:
Revised: We are given a set of 4 -vectors, represented by X ∈ R n × 4 , and our goal is to learn the equivariant classiﬁcation function f : R n × 4 → { 0 , 1 } n deﬁned by the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, given a set of vectors X ∈ R n × 4 , ( X ) ij = x ij , where each row represents an element in a set, our goal is to ﬁnd a subset maximizing the value while satisfying budget constraints.
Revised: Intuitively, given a set of vectors X ∈ R n × 4 , ( X ) ij = x ij , where each row represents an element in a set, our goal is to ﬁnd a subset maximizing the value while satisfying budget constraints.

Original: The ﬁrst column of X deﬁnes the value of each element, and the three other columns the costs.
Revised: The ﬁrst column of X deﬁnes the value of each element, and the three other columns the costs.

Revised: To evaluate the success of a trained model we record the percentage of sets for which the predicted subset is such that all the budget constrains are satisﬁed and the value is within 10% of the optimal value.

Original: In Subsection 5.1 we detail how we generated this dataset.
Revised: In Appendix C we detail how we generated this dataset.

Original: Equivariant regression.
Revised: Equivariant regression.

Original: The ﬁrst equivariant function we considered for regression is the function f ( X ) = 1 (cid:80) ni
Revised: The ﬁrst equivariant function we considered for regression is the function f ( X ) = 1 (cid:80) ni

Original: =1 (cid:80) kj =1 ( X i,j − 12 ) 2 .
Revised: =1 (cid:80) kj =1 ( X i,j − 12 ) 2 .

Original: Hanin & Sellke (2017) showed this function cannot be approximated by MLPs of small width.
Revised: Hanin & Sellke (2017) showed this function cannot be approximated by MLPs of small width.

Original: We drew 10 k training examples and 1 k test examples i.i.d.
Revised: We drew 10 k training examples and 1 k test examples i.i.d.

Original: from a
Revised: from a

Original: N ( 12 , 1) distribution (per entry of X ).
Revised: N ( 12 , 1) distribution (per entry of X ).

Original: The second equivariant function we considered is deﬁned on point clouds X ∈ R n × 3 .
Revised: The second equivariant function we considered is deﬁned on point clouds X ∈ R n × 3 .

Original: For each point cloud we computed a graph by connecting every point to its 10 nearest neighbors. 
Revised: For each point cloud we computed a graph by connecting every point to its 10 nearest neighbors. We then computed [Figure]

Revised: (with the same number of parameters as the equivariant models) performs poorly.

Original: We then computed the absolute value of the ﬁrst non trivial eigenvector of the graph Laplacian.
Revised: the absolute value of the ﬁrst non trivial eigenvector of the graph Laplacian.

Original: We used the ModelNet dataset (Wu et al., 2015) which contains ∼ 9 k training meshes and ∼ 2 k test meshes.
Revised: We used the ModelNet dataset (Wu et al., 2015) which contains ∼ 9 k training meshes and ∼ 2 k test meshes.

Original: The point clouds are generated by randomly sampling 512 points from each mesh.
Revised: The point clouds are generated by randomly sampling 512 points from each mesh.

Original:  Figure 2 summarizes train and test accuracy of the 6 models after training (training details in Subsection 5.1) as a function of the network width ω .
Revised: Result summary. Figure 2 summarizes train and test accuracy of the 6 models after training (training details in Appendix C) as a function of the network width ω .

Original: We have tested 15 ω values equidistant in [5 , nk in 2 ] .
Revised: We have tested 15 ω values equidistant in [5 , nk in 2 ] .

Original: As can be see in the graphs, in all three datasets the equivariant universal models (PointNetST,
Revised: As can be seen in the graphs, in all three datasets the equivariant universal models (PointNetST,

Original: PointNetQT , DeepSets, PointNetSeg) achieved comparable accuracy.
Revised: PointNetQT , DeepSets, PointNetSeg) achieved comparable accuracy.

Original: PointNet, which is not[Figure] equivariant universal, consistently achieved inferior performance compared to the universal models, as expected by the theory developed in this paper.
Revised: PointNet, which is not equivariant universal, consistently achieved inferior performance compared to the universal models, as expected by the theory developed in this paper.

Original: The non-equivariant MLP, although universal, used the same width (i.e., same number of parameters) as the equivariant models and was able to over-ﬁt only on one train set (the quadratic function); its performance on the test sets was inferior by a large margin to the equivariant models.
Revised: The non-equivariant MLP, although universal, used the same width (i.e., same number of parameters) as the equivariant models and was able to over-ﬁt only on one train set (the quadratic function); its performance on the test sets was inferior by a large margin to the equivariant models.

Revised: We also note that in general the GraphNet model achieved comparable results to the equivariant universal models but was still outperformed by the DeepSets model.

Original: An interesting point is that although the width used in the experiments in much smaller than the bound k out + k in + (cid:0) n + k in k in (cid:1) established by Theorem 1, the universal models are still able to learn well the functions we tested on.
Revised: An interesting point is that although the width used in the experiments in much smaller than the bound k out + k in + (cid:0) n + k in k in (cid:1) established by Theorem 1, the universal models are still able to learn well the functions we tested on.

Original: This raises the question of the tightness of this bound, which we leave to future work.  
Revised: This raises the question of the tightness of this bound, which we leave to future work.  C ONCLUSIONS

Revised: In this paper we analyze several set equivariant neural networks and compare their approximation power.

Original: Lastly, we see that  adding a quadratic transmission layer (PointNetQT) has a marginal effect on  the  model performance, while adding a single linear transmission layer toa PointNet network gives similar results to the DeepSets architecture, while having lower memory footprint.
Revised: We show that while vanilla PointNet (Qi et al., 2017) is not equivariant universal, adding a single linear transmission layer makes it equivariant universal. Our proof strategy is based on a characterization of polynomial equivariant functions. As a corollary we show that the DeepSets model (Zaheer et al., 2017) and PointNetSeg (Qi et al., 2017) are equivariant universal. Experimentally, we tested the different models on several classiﬁcation and regression tasks ﬁnding that adding a single linear transmitting layer to PointNet makes a signiﬁcant positive impact on performance.

Revised: A A PPROXIMATING GRAPH CONVOLUTION LAYER WITH D EEP S ETS

Revised: To test the ability of an equivariant universal model to approximate a graph convolution layer, we conducted an experiment where we applied a single graph convolution layer (see Appendix B for a full description of the graph convolution layers used in this paper) with 3 in features and 10 out features.

Revised: We constructed a knn graph by taking 10 neighbors.

Revised: We sampled 1000 examples in R 100 ×i.i.d from a N ( 12 , 1) distribution (per entry of X ).

Original: 5. I MPLEMENTATION DETAILS
Revised: The results are summarized in Figure 3. We regressed to the output of a graph convolution layer using the smooth L 1 loss. [Figure] B D ESCRIPTION OF LAYERS B. Q UADRATIC LAYER

Revised: One potential application of Theorem 2 is augmenting an equivariant neural network (Equation 7) with equivariant polynomial layers P : R n × k

Revised: → R n × l of some maximal degree d .

Revised: This can be done in the following way: look for all solutions to α, β 1 , β 2 , . . .

Revised: ∈ N k

Revised: so that | α | + (cid:80) i | β i | ≤ d .

Revised: Any solution to this equation will give a basis element of the form p ( X ) =

Revised: (cid:100)

Revised: [Equation]

Revised: In the paper we tested PointNetQT, an architecture that adds to PointNet a single quadratic equivariant layer.

Revised: We opted to use only the quadratic transmission operators: For a matrix X ∈ R n × k we deﬁne

Revised: L ( X ) ∈ R n × k as follows:

Revised: [Equation]

Revised: where (cid:12) is a point-wise multiplication and W i ∈ R n × k , i ∈

Revised: [5] are the learnable parameters.

Revised: B. G RAPH CONVOLUTION LAYER

Revised: We implement a graph convolution layers as follows

Revised: [Equation]

Revised: with W 1 , W 2 , c learnable parameters.

Revised: The matrix B is deﬁned as in Kipf & Welling (2016) from the knn graph of the set.

Revised: B = D − 12 AD − 12 where D is the degree matrix of the graph and A is the adjacency matrix of the graph with added self-connections.

Revised: C I MPLEMENTATION DETAILS

Original: Knapsack data generation.
Revised: Knapsack data generation.

Original: We constructed a dataset of 10 k training examples and 1 k test examples consisting of 50 × 4 matrices.
Revised: We constructed a dataset of 10 k training examples and 1 k test examples consisting of 50 × 4 matrices.

Original: We took w 1 = 100 , w 2 = 80 , w 3 = 50 .
Revised: We took w 1 = 100 , w 2 = 80 , w 3 = 50 .

Original: To generate X ∈ R 50 × 4 , we draw an integer uniformly at random between 1 and 100 and randomly choose 50 integers between 1 as the ﬁrst column of X .
Revised: To generate X ∈ R 50 × 4 , we draw an integer uniformly at random between 1 and 100 and randomly choose 50 integers between 1 as the ﬁrst column of X .

Original: We also randomly chose an integer between 1 and 25 and then randomly chose 150 integers in that range as the three last columns of X .
Revised: We also randomly chose an integer between 1 and 25 and then randomly chose 150 integers in that range as the three last columns of X .

Original: The labels for each input X were computed by a standard dynamic programming approach, see Martello & Toth (1990).
Revised: The labels for each input X were computed by a standard dynamic programming approach, see Martello & Toth (1990).

Original: Optimization.
Revised: Optimization.

Original: We implemented the experiments in Pytorch Paszke et al.
Revised: We implemented the experiments in Pytorch Paszke et al.

Original: (2017) with the Adam
Revised: (2017) with the Adam

Original: Kingma & Ba (2014) optimizer for learning.
Revised: Kingma & Ba (2014) optimizer for learning.

Original: For the classiﬁcation we used the cross entropy loss and trained for 150 epochs with learning rate 0.001, learning rate decay of 0.5 every 100 epochs and batch size 32.[-For the quadratic function regression we trained for 150 epochs with leaning rate of 0.001, learning rate decay 0.
 every 50 epochs A A PPENDIX-]
Revised: For the classiﬁcation we used the cross entropy loss and trained for 150 epochs with learning rate 0.001, learning rate decay of 0.5 every 100 epochs and batch size 32.[-For the quadratic function regression we trained for 150 epochs with leaning rate of 0.001, learning rate decay 0.
 every 50 epochs A A PPENDIX-]

Original: One potential application of Theorem 2 is augmenting an equivariant neural network (Equation 7) with equivariant polynomial layers P : R n × k

Original: → R n × l of some maximal degree d .

Original: This can be done in the following way: look for all solutions to α, β 1 , β 2 , . . .

Original: ∈ N k so that | α | + (cid:80) i | β i | ≤ d . Any solution to this equation will give a basis element of the form p ( X ) = (cid:100) x{+quadratic function regression we trained for 150 epochs with leaning rate of 0.001, learning rate decay 0.
+}
Revised: For the {+quadratic function regression we trained for 150 epochs with leaning rate of 0.001, learning rate decay 0.
+}

Original: α 1 (cid:101) (cid:81) j (cid:16)(cid:80) ni =1 x β j i (cid:17) . In the paper we tested PointNetQT, an architecture that adds to PointNet a single quadratic equivariant layer. We opted touse only the quadratic transmission operators: For a matrix X ∈ R n × k we deﬁne
Revised: every 50 epochs and batch size 64; for the regression to the leading eigen vector we trained for 50 epochs with leaning rate of 0.001 and batch size 32.

Original: L ( X ) ∈ R n × k as follows:

Original: [Equation] where (cid:12) is a pointwise multiplication and W i ∈ R n × k , i ∈ [5] are the learnable parameters.
Revised: To regress to the output of a single graph convolution layer we trained for 200 epochs with leaning rate of 0.001 and batch size 32.