L IMITATIONS FOR L EARNING FROM P OINT C LOUDS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we prove new universal approximation theorems for deep learning on point clouds that do not assume ﬁxed cardinality.
We do this by ﬁrst generalizing the classical universal approximation theorem to general compact Hausdorff spaces and then applying this to the permutation-invariant architectures presented in PointNet (Qi et al) and Deep Sets (Zaheer et al).
Moreover, though both architectures operate on the same domain, we show that the constant functions are the only functions they can mutually uniformly approximate.
In particular, DeepSets architectures cannot uniformly approximate the diameter function but can uniformly approximate the center-of-mass function but it is the other way around for
PointNet.
Additionally, even when the point clouds are limited to at most k points,
PointNet cannot uniformly approximate center-of-mass and we obtain explicit error bounds and a method to produce geometrically derived adversarial examples.
I NTRODUCTION
Recently, architectures proposed in PointNet (Qi et al., 2017) and Deep Sets (Zaheer et al., 2017) have allowed for the direct processing of point clouds within a deep learning framework.
These methods produce outputs that are permutation-invariant with respect to the member points and work for point clouds of arbitrarily large cardinality.
A common source of such data is LIDAR measurements from autonomous vehicles.
Zaheer et al.
(2017) also presents a permutation-equivariant architecture which we do not discuss here.
Each of of these works provide their own universal approximation theorem (UAT) to support the empirical success of their architectures.
However, both results assume the cardinality of the point cloud is ﬁxed to some size n .
In this work we reﬁne these results, remove the cardinality limitation, use weaker architecture assumptions, and arrive at three main results which can be summarized roughly as follows (assuming unrestricted ﬁnite cardinality for the input point clouds):
1) PointNet (DeepSets) architectures can uniformly approximate real-valued functions that are uniformly continuous with respect to the Hausdorff (Wasserstein) metric and nothing else (Theorem 3.4).
Only the constant functions can be uniformly approximated by both architectures.
In particular, PointNet architectures can uniformly approximate the diameter function but DeepSets architectures cannot.
Conversely, DeepSets architectures can uniformly approximate the center-of-mass function but PointNet architectures cannot (Theorem 4.1).
We prove explicit error lower bounds and produce adversarial examples to show that even when limited to point clouds of size k , PointNet cannot uniformly approximate center-ofmass (Theorem 4.2).
To do this we extend the many universal approximation results for feed-forward networks (Cybenko, 1989;
Hornik et al., 1989; Leshno et al., 1993; Stinchcombe, 1999) to the abstract setting of general compact Hausdorff spaces.
We then ﬁnd appropriate compact metric spaces over which PointNet and DeepSets architectures can be easily analyzed and then ﬁnally we observe the resulting consequences in the original setting of interest, i.e. point clouds.
P RELIMINARIES 2. P OINT N ET AND D EEP S ETS A RCHITECTURES
In practice, the implementations of the architectures presented in PointNet and Deep Sets can involve many additional tricks, but the essential ideas are quite simple.
We do however make a small modiﬁcation to the Deep Sets model.
For A ⊆ R n of cardinality | A | < ∞ , we have Qi et al.
(2017) and Zaheer et al.
(2017) suggesting scalar-output neural networks of the form
[Equation]
and
[Equation]
respectively.
Here ϕ : R n → R m creates features for each point in A , then a symmetric operation is applied, and then ρ : R m → R combines these features into a scalar output (here max is the component-wise maximum).
In practice, we need both ρ and ϕ to be neural networks.
Note that because we use a symmetric operation before ρ , the output will not depend on the ordering of points in the point cloud, and because the max and sum operations scale to arbitrary ﬁnite cardinalities the size of the point cloud is not an issue.
The original model in Deep Sets did not have a bias term b and used a sum instead of the averaging we use here.
This change will help us later in our theoretical analysis.
It will help to introduce some simplifying notation.
Let F (Ω) denote the set of all nonempty ﬁnite subsets of a set Ω (i.e. point clouds in Ω ), F ≤ k (Ω) the set of nonepmty subsets of size ≤ k , and F k (Ω) the set of k -point subsets.
Now consider Ω ⊆ R N and deﬁne max f , ave f,b : F (Ω) → R which are given by max f ( A ) = max a ∈ A f ( a ) and ave f,b ( A )
=
b + 1 | A | (cid:80) a ∈
A f ( a ) respectively.
We make sense of this in the natural way if we use vector-valued f and b by operating componentwise.
We call these operations max neurons and biased-averaging neurons respectively.
Once again letting ρ and ϕ be neural networks, F PN
= ρ ◦ max ϕ and F DS = ρ ◦ ave ϕ , b will be the general form of what we call the PointNet and DeepSets architectures (resp.)
in this paper.
Some natural questions are 1) is there a topology for F (Ω) that makes these architectures continuous, 2) how expressive are these approaches, and 3) how deep is deep enough for function approximation?
2. F UNCTION S PACES AND U NIFORM A PPROXIMATION
From now on, we only consider R -valued functions unless otherwise stated.
Let B ( A ) be the set of bounded functions on a set A , let C ( X ) and C b ( X ) be the set of continuous and bounded continuous functions on a topological space X (respectively), and let U ( M ) and U b ( M ) be the uniformly continuous and bounded uniformly continuous functions on a metric space ( M, d ) (respectively).
We equip all of these with the uniform norm i.e. ||| f |||
A = sup a ∈ A | f ( a ) | – we reserve (cid:107)∗(cid:107) for the
Euclidean norm. This makes them all normed spaces, with B ( A ) , C b ( X ) and U b ( M ) additionally being Banach spaces.
Moreover, if X is compact and ( M, d ) has compact metric completion, then C ( X ) =
C b ( X ) and U ( M ) =
U b ( M ) and hence are also Banach spaces.
For background see Rudin (2006).
If given an injective map i : A → X , then we say that ϕ
A → R (uniquely) continuously extends to X if there is a (unique) (cid:101)
ϕ ∈ C ( X ) such that (cid:101)
ϕ ◦ i
= ϕ .
We say a family of funcntions N on A (uniquely) continuously extends to X if every ϕ ∈ N (uniquely) continuously extends to X .
We will make use of the following lemma which is proved in the appendix.
Lemma 2.1.
Let N ⊆ B
( D ) where D is a dense subset of a compact metric space ( X, d ) .
Suppose
N has a continuous extension to X denoted by N (cid:48) ⊆
C ( X ) which is dense.
Then the uniform closure of N in B ( D ) is r ( C ( X ))
=
U ( D ) where r : C ( X ) → C b ( D ) is the domain restriction map.
Letting D = F (Ω) , this lemma suggest the following plan of attack: ﬁnd a compact metric space ( X, d ) in which we can realize F (Ω) as a dense subset and hope that our class of neural networks
N continuously extends to a dense subset of C ( X ) .
If we can do that, then we know the uniform closure of our class of neural networks are precisely the uniformly continuous functions on F (Ω) with respect to the metric inherited from X .
This motivates the next subsection.
2. M ETRICS ON THE S PACE OF P OINT C LOUDS
From now on we will assume (Ω , d ) is a compact metric space and when Ω ⊆ R n it will be compact and equipped with the Euclidean metric.
Let K (Ω) denote the set of all compact subsets of Ω and
P (Ω) denote the set of all Borel probability measures on Ω .
The Hausdorff metric d H (Munkres, 2000) is a natural metric for K (Ω) and 1-Wasserstein metric d W (Villani, 2009)
(also called the
Earth-mover distance) is a natural metric for P (Ω) .
With these metrics, K (Ω) and P (Ω) become compact metric spaces of their own.
From now on we will assume these two spaces are always equipped with the aforementioned metrics.
We also brieﬂy mention M (Ω) the Banach space of ﬁnite signed regular Borel measures on Ω .
By the Riesz-Markov theorem it is the topological dual space of C (Ω) .
Of interest to us is that
P (Ω) ⊆
M (Ω) and that the weak-* topology on P (Ω) coincides with the topology induced by d W .
This means that
d W ( µ n , µ ) → 0 iff (cid:82) f d µ n → (cid:82) f d µ for all f ∈ C (Ω) .
Next, note that F (Ω) ⊆
K (Ω) and let i K denote the natural inclusion map.
We can also deﬁne an injective map i P : F (Ω) → P (Ω) by mapping A ∈ F (Ω) to its associated empirical measure i P ( A ) =
µ A = 1 | A | (cid:80) a ∈
A δ a ∈ P (Ω) where δ a is the Dirac delta measure supported at a .
The injective maps
i K
and i P allow us to induce the d H and d W metrics on F (Ω) .
We will denote the metrized versions by F H (Ω) and F W (Ω) respectively and use the same convention for F kH (Ω) and
F kW (Ω) .
Another important fact to know is that i K and i P embed F (Ω) as dense subset of K (Ω) and P (Ω) .
The former follows from compactness of the members of K (Ω) and to see why the latter is true see Fournier & Guillin (2015);
Villani (2009)
For f ∈ C (Ω) and b ∈ R , deﬁne Max f : K (Ω) → R and Ave f,b : P (Ω) → R as the functions
Max f ( K ) =
max x ∈ K f ( x ) and Ave f,b ( µ )
=
b + (cid:82) Ω f d µ .
Lemma 2.2.
Let (Ω , d ) be compact, f ∈ C (Ω) , and b ∈ R .
Then Max f ∈ C ( K (Ω)) and Ave f,b ∈
C ( P (Ω)) and Max f ◦ i K = max f and Ave f,b ◦ i P = ave f,b .
As a consequence, PointNet and DeepSets are uniformly continuous on F H (Ω) and F W (Ω) respectively. This lemma (proved in the appendix) tells us that the max neurons and biased-averaging neurons continuously extend to K (Ω) and P (Ω) and hence so do PointNet and DeepSets architectures (since we merely compose with the continuous ρ after).
Thus, we wil be able analyze such architectures as continuous functions on compact metric spaces, which is mathematically a much nicer problem than studying them as set-theoretic functions on an un-metrized F (Ω) .
2. G ENERALIZED N EURAL N ETWORK N OTATION
For A a collection of functions from X to Y and B a collection of functions from Y to Z , we denote the set of all compositions by B ◦
A =
{ f ◦ g | f ∈ B , g ∈ A} .
In the case of a single function σ : Y → Z we let σ ◦
A
=
σ ◦ f | f ∈ A} and similarly for right-composition.
Next, let Aﬀ denote the set of all afﬁne functionals on R N , i.e. any function of the form f ( x ) = w · x + b .
Let N σ := span { σ ◦ Aﬀ } denote the set of single hidden-layer neural networks with linear output-layer and activation function σ , and then denote an H -layered network by N σ where σ = ( σ 1 , . . .
, σ H ) is a list of H -many activation functions where N ( σ ,τ ) :
= N σ ,τ := span( τ ◦ N σ ) .
Next we deﬁne various classes of PointNet networks whose weight functions are themselves neural networks.
Let N σPN := N σ ; ∅ PN :=
span { max f | f ∈ N σ } then deﬁne N σ ; τ PN :
= span { τ ◦ N σPN } .
Like before, we can inductively deﬁne deeper networks, but we can also use deeper weight networks as well to create N σ ; τ PN – thus we have two distinct notions of depth.
Next, we do the same for DeepSets.
Let N σDS := N σ ; ∅ DS := span { ave f,b | f ∈ N σ , b ∈ R } .
Note that ave f,b + ave g,c
= ave f
+ g,b
+ c
and α ave f,b = ave αf,αb .
Thus since N σ is a linear space, taking the span has no effect and N σDS = { ave f,b | f ∈ N σ , b ∈ R } .
Going one layer deeper yields
N σ ; τ DS := span { τ ◦ N σDS } which gets us new functions.
Like with PointNet, we can inductively develop deeper families in two ways.
By Lemma 2.2 we can extend all the operations of our neural networks to K (Ω) and P (Ω) in a natural way.
This let’s us talk about about PointNet networks on K (Ω) and DeepSets networks on
P (Ω) which we’ll deﬁne analogously by replacing max f with Max f and ave f,b with Ave f,b .
Thus
[Equation]
As before, the linear structure of N σ makes M σDS = { Ave f,b | f ∈ N σ }
M AIN R ESULTS 3. T OPOLOGICAL UAT
Leshno et al.
(1993) prove that N σ with σ ∈ C
( R ) has universal approximation property iff σ is not a polynomial.
For this reason, we will say a σ ∈ C ( R ) is ‘universal’ if it is non-polynomial and denote the set of all such such functions by U ( R ) .
Using this theorem and Stone-Weierstrass we prove a UAT for certain kinds of two-hidden-layer ‘neural networks’ on an abstract compact
Hausdorff space.
Recall that a family of functions S on Ω separates points if for any x (cid:54) =
y there is an f ∈ S so that f ( x ) (cid:54) =
f ( y ) .
Theorem 3.1 (Topological-UAT) .
Let X be a compact Hausdorff space and σ ∈ U ( R ) .
If S ⊆
C ( X ) separates points and contains a nonzero constant, then span( σ ◦ span S ) is dense in C ( X ) .
Additionally, if S also happens to be a linear subspace, then span( σ ◦ S ) is dense in C ( X ) .
Proof.
Let S and σ satisfy the above and let V = span S .
Let Alg( V ) denote the algebra generated by V , i.e. all possible ﬁnite products, sums and scalar multiples of the elements of V .
Then Alg( V ) is unital subalgebra of C ( X ) that seperates points.
By the Stone-Weierstrass theorem Alg( V ) is dense in C ( X ) .
Now let F ∈ C ( X ) and (cid:15) > 0 be arbitrary.
By density there is a G ∈ Alg( V ) such that | F ( a ) − G ( a )
| < (cid:15)/ 2 for all a ∈ X .
Since G ∈ Alg( V ) there is an N -variable polynomial p and s = ( s 1 , . . .
, s N )
where s i ∈ S , so that G =
p ◦ s .
Since all s i ∈ C
( X ) and X is compact, the image s ( X ) ⊆
R N is compact.
By the classical UAT (Leshno et al., 1993), there exists an η ∈ N σ such that | p ( x ) − η ( x ) | < (cid:15)/ 2 for all x ∈ R N .
Thus,
[Equation]
for every a ∈ X .
Finally note that η ( s ( a )) =
(cid:80) mi
=
a i σ ( w i · s ( a ) + b i ) for some a i , b i ∈ R and w i ∈ R N .
Since S constains a nonzero constant, span S contains every constant and so w i · s + b i ∈ span S .
Thus η ◦ s ∈ span( σ ◦ span S ) as desired.
Lastly, if S is also linear subspace, then S = span S and so span( σ ◦ S ) is dense in C ( X ) .
3. P OINT C LOUD UAT
We have met almost all the conditions required to use the topological-UAT on K (Ω) and P (Ω) .
We just need to show that Max f and Ave f,b yield nonzero constants and can separate points even when we limit ourselves to f ∈ N σ .
Lemma 3.2 (Separation Lemma) .
Let Ω ⊆ R N be compact and σ ∈ U
( R ) .
Then S PN =
{ Max f | f ∈ N σ } and S DS = { Ave f,b | f ∈ N σ , b ∈ R } separate points and contain constants.
Proof.
Let d denote the Euclidean distance.
First note that the constant function h = σ ( c ) ∈ N σ for some c ∈ R .
Since σ is not a polynomial, there is a choice of c for which σ ( c ) (cid:54) = 0 .
This means
Max h ∈ S PN and Ave h, 0 ∈ S DS are both constant.
Now we just need to show that S PN and S DS separate points.
( S PN separates points):
Let A, B ∈ K (Ω) with A (cid:54) =
B .
Without loss of generality, A \ B (cid:54) = ∅ so choose a ∈ A \ B .
Let f ( x ) =
min { 1 , d ( x, B ) /d ( a, B ) } and note that f ( a )
= 1 , f ( B ) = { 0 } and f (Ω) =
By the classical UAT (Leshno et al., 1993) N σ is dense in C (Ω) , so there is a g ∈ N σ
so that | f ( x ) − g ( x ) | < 1 / 2 for all x ∈ Ω .
Note Max g ∈ S PN and that Max g ( A ) > 1 /and Max g ( B ) < 1 / 2 .
Since A and B were arbitrary, this shows S PN separates point in K ( X ) .
( S DS separates points): Given µ 1 , µ 2 ∈ P (Ω) with µ 1 (cid:54)
= µ 2 , by the Hahn-Banach separation theorem there exists a weak-* continuous linear functional L : M (Ω) → R that separates them.
Let δ = | L ( µ 1 ) − L ( µ 2 ) | .
The topological dual of M (Ω) with the weak-
* topology is equivalent to
C (Ω) and so there is an f ∈ C (Ω) so that L ( η ) = (cid:82) f
d η for all η ∈ M (Ω) .
Since N σ is dense in
C (Ω) there is a g ∈ N σ
so the that | f ( x ) − g ( x ) | <
δ/ 2 for all x ∈ Ω .
Deﬁne J ( η ) =
(cid:82) g d η .
Then for all η ∈ P (Ω) we have | L ( η ) − J ( η ) | ≤ (cid:82)
|
f − g |
d η < δ 2 (cid:82) d η = δ/ 2 .
Applying the triangle inequality we obtain,
[Equation]
Thus 0 < | J ( µ 1 ) − J ( µ 2 ) | and so J
= Ave g, 0 ∈ S DS separates µ 1 and µ 2 .
Since µ 1 and µ 2 were arbitrary, it follows that S DS seperates points in P (Ω) .
The following theorems show that one hidden layer in the weight networks and one hidden layer of the of the other kind sufﬁce to prove the universal approximation theorems for PointNet and
DeepSets.
Theorem 3.3.
Let Ω ⊆ R N be compact and σ, τ ∈ U ( R ) .
Then M σ ; τ PN and M σ ; τDS are dense in C ( A ) and C ( B ) respectively, where A ⊆ K (Ω) and B ⊆ P (Ω) are closed subsets.
Proof.
Recall M σ ; τ PN = span { τ ◦ span S PN } and M σ ; τDS = span { τ ◦ S DS } .
Since K (Ω) and
P (Ω) are compact metric spaces, A and B are compact Hausdorff.
By Lemma 3.2 we know S PN and S DS separate points and contain nonzero constants and so the topological-UAT (Theorem 3.1) yields the desired result.
Theorem 3.4 (Point-Cloud-UAT) .
Let Ω ⊆ R N be compact.
If σ, τ ∈ U ( R ) , then the uniform closure of N σ ; τ PN and N σ ; τ DS within B ( F (Ω)) is U ( F H (Ω)) and U ( F W (Ω)) respectively.
Proof.
F H (Ω) and F W (Ω) are isometrically isomorphic to i K ( F (Ω)) and
i P ( F (Ω)) which are in turn dense in ( K (Ω) , d H ) and ( P (Ω) , d W ) .
By Lemma 2.2 we have that N σ ; τ PN and N σ ; τ DS continuously extend to K (Ω) and P (Ω) as M σ ; τ PN and M σ ; τDS .
By Theorem 3.3 we know M σ ; τ PN and M σ ; τDS are dense in C ( K (Ω)) and C ( P (Ω)) .
Finally, by Lemma 2.1 we have the desired result.
It’s worth noting that we could have used Stinchcombe’s generalization of the UAT to the case of neural networks on compact subsets of locally convex spaces (Stinchcombe, 1999) to prove that
M σ ; τDS is dense in C ( P (Ω)) but we chose the above route for consistency of technique and to be self-contained.
We now prove as a corollary a reﬁnement of the universal approximation theorems of Qi et al.
(2017) and Zaheer et al.
(2017), both of which applied to the the case of k -point point clouds.
In this version of the theorem we are able to restrict the depth of the neural network to just two hidden layers.
The proof is essentially the same as Theorem 3.4.
Corollary 3.5.
Let Ω ⊆ R N be compact.
If σ, τ ∈ U ( R ) , then the uniform closure of N σ ; τ PN and
N σ
; τ DS within B ( F k (Ω)) are U ( F k (Ω) H ) and U ( F k (Ω) W ) respectively.
Proof.
F kH (Ω) and F kW (Ω) are isometrically isomorphic to i K
( F k (Ω)) and i P ( F k (Ω)) which are in turn dense in their respective closures which we denote G H (Ω) ⊆
K (Ω) and G W (Ω) ⊆
P (Ω) .
Thus by Lemma 2.2 and Theorem 3.3 we have that M σ ; τ PN and M σ ; τDS are dense in C ( G H (Ω)) and
C ( G W (Ω)) .
Finally, by Lemma 2.1 we have the desired result.
L IMITATIONS OF P OINT N ETS AND D EEP S ETS
Note that unlike the classical universal approximation theorem we should not expect to be able uniformly approximate C ( F H (Ω)) or C ( F W (Ω)) since their elements might not even be bounded functions.
For example, α K ( A ) =
d H ( A, K ) − 1 is unbounded but continuous on F H (Ω) whenever
K ∈ K (Ω) but K / ∈ F (Ω) .
Subtler still, we do not even obtain all elements of C b ( F H (Ω)) and [Figure]
C b ( F W (Ω)) .
As an example of this, observe that β K = sin ◦
α K is bounded and is also continuous on F H (Ω) because α K is.
We’ll now compare the representation power of these two architectures.
Let Ω ⊆ R N be compact.
We deﬁne the point cloud diameter function Diam : F (Ω) → R and point cloud center-of-mass function Cent : F (Ω) → R N by Diam( A ) =
max x , y ∈
A d ( x , y ) and Cent( A ) = 1 | A | (cid:80) x ∈ A x .
Theorem 4.1.
Let (Ω , d ) be an inﬁnite compact metric space with no isolated points.
Then a function f : F (Ω) → R is continuous with respect to both d H and d W iff it is constant.
As a corollary,
Diam is uniformly approximable by PointNet networks but not DeepSets networks and Cent is uniformly approximable by DeepSets networks but not PointNet networks.
Proof.
Assume f : F (Ω) → R is both d H -continuous and d W -continuous.
Let A ∈ F (Ω) and let
p ∈
A .
For each n = 1 , 2 , . . . choose A (cid:48) n ∈ F (Ω) to be an n -point set contained within the 1 /n -ball around p .
We can do this because Ω is inﬁnite without isolated points.
Now let A n = A (cid:48) n ∪ ( A \{ p } ) .
Observe that A n d H → A and A n d W → { p } .
Thus,
[Equation]
Note that A was arbitrary so f must always give a set and any of its singleton subsets the same value.
Now let B, C ∈ F (Ω) such that B (cid:54) =
C .
Without loss of generality assume q ∈ B \ C and q (cid:54) =
r ∈ C .
Thus by the above,
[Equation]
thus f must be constant.
Conversely, constant maps are always continuous.
Finally, it’s known that the Diam satisﬁes | Diam( A ) − Diam( B ) | ≤ 2 d H
( A, B ) and hence is d Hcontinuous on K (Ω) and Cent is d W -continuous on P (Ω) because Ave π i , 0 is d W -continuous
(here π i is the projection onto the i -th component map).
This means they are uniformly continuous on F H (Ω) and F W (Ω) respectively and so the result follows from the above and Theorem 3.4.
While it is interesting to know that these neural networks describe fundamentally different kinds of functions when we allow for unbounded cloud cardinality, in practice there is always a bound due to computational resource limitations.
The next result shows that even when we bound the cloud cardinality, PointNet still cannot uniformly approximate the center-of-mass function.
Theorem 4.2.
Let Ω ⊆ R d be an inﬁnite compact set with no isolated points. Then for every d H -continuous F : F ≤ k (Ω) → R d there exists A ∈ F k (Ω) such that
[Equation]
In particular, since PointNet architectures are d H -continuous they cannot uniformly approximate center-of-mass when k ≥ 3 and suffer from the above uniform-norm error lower-bound.
It is possible to see that PointNet cannot uniformly approximate Cent by considering Figure 1a.
Moving k − 2 points in a k -element cloud to either of the remaining two points produce the same 2element cloud in a d H -continuous way.
By d H -continuity, this means PointNet must output similar centers for the cloud in the top-right and bottom-right – this introduces error.
To obtain the explicit error lower bound in Theorem 4.2, we need a slightly more detailed result.
Lemma 4.3.
Let Ω ⊆
R
d be inﬁnite with no isolated points and F : F ≤ k (Ω) → R d an arbitrary d H -continuous map.
Then for every distinct p, q ∈ Ω and 0 < τ < 1 there exists an A ∈ F k (Ω) such that p, q ∈ A and (cid:107) F
[Equation]
A
[Equation]
Cent
[Equation]
A
[Equation]
>
[Equation]
τ
[Equation]
In particular, since PointNet architectures are d H -continuous they must satisfy this error bound.
Proof.
Assume k ≥ 3 since the inequality is trivial for k = 1 , 2 .
Let C p :
= p + qk + ( k − 2 k ) p and
C q := p + qk + ( k − 2 k ) q and observe that (cid:107) C p − C q (cid:107) =
k − 2 k (cid:107) p − q (cid:107) .
Now let (cid:15) =
τ (cid:107) C p − C q (cid:107) /and
choose A (cid:15)p and A (cid:15)q to be k -point supersets of { p, q } such that A (cid:15)p \ { q } ⊆ B (cid:15) ( p ) and A (cid:15)q \ { p } ⊆
B (cid:15) ( q ) .
We can do this because p and q are not isolated points.
Since F is d H -continuous and
A (cid:15)p , A (cid:15)q d H → { p, q } as (cid:15) → 0 , we can additionally demand that (cid:13)(cid:13) F ( A (cid:15)p ) − F ( { p, q } ) (cid:13)(cid:13) < (cid:15) and (cid:13)(cid:13) F ( A (cid:15)q ) − F ( { p, q } ) (cid:13)(cid:13) < (cid:15) .
Next observe that
[Equation]
By the triangle inequality it follows that (cid:13)(cid:13)
Cent( A (cid:15)p ) − C p (cid:13)(cid:13) < (cid:15) and (cid:13)(cid:13)
Cent( A (cid:15)q ) − C q (cid:13)(cid:13) < (cid:15) .
Now we can consider the triangle in R d formed by C p , C q , and F ( { p, q } ) and realize by basic geometry that one of (cid:107) F ( { p, q } ) − C p (cid:107) or (cid:107) F ( { p, q } ) − C q (cid:107) must greater than or equal to (cid:107) C p − C q (cid:107) / 2 (see
Figure 1b).
Without loss of generality, let (cid:107) F ( { p, q } ) − C q (cid:107) ≥ (cid:107)
C p − C q (cid:107) / 2 .
Then, (cid:107) C p − C q (cid:107) ≤
[Equation]
{ p, q }
[Equation]
A (cid:15)q
[Equation]
(cid:124)
[Equation]
[Equation]
Cent
[Equation]
A
[Equation]
Cent( A (cid:15)q )
[Equation]
(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) <
(cid:15)
Thus, (cid:13)(cid:13)
[Equation]
A
[Equation]
[Equation]
Cent
[Equation]
Thus A (cid:15)q is the promised set which achieves the desired uniform-norm error.
Theorem 4.2 now follows easily.
proof of Theorem 4.2.
Choose p, q ∈ Ω so (cid:107) p − q
(cid:107) =
Diam(Ω)
and take τ → 0 in Lemma 4.3.
E XPERIMENTS
The proof of Lemma 4.3 not only establishes the error bound but also suggest an algorithmic approach to ﬁnding point clouds that exhibit the failure of uniform approximation.
This let’s us produce adversarial examples to the centor-of-mass problem for PointNet.
When Ω is additionally convex – e.g. the unit disk D in R 2 – it becomes fairly easy to construct many examples of A (cid:15)p and A (cid:15)q explicitly for a given PointNet model, allowing us to empirically verify the uniform-norm error lower
[Figure]
bound.
In the following experiment, we train a simple PointNet architecture to learn the center-ofmass for 10-element point clouds in D .
We train on a synthetic data set of 1 million point clouds (each element uniformly sampled from D ) labeled with their center-of-mass.
The PointNet architecture has 500K trainable parameters.
The network has the form F ( A )
=
ρ (max a ∈
A ϕ ( a )) where ϕ has 2-D input layer, 500-D hidden layer, and 500-D linear output layer, and ρ has 500-D input layer, 500-D hidden layer and 2-D linear output layer (in accordance with the Point-Cloud-UAT).
The hidden layers of ϕ and ρ are ReLU.
Since it is not possible to train with respect to the uniform-norm, we opt for the traditional L 2 loss.
In Figure 2a we can see the training curve settles after aboutepochs.
The loss at the 50th epoch however is 3-orders-of-magnitude greater than the predict worst case.
To form our adversarial examples, we pick a nonzero τ = 0 .
01 and two distinct points p, q ∈ D at random.
We set (cid:15) =
τ (cid:107) p − q (cid:107) / 4 .
We then sample D another 8 times.
We then linearly pull those 8 points towards p and towards q , sufﬁciently close so that the criteria in the proof of Lemma 4.
are satisﬁed.
This { p, q } adjoined with the 8 points pulled towards p and q form A (cid:15)p and A (cid:15)q .
We are theoretically ensured one of these two will have error larger than our bound.
In Figure 2b we plot the the produced adversarial error vs the distance between p and q that were used to make A (cid:15)p and
A (cid:15)q .
As predicted, all the adversarial errors lie above the line representing the uniform-norm error lower bound.
The failure of the perceptron model to learn the XOR function was a blow that motivated the search for new models.
When classical feed-forward neural networks began to be successful, the spectre of limited representation power loomed over the ﬁeld until the universal approximation theorem
Cybenko (1989)Hornik et al.
(1989)Leshno et al.
(1993) resolved the question.
In this work we resolved the same question for the case of two current deep learning models for point clouds, and laid out a program that could work for other models as well.
However, unlike the case of classical neural networks on compact domains of R N , the same question for point clouds has a less deﬁnitive answer even after having determined all of the uniformly approximable functions.
Each method explored here has their strengths and limitations but the presence of useful functions out-of-reach of
PointNet and DeepSets – even when considering bounded cardinality – opens the door for further research.
There are still many questions.
Will merging PointNet and DeepSets in a way to obtain greater approximation power?
Are there other useful topologies on F (Ω) for which new kinds of continuous neural networks can be constructed?
How much do these limitations matter in practice?
And ﬁnally, are there ways of developing architectures from the ground up with desirable yet different approximation capabilities on point clouds and what are they like?
A A PPENDIX
Proof of Lemma 2.1.
First we show that the r is a linear isometry.
Since X is compact for f ∈ C ( X ) there is a p ∈
X
so that | f ( p ) | = ||| f ||| X .
By density of D there is a sequence p n ∈ D that limits to p .
So
[Equation]
Next, since C ( X ) is complete, so is its isometric image r ( C ( X )) and because C b ( X ) is complete that means r ( C ( X )) is closed.
Thus,
[Equation]
where the ﬁrst subset results from continuity.
Thus N = r ( N (cid:48) )
=
r ( C ( X
[Equation]
Finally, to show r ( C ( X ))
=
U ( D ) note that every uniformly continuous function g on D continuously extends to a function on X (because D is dense in X ) placing this extension in C ( X ) and so g ∈ r ( C ( X )) .
The reverse inclusion follows as well because restriction preserves uniform continuity.
Proof of Lemma 2.2.
First we show that Max f is d H -continuous.
Let (cid:15) > 0 .
Since Ω is compact, f is uniformly continuous and so there is a δ >
so that | f ( x ) − f ( y ) | < (cid:15)/ 2 whenever d ( x, y ) < 2 δ .
Now let A, B ∈ K (Ω) and suppose d H ( A, B ) <
δ .
By deﬁnition this means A ⊆ B δ and B ⊆
A δ .
By the triangle inequality we have
[Equation]
| Max f ( A ) − Max f
( A δ ) | + | Max f
( A δ ) − Max f ( B ) |
Since A δ is compact there is a p ∈
A δ so that Max f ( A δ ) =
f
( p ) .
Observe that if q ∈ K ⊆
A δ with d ( p, q ) < 2 δ then | f ( p ) − f ( q ) | < (cid:15)/ 2 and f ( p ) =
Max f ( A δ ) ≥ Max f ( K ) ≥ f ( q ) .
This implies | Max f
( A δ ) − Max f ( K )
| < (cid:15)/ 2 .
In particular, since p ∈
A δ there is an a ∈ A such that d ( p, a ) <
δ , and since B is compact there is a b ∈ B closest to p and so,
[Equation]
Thus | Max f ( A δ ) − Max f ( A ) | and | Max f ( A δ )
− Max f ( B ) | are less than (cid:15)/ 2 and so | Max f ( A ) − Max f ( B )
| < (cid:15) as desired.
To see why Ave f,b is d W -continuous recall that the topology of d W is the same as the weak-* topology for measures and so the map µ (cid:55)→ (cid:82)
f d µ is by deﬁnition continuous whenever f ∈ C (Ω) .
[Equation]
It’s clear that Max f ◦ i K = max f .
The other identity follows from the linearity of integration.
Lastly, by composition of continuous functions it follows that PointNet and DeepSets continuously extend to K (Ω) and P (Ω) .
Since (Ω , d ) compact implies both K (Ω) and P (Ω) are compact, we can deduce that PointNet and DeepSets are uniformly continuous on F H (Ω) and F W (Ω) .