NPTC- NET : N ARROW -B AND P ARALLEL T RANSPORT C ONVOLUTIONAL N EURAL N ETWORK ON P OINT C LOUDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolution plays a crucial role in various applications in signal and image processing, analysis and recognition.
It is also the main building block of convolution neural networks (CNNs).
Designing appropriate convolution neural networks on manifold-structured point clouds can inherit and empower recent advances of
CNNs to analyzing and processing point cloud data.
However, one of the major challenges is to deﬁne a proper way to ”sweep” ﬁlters through the point cloud as a natural generalization of the planar convolution and to reﬂect the point cloud’s geometry at the same time.
In this paper, we consider generalizing convolution by adapting parallel transport on the point cloud.
Inspired by a triangulated surface based method Schonsheck et al.
(2018), we propose the Narrow-Band Parallel Transport Convolution (NPTC) using a speciﬁcally deﬁned connection on a voxelized narrow band approximation of point cloud data.
With that
, we further propose a deep convolutional neural network based on NPTC (called NPTC net) for point cloud classiﬁcation and segmentation
Comprehensive experiments show that the proposed NPTC net achieves similar or better results than current state- of
-the-art methods on point clouds classiﬁcation and segmentation
I NTRODUCTION
Convolution is one of the most widely used operators in applied mathematics, computer science and engineering.
It is also the most important building block of Convolutional Neural Netowrks (CNNs) which are the main driven force in the recent success of deep learning LeCun et al.
Goodfellow et al. (2016).
In the Euclidean space R n , the convolution of function f with a kernel (or ﬁlter) k is deﬁned as
[Equation]
This operation can be easily calculated in Euclidean spaces due to the shift-invariance of the space so that the translates of the ﬁlter k , i.e. k ( x − y ) is naturally deﬁned.
With the rapid development of data science, more and more non-Euclidean data emerged in various ﬁelds including network data from social science, 3D shapes from medical images and computer graphics, data from recommending systems, etc.
Therefore, geometric deep learning, i.e. deep learning of non-Euclidean data, is now a rapidly growing branch of deep learning Bronstein et al.
In this paper, we will discuss how we can generalize the deﬁnition of convolution to (manifold-structured) point clouds in a way that it inherits desirable properties of the planar convolution, thus it enables to design convolutional neural networks on point clouds.
One of the main challenges of deﬁning convolution on manifolds and point clouds (a discrete form of manifolds) is to deﬁne translation x − y on the non-Euclidean domain.
Other than convolutions, we also need to properly deﬁne pooling to enable networks to extract global features and to save memory during training.
Multiple types of generalization of convolutions on manifolds, graphs and point clouds have been proposed in recent years.
We shall recall some of them and discuss the relation between existing deﬁnitions of convolutions and the proposed narrow-band parallel transport convolution.
1. R ELATED W ORK OF C ONVOLUTIONS ON N ON -E UCLIDEAN D OMAINS
Spectral methods avoid the direct deﬁnition of translation x − y by utilizing the convolution theorem: for any two functions f and g , (cid:91)
f ∗ g
= ˆ f · ˆ g .
Therefore, we have f ∗ g =
( ˆ f · ˆ g ) ∨ , where ∧ and ∨ represent generalized Fourier transform and inverse Fourier transform provided through the associated Laplace-Beltrami (LB) eigensystem on manifolds.
To avoid computing convolution through full eigenvalue decomposition, polynomial approximation is proposed and yields convolution as action of polynomials of the LB operator Hammond et al.
(2011); Dong (2017).
Thus, convolutional neural networks can be designed Bruna et al.
(2014); Defferrard et al.
; Levie et al.
Spectral methods, however, suffer two major drawbacks.
First, these methods deﬁne convolution in the frequency domain.
As a result, the learned ﬁlters are not spatially localized.
Secondly, spectral convolutions are domain-dependent as deformation of the ground manifold will change the corresponding LB eigensystem.
This obstructs the use of learning networks from one training domain to a new testing domain Schonsheck et al.
Spatial mesh-based methods are more similar to the Euclidean case, and this is one of the reasons why most of the existing works fall into this category.
The philosophy behind these methods is that the tangent plane T x M of a 2 -dimensional manifold M in each point x ∈ M is embedded to a 2 -dimensional Euclidean domain where convolution can be easily deﬁned.
In this paper, we make the ﬁrst attempt to interpret some of the existing mesh-based methods in a uniﬁed framework.
We claim that most of the spatial mesh-based methods can be formulated as
[Equation]
Here, k : R 2 → R is a convolution kernel and T x,(cid:15)
M = { v ∈ T x M : (cid:104) v, v (cid:105) g x ≤ (cid:15) 2 } with (cid:15) >being the size of the kernel.
The mapping φ ( x, v ) : M × T x M → R 2 is deﬁned as
[Equation]
where (cid:126)u jx ∈ T x M , j = 1 , 2 .
For simplicity, we will denote (cid:126)u x =
( (cid:126)u 1 x , (cid:126)u 2 x ) .
Most of the designs of the existing manifold convolutions focused on the designs of (cid:126)u x .
We remark that possible singularities will lead to no convolution operation at those points.
These are isolated points on a closed manifold and for more discussion of singularities, please refer to the Appendix.
For example, GCNN Masci et al.
and ACNN Boscaini et al.
(2016) construct a local geodesic polar coordinate system on a manifold, formulating the convolution as
[Equation]
where Q is a local interpolation function with interpolation domain an isotropic disc for GCNN and an anisotropic ellipse for ACNN.
A local geodesic polar coordinate system on a manifold can also be transformed to a 2-dimensional planar coordinate system on its tangent plane.
Such transformation is the mapping ψ which is deﬁned by the inverse exponential map: v = exp − 1 ( z ( θ, r )) with z ( θ, r ) being a point in the local geodesic polar coordinate system at x ∈ M with coordinates ( θ, r ) .
With this, we can easily interpret ACNN within the framework of (2).
Indeed, ACNN essentially chooses (cid:126)u x as the directions of the principal curvature at point x .
For GCNN, on the other hand, it avoids choosing a speciﬁc vector ﬁeld on the manifold by taking max-pooling among all possible directions of (cid:126)u 1 x at each point.
Such deﬁnition of convolution, however, ignores the correspondence of the convolution kernels at different locations.
The newly proposed PTC Schonsheck et al.
(2018) deﬁnes convolution directly on the manifold, while using tangent planes to transport kernels by a properly chosen parallel transport.
PTC can be equivalently cast into the form of (2) using the inverse exponential map, and implementation of the proposed parallel transported is realized through choosing speciﬁc vector ﬁelds { (cid:126)u j x } x ∈M , j = 1 ,guided by a Eikonal equation for transforming vectors along geodesic curves on manifolds.
Spatial point-based methods have wider applications due to their weaker assumptions on the data structure, a point cloud P = { x i ∈ R 3 : i = 1 , . . .
, N } ⊂ R 3 consists of points in a 3 -dimensional
Euclidean space with the coordinates of the points as the only available information.
Manifolds can be approximated by point clouds via sampling.
Computing k -nearest neighborhoods of x or neighbors within a ﬁxed radius can easily convert a point cloud to a local graph or mesh.
This is why point cloud is simple, ﬂexible and attracts much attentions lately.
There are mainly two types of point-based convolution.
The ﬁrst type is to combine the information of points directly.
These methods can be formulated as
[Equation]
where N ( x i )
⊂ P is a neighborhood of x i and kernel k takes different forms in different methods.
PointNet Qi et al. (2017a) is an early attempt to extract features on point cloud.
PointNet is a network structure without convolution, or alternatively we can interpret the convolution deﬁned by PointNet has the simplest kernel k ( x i , x j ) =
δ ( x i , x j ) where δ is the Kronecker-Delta.
Various later works attempt to improve PointNet by choosing different forms of the kernel k .
For example, PointNet++ Qi et al.
(2017b) introduces a max pooling among local points, i.e. choosing kernel k as an indicator function:
k ( x i , x j ) =
I x j =argmax z ∈N ( xi ) f
( z ) .
Pointcnn Li et al.
(2018b) chooses k ( x i , · ) =
β (cid:48)
A x i where β ∈ R K and A x i ∈ R K × K are trainable variables with K
= |N
( x i ) | .
The second type of convolution is deﬁned by ﬁrst projecting the point cloud locally on an Euclidean domain and then employ regular convolution.
This type of methods can also be formulate as (2).
For example, Tangent convolutions Tatarchenko et al.
(2018) deﬁne kernels on the tangent plane, and use 2 principal directions of a local PCA as (cid:126)u x .
Pointconv Wu et al.
(2018) constructs local kernels by interpolation in R 3 , i.e. letting φ ( x, v ) =
x − v which is essentially a local Euclidean convolution.
1. T HE P ROPOSED C ONVOLUTION : NPTC
In this paper, we propose Narrow-Band Parallel
Transport Convolution ( NPTC ), which is a geometric convolution based on point cloud discretization of a manifold parallel transport deﬁned in a speciﬁc way.
As we observed in the previous section, convolutions in many methods can be written in the form of (2) and (3), while the differences mostly lie in the choices of the vector ﬁeld { (cid:126)u x } x ∈M .
As observed by Schonsheck et al.
(2018) that choosing the vector ﬁeld properly, the associated convolution can be interpreted as parallel transporting the kernels using the parallel transport associated to the prescribed vector ﬁeld.
We propose to deﬁne convolution on point clouds by combining narrow-band approach Adalsteinsson &
Sethian (1995) and geometric convolution.
The formal deﬁnition of parallel transport and general ideal will presented in Section 2 and detailed descriptions on NPTC will be given in Section 3.
Now, we describe how NPTC is computed on point clouds. Firstly, point clouds are approximated by voxel narrow-band with appropriate resolution. For a manifold M or point cloud P , its narrow-band is deﬁned as the region with distance from M (or P ) less than (cid:15) :
NB ( M ) :
= { x ∈ R 3 | dist ( x, M ) < (cid:15) } , where dist represents a distance function. For calculation efﬁciency and robustness to noise, we choose voxel narrow-band approximation.
It should be noted that the result of the traditional voxelization method Wu et al. (2015) is a solid 3-d structure and the voxel narrow-band can be understood as the ”shell” with a certain thickness of the former (if the point cloud is sampled from the surface of the object). It is very efﬁcient to calculate the distance function in voxel approximation with appropriate resolution. The vector ﬁeld { (cid:126)u x } x ∈M is deﬁned as the projections of the gradient ﬁeld of a narrow-band-based distance function on the approximated tangent plane of the point cloud.
Such deﬁnition of the vector ﬁeld is robust to noise, since if the distortion of the coordinates of the points by noise does not exceed the width of the narrow-band, the computed gradient ﬁeld of the distance function in the narrow-band remains unchanged.
After the vector ﬁeld { (cid:126)u x } x ∈M , we use local PCA to estimate normal vectors on the point cloud following Xu et al.
Finally, the convolution kernel can be constructed on the tangent plane and the corresponding f ( v ) can be obtained by interpolating the value f ( x ) on the point cloud.
Note that we prefer to use geometric convolution in NPTC because compared with methods that translate kernels in the ambient space of the manifold, NPTC translates kernels on the tangent planes which effectively avoids having convolution kernels deﬁned away from the underlying manifold of the point cloud.
In other words, NPTC can well reﬂect point cloud geometry and is a natural generalization of planar convolution in the sense that when the point cloud reduces to planar grids, the NPTC reduces to the planar convolution.
1. C ONTRIBUTIONS
• We introduce a new point cloud convolution, NPTC, based on parallel transport deﬁned by a narrow-band approximation of the point cloud.
The proposed NPTC is a natural generalization of planar convolution.
• The proposed NPTC combines voxelization and geometric convolution.
Voxelization with appropriate resolution brings robustness and geometric convolution can better reﬂect point cloud’s geometry.
• Based on NPTC, we designed convolutional neural networks, called NPTC-net, for point clouds classiﬁcation and segmentation with state-of-the-art performance.
B ACKGROUND AND G ENERAL I DEA
We recommend the book Nakahara (2003) for readers who are unfamiliar with differential geometry.
A short intuitive introduction to parallel transport will be provided after recalling the notion.
Let M be a 2 dimensional differential manifold embeded in R 3 and T x
M be the tangent plane at point x ∈ M .
T x M can be deﬁned as the 2 -dimensional linear space formed by the span oftangent vectors.
The disjoint union of the tangent planes at each point on the manifold deﬁnes the tangent bundle T M .
A vector ﬁeld X is a section of T M .
Collection of all smooth vector ﬁelds X ( x ) ∈ T x M , ∀ x ∈ M is denoted as Γ( T M ) .
An afﬁne connection is a bilinear mapping ∇ : Γ( T M ) × Γ( T M ) → Γ( T M ) .
A section of a vector ﬁeld is called parallel along a curve γ if ∇ ˙ γ X = 0 .
Suppose we are given an vector (cid:126)e ∈ T x 0 at x 0 = γ (0) ∈ M .
The parallel transport of (cid:126)e along γ is the extension of (cid:126)e to a parallel section X on γ .
[-= 
-]
Intuitively, in differential geometry parallel transport is a way of translating a vector along a smooth curve ”parallel” to the original.
In fact, by given smooth vector ﬁelds { (cid:126)u 1 , (cid:126)u 2 } , we can deﬁne linear transformation among tangent planes then the corresponding parallel transport can be induced
Knebelman (1951)
Therefore, parallel transport deﬁned in this way can be viewed as parallel transporting the vector on the manifold with respect to the connection that is reconstructed from the vector ﬁeld { (cid:126)u 1 x } x ∈M .
That is to say, if given a vector ﬁeld, we can deﬁne a way to ”transport” vectors between tangent planes.
To select a suitable vector ﬁeld, we ﬁrst recall the choice of the vector ﬁelds of PTC which deﬁnes convolution on triangulated surfaces via parallel transport with respect to the Levi-Civita connection
Schonsheck et al.
Geodesic curve represents, in some sense, the ”straightest” path between two points on a Riemannian manifold.
Given a geodesic connecting two points x and y , the tangential direction at x corresponds to the ascent direction of geodesic distance from y .
That is to say, all tangential directions of one geodesic form a parallel vector ﬁeld.
PTC chooses such direction as (cid:126)u 1 x and deﬁnes (cid:126)u 2 x = (cid:126)u 1 x × (cid:126)n x with (cid:126)n x the normal vector at x .
One naive approach to extend mesh based methods to point cloud is to generate a triangulated surface based on the point cloud.
However, this is not as convenient as working directly with the point cloud since in practice not every point cloud corresponds to a legitimate parameterized surface and pooling is not as easy to implement on triangulated surfaces as on point clouds.
In addition, it is time-consuming to construct meshes on point clouds.
When applied in practice, mesh construction time may be much longer than inference time of some methods directly applied on the point cloud.
If we want to construct a vector ﬁeld on point cloud, gradients of some sort of distance function can be a good choice.
However, unlike triangulated surfaces, distance function is not easily deﬁned on point clouds due to the lack of connectivity.
It is then natural to approximate the point cloud with another data structure with connectivity, so that distance function can be easily calculated.
We use voxel to approximate the point cloud in a narrow-band in R 3 covering the point cloud.
We denote such distance function as ρ : R 3 → R + .
Then, we review how a distance function is calculated. Distance functions can be easily computed by solving the Eikonal equation. The Eikonal equation is a non-linear partial differential equation describing wave propagation: [Equation]
where Λ ⊂ Ω ⊂ R n and h ( x ) is a strictly positive function.
The solution ρ ( x ) of (5) can be viewed as the shortest time needed to travel from x to Λ with h ( x ) being the speed of the wave at x . For the special case when h = 1 , the solution ρ ( x ) represents the distance from x to Λ limited in the Ω . The Eikonal equation can be solved by the fast marching method Sethian (1996). Note that if the point cloud is sampled from a plane, the narrow-band is ﬂat as well.
Then, by a proper choice of the distance function, the vector ﬁelds { (cid:126)u j x } x ∈M , j = 1 , 2 , can be reduced to the global coordinate { (cid:126)e j } , j = 1 , 2 on the plane.
This means that NPTC is reduced to the traditional planar convolution.
Once the distance function ρ is computed, we choose (cid:126)u 1 x = ∇ P ρ ( x ) , where ∇ P ρ ( x ) is a projection of ∇ ρ ( x ) on an approximated tangent plane at x .
Then, (cid:126)u 2 x can be calculated by the outer product (cid:126)u
[Equation] = [Equation]
x with (cid:126)n x the normal vector at x .
For convenience, instead of transporting the kernel on the manifold, we can locally construct the parallel transported kernel at every point x by formulating k as k ( φ ( x, · )) , where φ ( x, v ) is deﬁned in (3).
It is known in differential geometry that transporting the kernel to every point on the manifold is the same as locally reconstructing the kernel in the aforementioned way.  N ARROW - BAND P ARALLEL T RANSPORT C ONVOLUTION (NPTC) AND N ETWORK D ESIGN 3. I MPLEMENTATION OF NPTC
Note that some methods of voxelization, interpolation ,and initial point selection are not essential.
These can be replaced depending on the accuracy, speed requirements and datasets.
[Figure]
Assuming that the point cloud has been normalized so that it is enclosed by the unit cube. The unit cube is then discretized by N × N × N grids.
The narrow-band approximation of the point cloud is constructed by including each voxel with a distance less than kN from any point on the point cloud. Then the voxel narrow-band approximation of point cloud has been constructed.
N depends on the requirements of speed and accuracy. k need to be large enough to make the narrow-band continuous (if the point cloud represents a continuous manifold) and small enough to make the distance in narrow-band approximate the geodesic distance. In the next experiments, we choose N as 100, k as 2 and L ∞ as distance function.
If the sampling of point cloud is very sparse, it may be necessary to construct a narrow-band by the estimation of the normal or some other strategies. Then we can compute distance functions on the voxels using the well-known fast marching method based on regular grid provided by the voxelization Sethian (1996).
The solution ρ ( x ) of the Eikonal equation |∇ ρ ( x ) | = 1 presents the distance form Λ to x limited inside the narrow-band.
Here Λ is chosen as certain point on the point cloud.
3.1. C OMPUTING THE V ECTOR F IELDS ON P OINT C LOUDS
We ﬁrst compute the tangent plane on each point.
Tangent planes are important features of manifolds and have been well-studied in the literature Lai et al.
In this paper, we use local principal component analysis (LPCA) to estimate the tangent plane.
We estimate the local linear structure near x ∈ P using the covariance matrix
[Equation]
where N ( x ) is the set of neighboring points of x .
The eigenvectors of the covariance matrix form an orthogonal basis.
If the point cloud is sampled from a two dimensional manifold, and the local sampling is dense enough to resolve local features, the eigenvectors corresponding to the largest two eigenvalues provide the two orthogonal directions of the tangent plane, and the remaining vector represents the normal direction at x ∈ P .
Here, we denote the space spanned by the two eigenvectors of the covariance matrix at x as T x P ⊂ R 3 .
With the computed distance function ρ ( x ) , it is natural to deﬁne the vector ﬁeld by projecting ∇ ρ on the approximated tangent planes of the point cloud.
Given a point x k ∈ P close enough to x ∈ P , we have
[Equation]
where ρ ( x k ) and ρ ( x ) are known.
If we consider k -nearest neighbors of x , we have k − 1 equations with 3 unknowns that are the three components of ∇ ρ .
We can use least squares to ﬁnd ∇ ρ .
We then project the vector ∇ ρ ( x ) onto the tangent plane at x .
We denote the projected vector ∇ P ρ , which is the vector we eventually need to deﬁne NPTC. After deﬁning the vector ﬁeld on the tangent plane, we choose the convolution kernel of hexagon (Figure 1), that is, a convolution kernel is composed of a central point and six evenly distributed points around. The value f ( v ) is computed by f ( v ) = f ( z ) where z ∈ P is the closest point to v in R 3 .
Note that one may use a more sophisticated method to compute f ( v ) rather than using the closest point interpolation.
We choose the closest point interpolation because of its simplicity.
3. NPTC- NET : A RCHITECTURE D ESIGN FOR C LASSIFICATION AND S EGMENTATION
This section, we present how to use NPTC to design convolutional neural networks on point clouds for classiﬁcation and segmentation tasks.
For that, other than the NPTC, we need to deﬁne some other operations that are frequently used in neural networks.
Note that, some point-wise operations like MLP and ReLu are the same on point cloud as the Euclidean case.
Here, we only focus on the operations that are not readily deﬁned on point clouds.
Down-sampling:
In our implementation, the sub-sampled set of points of the next layer is generated by the farthest point sampling Eldar et al.
Convolution layer: Our k -th convolution layer takes points P k ∈ R N k × 3 and their corresponding feature maps F k ∈ R N k × c k as input, where N k is the number of the points and c k is the number of channels at layer k .
The corresponding output is F k +1 ∈ R N k
+
×
c k +1 living on the points
P k +1 ∈ R N k +1 × 3 .
The NPTC-net have encoding and decoding stages.
Normally, N k +1 < N k during encoding and N k < N k +1 during decoding.
Convolution at the k -th layer is only performed on the point set P k +1 , which resembles convolution with stride > 1 for planar convolutions.
Residual block: One residual block takes the feature maps F ∈ R N × c on the point set P ∈ R N ×as input and same number of points and same number of channels of features as output.
One residual layer consists of three components: MLP from c channels to c 2 channels, convolution layer from cchannels to c 2 channels, MLP from c 2 channels to c channels plus the feature maps from the bypass connection.
A residual block consists of several residual layers.
NPTC-net consists of the aforementioned operations and its architecture is given by Figure 3.
The left half of the NPTC-net is the encoder part of the network for feature extraction.
For classiﬁcation, features at the bottom of the network are directly attached to a classiﬁcation network; while for segmentation, features are decoded using the right half of the NPTC-net (decoder part of the network) to output the segmentation map.
E XPERIMENTS
In order to evaluate our new NPTC-nets, we conduct experiments on three widely used 3D datasets,
ModelNet Wu et al.
ShapeNet Yi et al.
S3DIS Armeni et al.
We implement the model on a GTX TITAN Xp GPU.
For more implementation details, please refer to the Appendix.
4. 3D S HAPE C LASSIFICATION AND S EGMENTATION
We test the NPTC-net on ModelNet40 for classiﬁcation tasks.
ModelNet40 contains 12,311 CAD models from 40 categories with 9,842 samples for training and 2,468 samples for testing.
For comparison, we use the data provided by Qi et al.
(2017a)
sampling 2,048 points uniformly and computing the normal vectors from the mesh.
As shown on Tabel 1, our networks outperform other state-of-art methods.
(If a compared method has results on both 2048 (or 1024) and 5000 points, we only compare with the former.).
We also evaluate the NPTC-net on ShapeNet Part for segmentation tasks.
ShapeNet Part contains 16,680 models from 16 shape categories with 14,006 for training and 2,874 for testing, each annotated with 2 to 6 parts and there are 50 different parts in total.
We follow the experiment setup of previous works, putting object category into networks as known information.
We use point intersection-over-union (IoU) to evaluate our NPTC-net.
Table 1 shows that our model ranks second on this dataset and is fairly close to the best known result.
[Table]
4. R UBOSTNESS TEST
As pointed out earlier [Table] using voxelization can bring rubustness to the deﬁnition of geometric convolution.
In order to show the robustness of our model for real data [Table] we tested scene semantic segmentation on the ”Stanford Large-Scale 3D Indoor Spaces Dataset” (S3DIS).
S3DIS covers six large-scale indoor areas from 3 different buildings for a total of 273 million points annotated with 13 classes.
This is a real-word scanned dataset without normal and with noise.
Following Tchapmi et al.
(2017), we advocate the use of Area-5 as test scene to better measure the generalization ability of our method.
Table 2 shows that geometric convolution methods are close to the methods which do not need normal estimation.
NPTC-net outperforms the best known geometric convolution method
TangentConvXu et al.
[Table]
C ONCLUSION
This paper proposed a new way of deﬁning convolution on point clouds, called the narrow-band parallel transport convolution (NPTC), based on a point cloud discretization of a manifold parallel transport.
The parallel transport was deﬁned speciﬁcally by a vector ﬁeld generated by the gradient ﬁeld of a distance function on a narrow-band approximation of the point cloud.
The NPTC was used to design a convolutional neural network (NPTC-net) for point cloud classiﬁcation and segmentation.
Comparisons with state-of-the-art methods indicated that the proposed NPTC-net is competitive with the best existing methods.
A A PPENDIX A. I MPLEMENTATION D ETAILS
We implement the model with Tensorﬂow using SGD optimizer with an initial learning rate 0 .
1 for
ModelNet40 and ADAM optimizer with an initial learning rate 0 .
002 for ShapeNet Part and S3DIS on a GTX TITAN Xp GPU.
During the training procedure of classiﬁcation, the data is augmented by random rotation, scaling and Gaussian perturbation on the coordinates of the points.
During the training procedure of segmentation, the data is augmented by scaling and Gaussian perturbation on the coordinates of the points.
Following Li et al.
(2018b), the data of S3DIS is ﬁrstly split by room, and then the rooms are sliced into 1.5m by 1.5m blocks, with 0.3m padding on each side.
The points in the padding areas serve as context of the internal points, and themselves are not linked to loss in the training phase, nor used for prediction in the testing phase.
Each block will be viewed as a point cloud during training and testing.
We do inferences on the augmented data during testing by voting following Qi et al.
(2017b).
To clarify this, let x i be one data from the test set and x ij be the augmented data generated from x i .
p ij = f
( x ij ) denotes the predict possibility vector after Softmax.
The ﬁnal inference result of x i is
Predict ( x i ) =
arg max k
[Equation]
A. R UNNING S TATISTICS
[Table] [Table]
As shown in Table
[Table]
we summarize our running statistics based with model for ModelNet40 with batch size 16.
In comparison with several other methods, although we use ResNet structure, the fewer channels, smaller kernels and simpler interpolation (nearest neighboring) make NPTC use similiar parameters and even fewer FLOPs.
Total pre-processing time mainly depends on the grid’s density.
For most cases, the resolution of 100 3 is enough to describe the shape of the point cloud, which is what we chose for our experiments.
We also remark that the whole computation cost of constructing convolution is linearly depending on the resolution of voxel and the size of data.
On a PC with Core i7-7700 CPU, the pre-processing takes about 0.5 seconds per point cloud with Matlab, and the whole dataset of ModelNet40 (12, shapes with 10,000 points each) takes only about 1.5 hours.
It is negligible compared to the training of the deep neural networks and acceptable to do inference in practice. A. V ISUALIZATION
[Figure]
To visualize the effects of the proposed NPTC in the NPTC-net, we trained the network on ShapeNet
Part and visualize learned features by coloring the points according to their level of activation.
In
Figure 4, ﬁlters from the the ﬁrst Convolution layer in the the ﬁrst Residual block and ﬁnal Convolution layer in the second Residual block are chosen.
In order to easily compare the features at different levels, we interpolate them on the input point cloud.
Observe that low-level features mostly represent simple structures like edges (top of (a)) and planes (bottom of (a)) with low variation in their magnitudes.
In deeper layers, features are richer and more distinct from each other, like bottleneck (upper b ”big A. S INGULARITIES [Figure]
depend on the distance to the origin at each point: (cid:126)v x = d ( y 1 , x ) (cid:126)v 1 x − d ( y 2 , x ) (cid:126)v 2 x .
{ (cid:126)v i } represent the vector ﬁeld with the origin y i and d ( y i , x ) represent the geodesic distance. The absolute value of combination coefﬁcient for one vector ﬁeld will descend to 0 as x approaches the origin to prevent the singularity at the origin. We note that on non-parallelizable manifolds such as the sphere, it is not possible to construct a smooth vector ﬁeld. Helgason (1979)
So that the vector ﬁeld must have at least one a singularity, and the frame around the singularity will rotate violently.
Figure 5 shows vector ﬁelds deﬁned on the surface with and without singularity.
If singularity can not be avoided.
It is natural to ask how much singularities affect the result.
This has been discussed in section 4.4 of PTC.
Schonsheck et al.
In surface MNIST experiment, convolutions deﬁned by taking the middle point of a surface as the origin of the distance function (which means that there is a singularity in the middle like Figure 5 (b)) will reach the accuracy of 94.92% and convolutions using the smooth strategy will reach 96.36%.
Singularity does cause some problems, but we think it’s in the acceptable range.
In our experience, activation in segmentation task have been visualized as before, we ﬁnd no direct relationship with the location of the singularity.
If we have some prior knowledge about a shape that can have no singularity.
Strategies can be applied to get a smooth vector ﬁeld or reduce the inﬂuence of singularity.
For example, if the shape is essentially like a wrinkled sheet, then we can choose the starting point(s) Λ of the Eikonal equation as points on one edge of the sheet or the linear combination of two vector ﬁelds with different origins
Fig 5 (c).
Most of the shapes in the three experiment datasets are closed manifolds like boxes, beds, airplanes, on which singularity can not be avoided.
Experiments show that applying more strategies we know can not improve the results obviously, so we choose to select one initial point directly.