P RE -T RAINING BY C OMPLETING P OINT C LOUDS
Anonymous authors
Paper under double-blind review
A BSTRACT
There has recently been a ﬂurry of exciting advances in deep learning models on point clouds.
However, these advances have been hampered by the difﬁculty of creating labelled point cloud datasets: sparse point clouds often have unclear label identities for certain points, while dense point clouds are time-consuming to annotate.
Inspired by mask-based pre-training in the natural language processing community, we propose a novel pre-training mechanism for point clouds.
It works by masking occluded points that result from observations at different camera views.
It then optimizes a completion model that learns how to reconstruct the occluded points, given the partial point cloud.
In this way, our method learns a pre-trained representation that can identify the visual constraints inherently embedded in real-world point clouds.
We call our method Occlusion Completion (OcCo).
We demonstrate that OcCo learns representations that improve the semantic understandings as well as generalization on downstream tasks over prior methods, transfer to different datasets, reduce training time and improve label efﬁciency.
I NTRODUCTION
Point clouds are a natural representation of 3D objects.
Recently, there has been a ﬂurry of exciting new point cloud models in areas such as segmentation (Landrieu & Simonovsky, 2018; Yang et al., 2019a;
Hu et al., 2020a) and object detection (Zhou & Tuzel, 2018; Lang et al., 2019; Wang et al., 2020b).
Current 3D sensing modalities ( i.e. , 3D scanners, stereo cameras, lidars) have enabled the creation of large repositories of point cloud data (Rusu & Cousins, 2011; Hackel et al., 2017).
However, annotating point clouds is challenging as: (1) Point cloud data can be sparse and at low resolutions, making the identity of points ambiguous; (2) Datasets that are not sparse can easily reach hundreds of millions of points (e.g., small dense point clouds for object classiﬁcation (Zhou & Neumann, 2013) and large vast point clouds for 3D reconstruction (Zolanvari et al., 2019)); (3) Labelling individual points or drawing 3D bounding boxes are both more complex and timeconsuming compared with annotating 2D images (Wang et al., 2019a).
Since most methods require dense supervision, the lack of annotated point cloud data impedes the development of novel models.
On the other hand, because of the rapid development of 3D sensors, unlabelled point cloud datasets are abundant.
Recent work has developed unsupervised pre-training methods to learn initialization for point cloud models.
These are based on designing novel generative adversarial networks (GANs) (Wu et al., 2016; Han et al., 2019; Achlioptas et al., 2018) and autoencoders (Hassani & Haley, 2019; Li et al., 2018a; Yang et al., 2018).
However, completely unsupervised pre-training methods have been recently outperformed by the self-supervised pre-training techniques of (Sauder & Sievers, 2019) and (Alliegro et al., 2020).
Both methods work by ﬁrst voxelizing point clouds, then splitting each axis into k parts, yielding k 3 voxels.
Then, voxels are randomly permuted, and a model is trained to rearrange the permuted voxels back to their original positions.
The intuition is that such a model learns the spatial conﬁguration of objects and scenes.
However, such random permutation destroys all spatial information that the model could have used to predict the ﬁnal object point cloud.
Our insight is that partial point-cloud masking is a good candidate for pre-training in point-clouds because of two reasons: (1) The pre-trained model requires spatial and semantic understanding of the input point clouds to be able to reconstruct masked shapes.
(2) Mask-based completion tasks have become the de facto standard for learning pre-trained representations in natural language processing (NLP)
(Mikolov et al., 2013; Devlin et al., 2018; Peters et al., 2018).
Different from random permutations, masking respects the spatial constraints that are naturally encoded in point clouds of real-world objects and scenes.
Given this insight, we propose Occlusion Completion (OcCo) [Table] [Table] [Table]
=</latexit> a novel self-supervised pre-training method that consists of (a) a mechanism to generate occluded point clouds, and (b) a completion task to reconstruct the occluded point cloud.
Speciﬁcally, in (a) point clouds are generated by determining what part of objects would be occluded if the underlying object was observed from a particular view-point.
In fact, many point clouds generated from a ﬁxed 3D sensor will have occlusions exactly like this.
Given an occluded point cloud, the goal of the completion task (b) is to learn a model that accurately reconstructs the missing parts of the point cloud.
For a model to perform this task well, it needs to learn to encode localized structural information, based on the context and geometry of partial objects.
This is something that is useful for any point cloud model to know, even if used only for classiﬁcation or segmentation.
We demonstrate that the weights learned by our pre-training method on a single unsupervised dataset can be used as initialization for models in downstream tasks (e.g., object classiﬁcation, part and semantic segmentation) to improve them, even on completely different datasets .
Speciﬁcally our pre-training technique: (i) leads to improved generalization over prior baselines on the downstream tasks of object classiﬁcation, object part and scene semantic segmentation; (ii) speeds up model convergence, in some cases, by up to 5 × ; (iii) maintains improvements as the size of the labelled downstream dataset decreases; (iv) can be used for a variety of state-of-the-art point cloud models.
O CCLUSION C OMPLETION
We now introduce Occlusion Completion (OcCo).
Our approach is shown in Figure 1.
Our main insight is that by continually occluding point clouds and learning a model c ( · ) to complete them, the weights of the completion model can be used as initialization for downstream tasks (e.g., classiﬁcation, segmentation) , speeding up training and improving generalization over other initialization techniques.
Throughout we assume point clouds P are sets of points in 3D Euclidean space, P = { p 1 , p 2 , ..., p n } , where each point p i is a vector of coordinates ( x i , y i , z i ) and features (e.g. color and normal).
We begin by describing the components that make up our occlusion mapping o ( · ) .
Then we detail how to learn a completion model c ( · ) , giving pseudocode and the architectural details in appendix. Finally we discuss the criteria on validating the effectiveness of a pre-training model for 3D point clouds.
2. G ENERATING O CCLUSIONS
We ﬁrst describe a randomized occlusion mapping o : P → P (where P is the space of all point clouds) from a full point cloud P to an occluded point cloud ˜ P .
We will do so by determining which points are occluded when the point cloud is viewed from a particular camera position.
This requires three steps: (1) A projection of the point cloud (in a world reference frame) into the coordinates of a camera reference frame; (2) Determining which points are occluded based on the camera view-points; (3) Mapping the points back from the camera reference frame to the world reference frame.
Viewing the point cloud from a camera.
A camera deﬁnes a projection from a 3D world reference frame into a distinctive 3D camera reference frame.
It does so by specifying a camera model and a camera view-point from which the projection occurs.
The simplest camera model is the pinhole camera, and view-point projection for it is given by a simple linear equation: (cid:34)
x cam y cam z cam (cid:35) = (cid:34) f γ w/ f h/    (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) intrinsic [ K ] (cid:34) rrrtrrrtrrrt(cid:35)
(cid:124) (cid:123)(cid:122) (cid:125) rotation | translation [ R | t ]   xyz   . In the above, ( x, y, z ) are the original point cloud coordinates, the matrix including r and t entries is the concatenation of a 3D rotation matrix with a 3D translation vector, and the ﬁnal matrix to the left is the camera intrinsic matrix ( f speciﬁes the camera focal length, γ is the skewness between the x and y axes in the camera, and w, h are the width and height of the camera image).
Given these, the ﬁnal coordinates ( x cam , y cam , z cam ) are the positions of the point in the camera reference frame.
We will refer to the intrinsic matrix as K and the rotation/translation matrix as [ R | t ] .
Determining occluded points.
We can think of the point ( x cam , y cam , z cam ) in multiple ways: (a) as a 3D point in the camera reference frame, (b) as a 2D pixel with coordinates ( fx cam /z
cam , fy cam /z
cam ) with a depth of z cam .
In this way, some 2D points resulting from the projection may be occluded by others if they have the same pixel coordinates, but appear at a larger depth.
To determine which points are occluded, we ﬁrst use Delaunay triangulation to reconstruct a polygon mesh from the points, and remove the points which belong to the hidden surfaces that are determined via z-buffering .
Mapping back from camera frame to world frame.
Once occluded points are removed, we reproject the point cloud to the original world reference frame, via the following linear transformation:
[Equation]
[-(cid:123)(cid:122) (cid:125) (cid:2) R | t 
 | 1 (cid:3) (cid:62) -] 
[Equation]

[Equation]
Our randomized occlusion mapping o ( · ) is constructed as follows.
Fix an initial point cloud P .
Given a camera intrinsics matrix K , sample rotation/translation matrices [[ R 1 | t 1 ] , . . .
, [ R V | t V ]] , where V is the number of views.
For each view v ∈
[ V ] , project P into the camera frame of that view-point using eq.
(1), ﬁnd occluded points and remove them, then map the rest back to the world reference using eq.
This yields the ﬁnal occluded world frame point cloud for view-point v : ˜
P v .
2. T HE C OMPLETION T ASK
Given an occluded point cloud ˜ P produced by o ( · ) , the goal of the completion task is to learn a completion mapping c : P → P from ˜ P to a completed point cloud P .
We say that a completion mapping is accurate w.r.t.
loss (cid:96) ( · , · ) if E ˜ P∼ o ( P ) (cid:96) ( c ( ˜ P ) , P ) → 0 .
The structure of the completion model c ( · ) is an “encoder-decoder” network (Dai et al., 2017b; Yuan et al., 2018; Tchapmi et al., 2019; Wang et al., 2020a).
The encoder maps an occluded point cloud to a vector, and the decoder reconstructs the full shape.
After pre-training, the encoder weights can be used as initialization for downstream tasks.
In appendix we gives pseudocode for OcCo and describes the architectures.
E XPERIMENTS 3. P RE -T RAINING AND D OWNSTREAM T RAINING D ETAILS
We evaluate how OcCo improves the learning and generalization of a number of classiﬁcation and segmentation tasks. Here we describe the details of training in each setting.
OcCo pre-training.
For all experiments, we will use a single pre-training dataset based on ModelNet40 (Wu et al., 2015).
It includes 12,311 synthesized objects from 40 object categories, divided into 9,843 training objects and 2,468 testing objects.
To construct the pre-training dataset, we generate occluded point clouds based on the training objects with a ﬁxed camera intrinsics { f = 1000 , γ = 0 , ω = 1600 , h = 1200 } , 10 random selected viewpoints and zero translation.
Figure 2 shows examples of the resulting occluded point clouds.
Given these, we train an “encoderdecoder” style completion model c ( · ) .
For encoders, similar to prior completion models (Tchapmi et al., 2019; Wang et al., 2020a; Wen et al., 2020), we consider PointNet (Qi et al., 2017a), PCN (Yuan et al., 2018) and DGCNN (Wang et al., 2019b).
These networks encode an occluded point cloud into a 1024-dimensional vector.
We adapted the folding-based decoder from (Yuan et al., 2018) to complete the point clouds in a two-stage procedure.
We use the Chamfer Distance (CD) as our loss function (cid:96) ( · , · ) .
We use Adam (Kingma & Ba, 2015) with an initial learning rate of 1e-4, decayed by 0.7 every 10 epochs to a minimum value of 1e-6, for a total of 50 epochs.
We use a batch size ofand set the momentum in the batch normalisation to be 0.9.
Few-shot learning. We use ModelNet40 and Syndey10 (De Deuge et al., 2013) for “ K -way Nshot” learning. During training, K classes are randomly selected and for each class we sample N random samples, then the model is tested on the same K classes . As in Sharma & Kaul (2020), we represent each object with 100 points.
We use the same training settings as used in the next paragraph.
Object classiﬁcation.
We use three 3D object recognition benchmarks: ModelNet40, ScanNet10 (Qin et al., 2019) and ScanObjectNN (Uy et al., 2019); we describe them in the appendix.
All objects are represented with 1024 points.
We use the same training settings as the original works.
Concretely, for PCN and PointNet, we use the Adam optimizer with an initial learning rate 1e-3, decayed by 0.7 every 20 epochs to a minimum value of 1e-5. For DGCNN, we use the SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4. The learning rate starts from 0.1 and then reduces using cosine annealing Loshchilov & Hutter (2017) with a minimum value of 1e-3. We use dropout Srivastava et al.
(2014) in the fully connected layers before the softmax output layer. The dropout rate of PointNet and PCN is set to 0.7, and is 0.5 for DGCNN. For all three models, we train them for 200 epochs with a batch size of 32.
We report the results based on three runs.
Part segmentation.
We use the ShapeNetPart (Armeni et al., 2016) benchmark for object part segmentation.
This dataset contains 16,881 objects from 16 categories, and has 50 parts in total.
Each object is represented with 2048 points, and we use the same training settings as the original work.
Semantic segmentation.
We use the S3DIS benchmark (Armeni et al., 2016) for semantic indoor scene segmentation.
It contains 3D scans collected via Matterport scanners in 6 different places, encompassing 271 rooms.
Each point, described by a 9-dimensional vector (including coordinates, RGB values and normalised location), is labeled as one of 13 semantic categories (e.g. chair, table and ﬂoor).
We use the same preprocessing procedures and training settings as the original work.
3. W HAT IS L EARNED FROM O C C O P RE -T RAINING
Alongside OcCo’s ability to improve learning tasks we analyze the properties of the pre-trained representation itself. Here we describe the approaches we use for such analyses. Visualisation of learned features. Feature visualisation (Olah et al., 2017) is widely used to qualitatively understand the role of a convolutional neural network unit.
It links highly activated parts of a CNN channel with human concepts which have semantic meaning.
Ideally the pre-training process learns disentangled features that are useful to distinguish different parts of an object or a scene.
These learned features will be beneﬁcial to not only few-shot learning, but also object recognition and part and scene segmentation.
Detection of semantic concepts.
To quantitatively analyse the learned features of pre-training, we adapt network dissection (Bau et al., 2017; 2020) to determine the number of concept detectors in a pre-trained point cloud feature encoder.
Speciﬁcally, for the k -th channel, we ﬁrst create a binary activation mask M k based on highly activated point subsets.
Since point cloud encoders usually learn each point feature either independently or via neighborhood aggregation, the feature maps usually do not change in the vertical direction (see Figure 3).
Therefore we can skip the retrieval step and directly quantify the alignment between an activation mask M k and the n -th concept mask C n (i.e., object parts) via mean intersection of union (mIoU) over a collection of point clouds D P :
[Equation]
where | · | is the set cardinality.
mIoU ( k,n ) can be interpreted as how well unit k detects concept c .
Structural invariance/equivariance under SO(3) transformation.
Pre-training should learn a representation that is robust under under rigid SO(3) transformations (i.e., rotation, translation, permutation).
Although a single representation might vary after transformation, the cluster structures should be preserved.
We use adjusted mutual information (AMI) (Nguyen et al., 2009) based on the clustering Ω and the ground truth label C , which prevents the score from monotonically increasing when the number of clusters increases,
[Equation]
where Ω is the clustering determined by the learned embeddings Enc( · ) and unsupervised clustering methods such as K-means.
I
[Equation]
k (cid:80) j P ( w k ∩ c j ) log P ( w k ∩ c j ) P ( w k ) P ( c j ) denotes the mutual information, H( · ) is the entropy.
AMI has a maximal of 1 when two partitions are identical, and reaches a minimal of 0 if two clusters are total uncorrelated.
It is calculated as:
[Equation]
Once we ﬁnish the pre-training on ModelNet40, we ﬁrst analyse the learned features and embeddings of the OcCo PointNet via the tests or probes described above.
Speciﬁcally, we examine the learned concepts of the pre-trained encoders on ShapeNetPart.
We assign activation mask M k with points that have top 20% highest values in the k -th unit of the feature, and the n -th concept mask C n is derived from the ground-truth annotations of the n -th object parts.
We ignore the object parts which have less than 100 points.
We call k -th channel a detector of concept n when mIoU ( k,n ) > 0 . 5 .
We analyze the learned embeddings of the pre-trained encoders on ShapeNet10 and ScanObjectNN.
We cluster the learned embeddings Enc( P ) into ω with K-means and calculate the AMI w.r.t labels.
Since the encoders are permutation invariant, here we consider rotation, translation and jittering.
3. C OMPLETION R ESULTS AND P ROBE T ESTS
Visualisation of learned features. In Figure 3, we ﬁrst visualize the features learned by OcCo PointNet on the objects from test split of ModelNet40. We visualize each learned feature by
[Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Figure] coloring the points according to their channel values. We ﬁnd that, in early stage the encoder is able to learn low-level geometric primitives, i.e., planes, cylinders and cones, while later the network recognises more complex shapes like wings, leafs and upper bodies.
We use t-SNE on the embeddings of OcCo encoders based on ShapeNet10, distinguishable clusters are formed for different categories. Number of concept detectors.
In Figure 4, we sketch the number of detected parts based on random, Jigsaw and OcCo (trained for 10 epochs and 50 epochs)-initialised PointNet.
We ﬁnd that, while keeping the previously learned concepts, OcCo helps the encoder progressively detect more object parts as the training proceeds.
We show that OcCo pre-training have outperformed prior methods in terms of total detected parts. We provide qualitative visualisation results in the appendix. [Table] Invariance/Equivariance under SO(3) transformation.
We compare Jigsaw and OcCoinitialised PointNet encoder with two hand-crafted point cloud global descriptors: viewpoint feature histogram (VFH)
(Rusu et al., 2010) and M2DP (He et al., 2016) in Table 1.
Each point cloud is represented as a vector, and we use K-means for clustering, where K is set as the number of categories.
We show that OcCo pre-training helps the networks to learn better embeddings of point cloud objects, especially when they are occluded and with outlier points (ScanObjectNN).
3. F EW -S HOT L EARNING
We use the same setting and train/test split as (Sharma & Kaul, 2020)
(cTree), and report the mean and standard deviation across on 10 runs.
The top half of the table reports results for eight randomly initialized point cloud models, while the bottom-half reports results on two models across three pre-training methods.
We bold the best results (and those whose standard deviation overlaps the mean of the best result).
It is worth mentioning (Sharma & Kaul, 2020) pre-trained the encoders on both datasets before ﬁne tuning, while we only pre-trained once on ModelNet40.
The results show that models pre-trained with OcCo either outperform or have standard deviations that overlap with the best method in 7 out of 8 settings.
[Table]
3. O BJECT C LASSIFICATION R ESULTS
We now compare OcCo against prior initialization approaches on object classiﬁcation tasks.
Tablecompares OcCo-initialization to random (Rand) and (Sauder & Sievers, 2019)’s (Jigsaw) initialization on various object classiﬁcation datasets among different encoders.
“MN40”, “ScN10” and “SO15” stand for ModelNet40, ScanNet10 and ScanObjectNN respectively.
Recall that OcCo-initialization is pre-trained only on occlusions generated from the train split of ModelNet40.
We color blue the best results for each encoder and bold in black the overall best result (and those whose standard deviation overlaps the mean of the best result) for each dataset.
We show that OcCo-initialized models outperform all baselines.
These results demonstrate that the OcCo-initialized models have strong transfer capabilities on out-of-domain datasets.
We make more comparisons in the appendix.
[Table]
3. O BJECT P ART S EGMENTATION R ESULTS
Table 4 compares OcCo-initialization to random and (Sauder & Sievers, 2019)’s (Jigsaw) initialization on object part segmentation task.
We show that OcCo-initialized models outperform or match others in terms of accuracy and IoU in all three encoders, demonstrating representations derived from completing occluded ModelNet40 improves the performance of part segmentation
[Table]
[Table]
S EMANTIC S EGMENTATION
Here we compare random, Jigsaw and OcCo initialization on semantic segmentation task.
We follow the same design of PointNet and DGCNN, use a k -fold train-test procedure as in (Armeni et al., 2016).
The results are reported in Table 5.
OcCo-initialized models outperform random and jigsawinitialized ones, demonstrating that the pre-trained representations derived from completing occluded
ModelNet40 brings improvements on segmenting indoor scenes which consist of occluded objects.
not achieve [Figure] , PointNet
[Table]
[Table] ScanObjectNN (OBJ_ [Table] ), DGCNN
[Table]
[Figure]
[Table]
[Table] [Table]
L EARNING C URVES
We plot the learning curves for classiﬁcation and segmentation tasks in Figure 5.
We observe that the models with OcCo initialization converge faster to better test accuracy than the random and sometimes
Jigsaw-initialized models.
For example, on ModelNet40 with a PCN encoder, the OcCo-initialized model takes around 10 epochs to converge, while the randomly initialized model takes aroundepochs.
Similarly, for ScanObjectNN with DGCNN encoder, the OcCo-initialized model converges around 20 epochs and to a better test accuracy than the random and Jigsaw-initialized model.
R ELATED W ORK 4.
[Equation]
LOUDS
Work on deep models for point clouds can largely be divided into three different structural approaches: (a) pointwise-based networks , (b) convolution-based networks , and (c) graph-based networks .
We call the networks that independently process each point, before aggregating these point representations: pointwise-based networks (Qi et al., 2017a;b; Joseph-Rivlin et al., 2019; Duan et al., 2019; Zhao et al., 2019; Yang et al., 2019c; Lin et al., 2019).
One well-known method, PointNet, devises a novel neural network that is designed to respect the permutation invariance of point clouds.
Each point is independently fed into a multi-layer perceptron, then outputs are aggregated using a permutationinvariant function (e.g., max-pooling) to obtain a global point cloud representation.
Another class of methods are convolution-based networks (Hua et al., 2018;
Su et al., 2018;
Li et al., 2018b; Atzmon et al., 2018; Landrieu & Simonovsky, 2018; Hermosilla et al., 2018;
Groh et al., 2018;
Rao et al., 2019).
These works map point clouds to regular grid structures and extend the classic convolution operator to handle these grid structures.
A representative model, PCNN (Atzmon et al., 2018), deﬁnes two operators, extension and restriction, for mapping point cloud functions to volumetric functions and vise versa.
The third class of models is graph-based networks (Simonovsky & Komodakis, 2017; Wang et al., 2019b;
Shen et al., 2018; Wang et al., 2018; Zhang & Rabbat, 2018; Chen et al., 2019).
These networks regard each point as a vertex of a graph and generate edges based on spatial information and node similarities.
A popular method is DGCNN (Wang et al., 2019b), which introduces a new operation, EdgeConv, to aggregate local features and a graph update module to learn dynamic graph relations from layer to layer.
NRS (Cao et al., 2020) uses a neural random subspace method based on the encoded embeddings to further improve the performance of the models.
4. P RE -T RAINING FOR P OINT C LOUDS
Pre-training models on unlabelled data are gaining popularity recently due to its success on a wide range of tasks, such as natural language understanding (Mikolov et al., 2013; Devlin et al., 2018), object detection (He et al., 2020; Chen et al., 2020) and graph representations (Hu et al., 2020b;c).
The representations learned from these pre-trained models can be used as a good initializer in downstream tasks, where task-speciﬁc annotated samples are scarce.
The three most common pretraining objectives for point clouds are based on: (i) generative adversarial networks (GAN), (ii) autoencoders, and (iii) spatial relation reconstruction (Sauder & Sievers, 2019; Sharma & Kaul, 2020).
However, GANs for point clouds are limited to non-point-set inputs, i.e., voxelized representations (Wu et al., 2016), 2D depth images of point clouds (Han et al., 2019), and latent representations from autoencoders (Achlioptas et al., 2018), as sampling point sets from a neural network is non-trivial.
Thus these GAN approaches cannot leverage the natural order-invariance of point-sets.
Autoencoders (Yang et al., 2018; Li et al., 2018a; Hassani & Haley, 2019; Shi et al., 2020) learn to encode point clouds into a latent space before reconstructing these point clouds from their latent representation.
Yet, there is recent evidence that the representations learned by autoencoders are not necessarily good when used as initialization for downstream tasks (Alberti et al., 2017).
In fact, both GAN and autoencoder-based pre-training methods have been recently outperformed on downstream tasks by the pre-training technique of Sauder & Sievers (2019) or Sharma & Kaul (2020) in few-shot learning.
These methods are based on spatial relation reconstruction, which aims to reconstruct points clouds given rearranged point clouds as input.
To this end, Sauder & Sievers (2019) equally split the 3D space into k 3 voxels, rearrange k 3 voxels and train a model to predict the original voxel label for each point.
However, these random permutations destroy all spatial information that the model could have used to predict the true point cloud.
Inspired by cover-trees (Beygelzimer et al., 2006), Sharma &
Kaul (2020) utilised ball covers for hierarchical partitioning of points.
They then train a model to classify each point to their assigned clusters.
However, the selection of the ball centroids is somewhat random and they need to pre-train from scratch for each ﬁne-tuning task.
Instead, our method creates spatially realistic occlusions that a completion model learns to reconstruct.
As such, this model learns how to naturally encode 3D object shape and contextual information.
Recently there is a new method called PointContrast (Xie et al., 2020) which mainly uses contrastive learning for pre-training of indoor segmentation models.
Our method is more general and transferable compared with theirs.
Completing 3D shapes for model initialisation has been considered before. Sch ¨ onberger et al. (2018) used scene completion as an auxiliary task to initialise 3D descriptors for visual localisation. SGNN (Dai et al., 2020) and RevealNet (Hou et al., 2020) speciﬁcally tackles the unsupervised scene completion task by inferring unobserved scene geometry.
Besides completion, generative models based on normalizing ﬂow (Yang et al., 2019b) and approximate convex decomposition (Gadelha et al., 2020) have been shown effective for the unsupervised learning on point clouds.
Compared to these works OcCo is different in a signiﬁcant way: our approach only learn pre-trained weights on a single dataset (ModelNet40).
We have shown that such pre-trained weights outperform all point cloud baselines in few-shot learning, improve generalization on many different classiﬁcation tasks, and have better transformation invariance than prior work.
As far as we are aware it was not known that such pre-training would yield such improvements across tasks and datasets.
D ISCUSSION
In this work, we have demonstrated that Occlusion Completion (OcCo) learns pre-trained representations on point clouds that are more transformation invariant, more accurate in few-shot learning, and in various classiﬁcation and segmentation ﬁne tuning tasks, compared to prior work.
There are a few exciting directions that we would like to explore in future.
First, we believe this method would be useful as a data-augmentation technique during training to improve robustness, similar to input dropout in the 2D case Bouthillier et al. (2015).
Second, it would be interesting to design a completion model that is explicitly aware that it is completing an occluded view-point.
A model like this would likely converge even quicker, and require fewer parameters, as this knowledge could act as a stronger inductive bias during learning.
In general, we advocate for structuring deep models using graphical constraints as an inductive bias to improve learning.
A D ESIGN OF THE C OMPLETION M ODEL
Previous point completion models (Dai et al., 2017b; Yuan et al., 2018; Tchapmi et al., 2019; Wang et al., 2020a) all use an ”encoder-decoder” architecture.
The encoder maps a partial point cloud to a vector of a ﬁxed dimension, and the decoder reconstructs the full point cloud.
In the OcCo experiments, we exclude the last few MLPs of PointNet and DGCNN, and use the remaining architecture as the encoder to map a partial point cloud into a 1024-dimensional vector.
We adapt the folding-based decoder design from PCN, which is a two-stage point cloud generator that produces a coarse and a ﬁne-grained output point cloud ( Y coarse , Y fine ) for each input.
We removed all the batch normalisation layers in the folding-based decoder since we ﬁnd it brings negative effects in the completion process in terms of Chamfer distance loss and convergent speed.
On the basis of prior self-supervised learning methods, SimCLR (Chen et al., 2020), MoCo (He et al., 2020) and
BYOL (Guo et al., 2020), we ﬁnd the batch normalisation is important in the encoder yet harmful for our decoder.
Also, we ﬁnd the L2 normalisation in the Adam optimiser is undesirable for completion training but brings improvements on the downstream ﬁne-tuning tasks.
The predicted coarse point cloud ˆ Y coarse , which represents the global geometry of a shape, is generated via a set of fully connected layers.
A folding-based generator is used to predict the local ﬁne structures of each point in ˆ Y coarse , this results in ˆ Y fine .
The folding based structures is proved to be good at approximating a smooth surface which reﬂects the local geometry.
During training, Y coarse and Y fine are generated via randomly sampling 1024 and 16384 points from the mesh, respectively.
We use either Chamfer Distance (CD) or Earth Mover Distance (EMD) as the loss function for the completion model.
We use a normalised and symmetric (thus commutative) version of Chamfer
Distance (CD) to quantify the differences between two point clouds ˆ P and P :
[Equation]
Note that it is no need that the two point cloud ˆ P and P have the same size.
But when calculating the
Earth Mover Distance (EMD), ˆ P and P are usually required to have the same number of points:
[Equation]
where φ is a bijection between points in ˆ P and P .
Note that EMD is not commutative.
Since ﬁnding the optimal mapping φ is quite time consuming, we use its approximation form Bertsekas (1985).
The loss l of the completion task is a adaptive weighted sum of coarse and ﬁne generations:
[Equation]
where the step-wise trade-off coefﬁcient α incrementally grows during training.
In our experiments, we ﬁnd that even with approximation, it is still suboptimal to use EMD for d 2 , since it is inefﬁcient to solve the approximate bijection mapping φ for over 16k point pairs.
We evaluate both ‘EMD+CD’ and ‘CD+CD’ combinations for the loss l .
We have found that OcCo with ‘EMD+CD’ loss has achieved comparable performance yet longer time in the downstream classiﬁcation tasks compared with the ‘CD+CD’.
We use ‘CD+CD’ as the loss function in the OcCo pre-training process described in Section.
3.1 in terms of simplicity and efﬁciency.
B Q UALITATIVE R ESULTS FROM O C C O P RE -T RAINING
In this section, we show some qualitative results of OcCo pre-training by visualising the input, coarse output, ﬁne output and ground truth at different training epochs and encoders.
In Figure.
Figure.
7 and Figure.
8, we notice that the trained completion models are able to complete even difﬁcult occluded shapes such as plants and planes.
In Figure.
9 we plot some failure examples of completed shapes, possibly due to their complicated ﬁne structures, while it is worth mentioning that the completed model can still completed these objects under the same category. C V ISUALISATION OF FEATURE ACTIVATION MAP AND GROUND TRUTH
D A NALYSIS ON THE E FFECTS OF O C C O P RE -T RAINING D ATASETS
We compare the occluded datasets based on ModelNet40 and ShapeNet8 for the OcCo pre-training.
We construct the ModelNet Occluded using the methods described in Section 2 and for ShapeNet
Occluded we directly use the data provided in the PCN, whose generation method are similar but not exactly the same with ours.
Basic statistics of these two datasets are reported in Table 6.
Compared with the ShapeNet Occluded dataset which is publicized by PCN and used in all the follow-ups(Tchapmi et al., 2019; Wang et al., 2020a), our occluded ModelNet dataset has more object categories, more view-points, more points per object and therefore is more challenging.
We believe such differences will help the encoder models learn a more comprehensive and robust representation which is transferable to downstream tasks.
To support our idea, we perform OcCo pre-training on these two datasets respectively, and test their performance on ModelNet40 and ShapeNet Occluded classiﬁcation benchmarks. [Table]
[Table] The reason of choosing these two datasets for benchmarking is, ShapeNet Occluded is the out-ofdomain data for the models pre-trained on ModelNet Occluded, and vice versa.
We believe it will give us sufﬁcient information on which occluded dataset should be preferred the OcCo pre-training.
The Results are shown in Table 7.
[Table]
From Table 7, we see that the OcCo models pre-trained on ShapeNet Occluded do not perform as well as the ones pre-trained on ModelNet Occluded in most cases.
Thus in our experiments, we reports the results pre-trained on ModelNet Occluded.
By visualising the objects from the ShapeNet Occluded (in Figure. 11), we believe this performance deﬁciency in downstream ﬁne-training of pre-trained models is due to the quality of the generated occluded point clouds (in comparison with our generated dataset shown in Figure. 2).
Further, we think our dataset is a more challenging task for all the present completion models.
E D ETAILED R ESULTS ON TRAINING A L INEAR SVM FOR C LASSIFICATION
We ﬁrst described the benchmark datasets that are used for classiﬁcation in Table 8.
[Table]
To make a comprehensive and convincing comparison, we follow the similar procedures from (Achlioptas et al., 2018; Han et al., 2019; Sauder & Sievers, 2019; Wu et al., 2016; Yang et al., 2018), to train a linear Support Vector Machine (SVM) to examine the generalisation of OcCo encoders that are pre-trained on occluded objects from ModelNet40. For all six classiﬁcation datasets, we ﬁt a linear SVM on the output 1024-dimensional embeddings of the train split and evaluate it on the test split. Since Sauder & Sievers (2019) have already proven their methods are better than the prior, here we only systematically compare with theirs.
We report the results 2 in Table 9, we can see that all OcCo models achieve superior results compared to the randomly-initialized counterparts, demonstrating that OcCo pre-training helps the generalisation both in-domain and cross-domain. [Table]
F R E -I MPLEMENTATION D ETAILS OF ”J IGSAW ” P RE -T RAINING M ETHODS
In this section [Table] we describe how we reproduce the ’Jigsaw’ pre-training methods from (Sauder &
Sievers, 2019).
Following their description [Table] we ﬁrst separate the objects/chopped indoor scenes into 3 3 = 27 small cubes and assign each point a label indicting which small cube it belongs to.
We then shufﬂe all the small cubes
[Table]
and train a model to make a prediction for each point.
We reformulate this task as a 27-class semantic segmentation, for the details on the data generation and model training, please refer to our released code
[Table]
G M ORE COMPARISONS
In Table
[Table]
we compare OcCo with prior point-cloud-speciﬁc pre-training methods (Alliegro et al., 2020).
Our method obtains the best results on all settings.
These results conﬁrm that the inductive bias learned by reconstructing occluded point clouds is stronger than one based in reconstructing permuted clouds (Alliegro et al., 2020; Sauder & Sievers, 2019).
Speciﬁcally, we believe that because
OcCo does not rearrange object parts but instead creates point clouds that resemble real-world 3D sensor occlusions, the initialization better encodes realistic object shape and context.
[Table]
H L ABELLED S AMPLE E FFICIENCY
We investigate whether OcCo pre-training can improve the labelled sample efﬁciency of downstream tasks. Speciﬁcally [Table] we reduce the labelled samples to 1%, 5%, 10% and 20% of the original training set for the ModelNet40 object classiﬁcation task
[Table]
and evaluate on the full test set.
As shown in Table 11, OcCo-initialized models achieve superior results compared to the randomly-initialized models, demonstrating that OcCo with in-domain pre-training improves labelled sample efﬁciency.
[Table]
We sketch the network structures of PCN encoder and output layers for downstream tasks in Figure 12. [Figure]
J D ETAILED RESULTS OF THE PART SEGMENTATION
Here in Table 12 we report the detailed scores on each individual shape category from ShapeNetPart, we bold the best scores for each class respectively.
We show that for all three encoders, OcCoinitialisation has achieved better results over two thirds of these 15 object classes. [Table] Aero 83. Bag 78.
Cap 82.
Car 74.
Chair 89. Earphone 73. [Table] [Table]
Guitar 91.
[Table]
Knife 85.
[Table]
[Table]
Lamp
[Table]
[Table]
[Table]
[Table]
[Table]
Laptop 95.
[Table]
[Table]
Motor 65.
Mug 93.
Pistol 81.
[Table]
K A LGORITHMIC D ESCRIPTION OF O C C O
# P: an initial point cloud # K: camera intrinsic matrix # V: number of total view points # loss: a loss function between point clouds # c: encoder-decoder completion model # p: downstream prediction model while i < V:
# sample a random view-point
R_t =
[random.rotation(), random.translation()] # map point cloud to camera reference frame P_cam = dot(K, dot(R_t, P))
# create occluded point cloud P_cam_oc = occlude(P_cam, alg=’z-buffering’)
# point cloud back to world frame K_inv =
[inv(K), zeros(3,1); zeros(1,3), 1]
R_t_inv = transpose([R_t; zeros(3,1), 1]) P_oc = dot(R_t_inv, dot(K_inv, P_cam_oc))
# complete point cloud P_c = c.decoder(c.encoder(P_oc))
# compute loss, update via gradient descent l = loss(P_c, P) l
[Table]
[Table]
+=
[Table]
[Table]