Self-Supervised Pretraining for Large-Scale Point Clouds
Anonymous Author(s)
AffiliationAddress
Abstract
Pretraining on large unlabeled datasets has been proven to improve the downstream task performance on many computer vision tasks, such as 2D object detection and video classification.
However, for large-scale 3D scenes, such as outdoor
LiDAR point clouds, pretraining is not widely used.
Due to the special data characteristics of large 3D point clouds, approaches for 2D pretraining frameworkstend to not generalize well to this domain.
In this paper, we propose a new selfsupervised pretraining method that targets large-scale 3D scenes.
We pretraincommonly used point-based and voxel-based model architectures and show thetransfer learning performance on 3D object detection and semantic segmentation.
We demonstrate the effectiveness of our approach on both dense 3D indoor pointclouds and sparse outdoor LiDAR point clouds.
Introduction
Accurate and reliable perception is core to the capabilities and safety of autonomous and roboticsystems such as self-driving vehicles and indoor robotics.
To accurately measure the structure oftheir surroundings, such platforms usually rely on LiDARs and depth cameras for outdoor and indoorscenes, respectively.
Given the 3D point clouds, models are built for a variety of perception tasks,such as object detection, scene segmentation, localization, and mapping.
Modern deep learning models have shown great capabilities when large quantities of annotated dataare given.
However, generating the required amount of annotations is a laborious and expensivemanual process [11; 52; 15].
This is especially true for labeled datasets targeting object detectionand semantic segmentation of large real-world scenes, which can contain dozens of independentlymoving objects and irregularly shaped background material.
Moreover, the 3D data is difficult toclearly visualize, manipulate, and annotate on a 2D screen.
In the case of outdoor LiDAR scans, theannotation tasks are further complicated by the sparsity of the typical point cloud.
However, the costof acquiring a large quantity of unlabeled sensor data is usually much lower compared to manualannotation.
Hence, developing algorithms to automatically discover useful patterns and the of models few annotated examples is of great value
Recently researchers have explored self-supervised learning (SSL) to bootstrap model training in various domains.
In computer vision, [23; 7; 19; 22] make use of datasets such as ImageNet [45] with carefully designed pretext tasks to learn useful features for downstream tasks such as object detection and segmentation.
These techniques rely on the enormous ImageNet dataset which depicts tens of thousands of object types.
Most images prominently feature a single foreground object, which can be well-described by a global feature vector for each sample.
In the 3D point cloud domain, several works [67; 60] have proposed pretraining techniques to learn useful features for downstream tasks.
However, these methods likewise primarily focus on much simpler scenes such as single objects or a limited indoor scene captured by a depth camera from a single viewpoint
On the other hand, we target complex 3D scenes with a multitude of objects and background material.
We visualize the differences between the subject datasets of existing approaches and ourmethod in Figure 1.
We first consider large-scale, complete 3D indoor scene models generated fromcombined depth images.
Next, we explore outdoor environments with LiDAR points up to 100maway on dozens of objects and complex background regions.
We refer to our method as SSPL ( s elfs upervised p retraining for l arge-scale point clouds).
Unlike most of the prior work, we propose atechnique which performs contrastive learning on local features extracted from sub-regions (volumes) within large-scale point clouds.
Our algorithm makes use of a local contrastive loss within ascene and as well as a global contrastive loss across different scenes.
As we will demonstrate, ourmethod involving local feature reasoning greatly outperforms the existing methods in our settings.
In practice, [66] has noted that there are significant domain gaps attributable to the LiDAR sensor dueto its sparsity, even when the observed scenes and downstream tasks are highly similar.
Motivatedby this, we explore the setting of in-domain self-supervised pre-training for LiDAR point clouds.
Aswe will demonstrate, existing 3D SSL methods [67; 60] are unsuitable for highly complex outdoorscenes, motivating the need for new algorithms.
In this paper, we make the following contributions:
• We propose a self-supervised pretraining method which leverages reasoning over local volume level features for 3D point clouds.
• We introduce a novel ClusterInfoNCE loss to efficiently perform contrastive learning glob-
ally for a large set of local volume features.
• We show that our pretraining
approach outperforms baselines on multiple downstream taskson both single-view outdoor LiDAR and large-scale indoor point clouds, with up to +6.2%higher
AP25 in object detection and +3.6% mIoU in semantic segmentation.
Related work
Our method builds on and extends the work from the self-supervised learning literature, with afocus on large-scale 3D point clouds.
In this section, we give an overview of the recent advances inself-supervised learning, as well as perception and representation learning in 3D.
Self-supervised learning for images Self-supervised learning has been studied extensively in machine learning and computer vision [56; 39; 44; 46; 35].
There are many approaches for 2D representation learning, such as clustering [5; 6; 28], GANs [13; 36], various pretext tasks [12; 38; 57]etc .
Recent advances [37; 23; 24; 7; 8; 54; 31; 17; 19; 22] have shown that self-supervised pretraining exhibits great advantages in transfer learning, especially for benchmark with limited labels.
Ourwork adopts the momentum encoder from MoCo [23] to extract features for instance discrimination,instead of a memory bank.
Self-supervised learning for 3D data Studies on unsupervised feature learning for 3D data havemainly been focused on single 3D object representation with applications on reconstruction, classification or part segmentation [2; 14; 20; 21; 33; 47; 58; 64; 30; 1].
However, due to the domain gapissues, features learned from single 3D object representations does not transfer well to scene level understanding tasks [60].
Recently, there has been a growing interest in designing self-supervisedmethods to build representations of scene level point clouds [60; 27; 67].
Xie et al .
[60] proposed aself-supervised method based on enforcing contrastive learning on point correspondences betweendifferent views of 3D scenes.
These point-wise correspondences can be acquired after aligningdifferent depth maps into a unified 3D scene, which requires time-consuming 3D registration algorithms or specific human labeling efforts on identifying static correspondences.
In the case of sparseoutdoor LiDAR point clouds, temporal correspondences are often not well defined.
Similarly, another line of work [27; 16] rely on multiple scene scans’ alignments.
On the other hand, our methodonly requires single frame 3D scans, making it more generally applicable.
Zhang et al .
[67] conducted an instance discrimination task on global scene-level features while our method focuses onreasoning across local patches.
Supervised 3D perception With the significant applications of robotics and self-driving vehicles,3D perception has been a widely studied topic.
Among the many perception tasks, supervised objectdetection [41; 49; 62; 48; 50; 32; 65; 9; 63; 68] and semantic segmentation [69; 55; 61; 25; 26; 29]have been explored by a large number of prior works.
Because of the popularity and usefulness ofthese tasks, we select them as the downstream tasks to analyze the effectiveness of our approach.
However, we note that as our self-supervision is task agnostic, it can be applied to other downstreamtasks.
To achieve these tasks, a number of neural network backbones have been proposed to process3D point cloud data, which is usually given as an unordered and irregular set of points [42; 18; 53].
In our work, we use both the popular PointNet++ [43] based on pointwise reasoning and 3D U-Net[61] based on sparse voxel convolutions to demonstrate the general effectiveness of our approach.
Method
We introduce our self-supervised pretraining framework: SSPL as outlined in Algorithm 1, and visualized in Figure 2.
As shown in Figure 1, we focus on large-scale point clouds, which containnumerous objects and distinct structures within each scene.
In these cases, a global vector encoding requires a combinatorial and intractable level of expressiveness to describe the attributes andlocations of the various objects.
Therefore, we propose to perform representation learning on local features.
We first describe local volume features extraction in Section 3.1, and introduce thelocal contrastive loss in Section 3.2.
Finally, we discuss the formulation of the global contrastivereasoning in Section 3.3.
3.1 Local feature extraction
Given a large scale 3D point cloud instance x i , we split it into M occupied volumes.
Similar topatches in 2D images, we can predefine a volume size and sample up to M occupied volumes basedon typical scale of input scenes.
x i is then transformed into two augmented views x 1 i and x 2 i by applying random rotation and scaling.
We then sample points in each occupied volume and generatetwo crops with different sizes for two augmented views.
The smaller crops v 1 i , generated from x 1 i ,are defined as local crops while larger ones v 2 i , generated from x 2 i , are global crops.
This schemecreates natural local-global correspondences, which enables the network to predict global featuresfrom local information in feature learning.
x 1 i and x 2 i are mapped to feature vectors by applyinga neural network.
After acquiring the per-point/per-voxel features, we group the features by theircorresponding volumes and apply max-pooling to the last layer’s features.
We then apply a two layer
MLP as in [8] and L2 normalization to compute the volume features f 1 and f 2 .
Momentum encoder Adapting the method of He et al .
[23], we use an encoder θ s to extract featuresfor x 1 i and a momentum encoder θ t to extract features for the augmented view x 2 i .
The weights ofthe momentum encoder are updated with an exponential moving average (EMA) from the encoder’sweights.
The update rule is θ t ← λθ t + (1 − λ ) θ s with a fixed λ = 0 . 999 .
3.2 Local volume reasoning
Traditionally, given a pair of volume features f 1 , f 2 ∈ R M × D from two augmented versions ofx i instance, feature consistency is usually applied as a pre-text task.
For contrastive learning approaches, InfoNCE [23] is usually enforced, where volume features f 1 and f 2 are pushed to be similar and volume features f 1 are dissimilar to all other volume features from other instances.
However,computation costs increase linearly as the number of feature embedding increases for each batch. Traditionally, InfoNCE assumes that there is one feature embedding for each data instance. Sincein average there are about 300 occupied volumes for each scene in our experiments, it is computationally infeasible to perform InfoNCE on each volume feature with a large set of global features asnegative examples.
Therefore, we adapt the PointInfoNCE [60] loss to conduct contrastive learning between differentvolumes of the same point cloud.
Compared to PointInfoNCE, we reason about volume-level information, where the loss is defined as:
[Equation]
Here, M is the set of all occupied volumes for the x i instance, and τ is a temperature parameter [59].
The positive examples are the two augmented crops from the same volume, while negative examplesare from other volumes under the same instance.
In practice, we find that performing this localvolume reasoning alone as a pre-text task gives comparable or better pretrained models compared tobaseline methods.
Please refer to Section 5.4 for experimental evidences.
Although applying BYOL [19] as a pretext task does not yield computational issues, we find that itresults in minimal performance gains in our settings.
Please see Section 5.3 for more details.
3.3 Global volume reasoning
One natural objective for global contrastive reasoning is to use instance discrimination on volumefeatures.
As mentioned before, this objective does not scale well with large numbers of occupied volumes.
Across different large scale scene datasets, we observe that there are common objectsappearing in almost every scene, e.g . road and car in outdoor scenes or chair and sofa in indoorscenes.
Inspired by this, we propose to conduct global volume reasoning through a fixed number ofglobal clusters.
This reduces the computational burden and adds regularization during pretraining.
Preliminary on unsupervised feature clustering Traditional methods on feature clustering, suchas k-means and DBSCAN have proven to be effective but time-consuming to apply on large dataset.
Recently, a line of methods [3; 7] show that pseudo-label assignment can be formulated as an optimaltransport problem, where clustering can be done efficiently by the Sinkhorn-Knopp algorithm [10].
We follow a similar approach for our feature clustering steps.
Global feature queue We first build a global feature queue F g to store features of occupied volumesover multiple data instances.
Adapting from He et al .
[23], the global feature queue is created withvolume features f 2 from the momentum encoder.
The following steps are then used during training.
Global feature clustering For building global clusters, we follow the same method proposed by
Asano et al .
Given global feature queue F g ∈ R C × D , where C is the queue size and D isthe feature dimension, we apply a classification head h g : R D
→ R K to convert the global featurevectors to class (cluster ID) scores.
We then map them to class probabilities via a softmax operator:
P = softmax ( F g · h g ) .
We find the class label assignments Q by performing iterative SinkhornKnopp updates [10].
Global contrastive loss Given a pair of volume features f 1 , f 2 , we query the global feature queuewith local features f 2 to extract the corresponding cluster labels for each volume.
We use kNN onl 2 feature distances when querying, and we set k = 1 for clustering.
With volume level class labels, it is natural to use a classification pretext task for pretraining.
However, compared with pretraining on object-level features ( e.g . ImageNet), our volume-level featuresusually lack class diversity and feature diversity.
For indoor scenes, there are repeating wall andfloor volumes, while a majority of volumes are roads and buildings in outdoors scenes.
We observethat such pretraining strategy provides minimal improvements.
(Please see section 5.4 for details.)
To directly apply feature representation learning across scenes, we propose a ClusterInfoNCE loss:
[Equation]
where K is the number of clusters and f 1 i and f 2 i are drawn from f 1 and f 2 if their correspondingclass labels are i .
knn ( f 2 i , F g ) extracts the most similar features of f 2 i from F g , which serve aspositive examples.
For negative examples, F jg are sampled from F g if their corresponding classlabels are j .
In our experiments, we find that using 20k negative examples works well.
Since theruntime of ClusterInfoNCE loss depends only on number of clusters used, it scales well with moreoccupied volumes.
By optimizing this loss, we are explicitly enforcing feature consistency acrossscenes and encouraging the volume features to form clusters globally.
As shown in Section 4, weobserve that adding the ClusterInfoNCE loss results in improvements on various downstream tasks.
As well, we compare it with different negative sampling strategies in Section 5.4.
3.4 Implementation details
In our experiments, we set the size of global feature queue to be 300K.
We use a temperature valueof 0.1 while computing the non-parametric softmax in Eq 1 and 3.
The local and global contrastiveloss are equally weighted.
For training, we use a standard SGD optimizer with momentum 0.9, andwe use a cosine learning rate scheduler [34] which decreases from 0.06 to 0.00006 and train themodel for 500 epochs with a batch size of 96.
Experimental results
In this section, we thoroughly analyze the efficiency of our proposed method.
First, we describe thetasks and datasets involved in our experiments.
Then, we present quantitative results, demonstratingthe increase in performance relative to our comparison baselines.
Finally, we take a closer look atthe contributions of each of our proposed components.
[Table]
The key goal of self-supervised learning is to automatically learn feature extractors that are usefulfor actual downstream tasks.
Moreover, it is beneficial if the learned features are useful for differenttasks, as this improves the usability of the technique and reduces the need for task-specific redesign.
Our approach is chiefly designed for large scale scenes that are not dominated by a single, well defined foreground object.
We demonstrate the effectiveness of our self-supervised learning approachby showing that downstream models can achieve significantly higher performance using a smallfraction of available annotations.
Next, we dive into the features learned by our model to gain additional insights, followed by various ablation studies to justify the design choices in our approach.
Asmentioned in Section 1, for LiDAR point clouds, the sensor configuration has a tremendous impacton the structure of perceived 3D points.
Therefore, we opt to pretrain and finetune our models ineach domain individually for outdoor datasets.
4.1 Baseline methods
We limit our input data to measurements taken at single instants in time (instead of temporal sequences) for generalization.
As such, we use the recent BYOL [19] and DepthContrast [67] (basedon MoCo [23]) as baselines.
In both cases, we use the same full scene augmentation scheme as proposed in DepthContrast.
For indoor scene pretraining, to introduce more scene complexity, we usethe point clouds of fully reconstructed scenes built by aggregating multiple sequential measurementsof the scene.
Therefore, we also use PointContrast [60] as a comparison for the indoor setting.
4.2 Large scale dense point clouds
First, we evaluate our feature learning framework on point clouds produced with common depthcameras.
In our work, we choose the ScanNet [11] dataset for pretraining.
ScanNet contains 1500large scale indoor scenes, with each scene consisting of a variety of objects.
Additionally, it containsmore than 10 different room types with diverse room scales.
We believe that ScanNet covers adiverse set of indoor object distributions.
Compared to DepthContrast [67] and PointContrast [60], we only utilize the full scans from ScanNettraining dataset for pretraining.
We aim to learn sharp local features from large scale scenes, and thuswe are only interested in feature learning with full views instead of partial views.
The pretrainingdataset contains 1201 full view scans with 50k points per scan.
We use a PointNet++ [43] backbonenetwork for pretraining, as it is widely used for different 3D perception benchmarks.
To evaluate the transfer learning performance of our pretrained models, we use two commonly usedindoor 3D object detection benchmarks: ScanNet (SCAN)
[11] and SUN RGB-D (SUN) [51].
Wefinetune our pretrained model by using it as the backbone initialization of VoteNet [40] (a wellknown 3D object detector), and report the detection performance using the mean Average Precisionat IoU=0.25 (AP25) metric.
To study the label efficiency of our pretrained models, we also subsample different sets of the training data used for finetuning.
We follow the setup in VoteNet [40] forfinetuning.
Based on the results in Table 1, our approach significantly improves downstream object detectionperformance across all settings, with up to 4.4% improvement on ScanNet and 2.7% on SUN RGBD.
Moreover, on ScanNet, our pretraining approach together with 20% of labeled data allows thedownstream detection task to achieve the equivalent performance of using 50% of labeled data andtraining from scratch.
Practically, this can lead to significant savings in labeling cost.
Our ap- [Table] [Table] proach also outperforms the top-performing baseline method DepthContrast.
(Note that we reproduce DepthContrast with the original Votenet model and pretrain only on the 1200 instances for faircomparison.)
We see a more prominent performance gain on ScanNet compared to on SUN RGB-D,as the former contains only 1201 training scenes while the latter contains more than 5k.
4.3 Large scale sparse point clouds
Next, we further increase the size and complexity of the 3D scene by using outdoor scenes collectedfor autonomous vehicles using LiDAR units.
These scenes consist of a large number of differentobjects and regions, spanning dozens of semantic classes.
As we will show, it is critical to considerlocal regions separately in the self-supervised learning framework.
Our algorithm aims to learn useful features for all regions in a point cloud.
While detectable individual objects cover a large fraction of the 3D structures present in the previous indoor datasets,compact and countable foreground objects suitable for detection such as cars and pedestrians consist of only 10% of a typical outdoor scene.
As such, we select the semantic segmentation task tovalidate our method, as it aims to give a meaningful label to every 3D point captured by the sensor.
We use two large scale, well-established public datasets for this task, which contain a large varietyof scenes.
SemanticKITTI (SK) [4] is based on the original KITTI [15] dataset, which is one ofthe original autonomous driving datasets that propelled a tremendous amount of progress in theperception field.
This dataset contains 19k training frames over 10 sequences with approximately122k points per scene.
The more recent Waymo Open Dataset (WOD) [52] provides a further leapin the scale and variability of data by providing 24k training frames sampled from 850 drivingsequences, with a higher point density of 161k points per scene.
Both datasets define approximately20 semantic classes.
For these experiments, we use the sparse 3D convolution backbone based on
U-Net [70] as it is used in current state of the art semantic segmentation models [61; 69].
As well,we largely follow the training settings of [61].
To show that our approach extracts meaningful representations, we present semantic segmentationresults on subsampled sets of the training data provided by SemanticKITTI and WOD.
The samplesare uniformly and randomly selected from the overall training datasets.
To simulate the scenariowhere different levels of human annotation effort is available, the samples are drawn such that anysmaller fractional subset is contained within any larger subset.
The results are shown in Table 2.
We add a light-weight decoder with two convolutional layers to the pretrained point-wise featureencoding backbone network, and finetune until convergence.
In each case, we report the averageperformance over the checkpoints of the last 15% of the training process to reduce noise.
Our approach significantly improves downstream semantic segmentation when the annotation islimited.
For example, we observe 3.6% and 2.3% improvements in mIoU on SemanticKITTI and
WOD, respectively, when annotations for 1% subsets of the full datasets are provided.
Moreover,these improvements are much more significant than those gained from pre-training with BYOL and
DepthContrast, suggesting that beneficial pre-training on large scale point clouds require designswith specific considerations.
On the other hand, we observe that the gains are reduced when additional annotated data is provided, as seen in the case with 5% and 10% annotations, as expected.
4.4 Feature analysis
Here, we examine the features learned by our self-supervision scheme.
As with the other analysisin our work, we compare with the BYOL and DepthContrast baselines, and use the SemanticKITTI dataset for the study.
We visualize the final per-point features from the 3D UNet model using thet-SNE dimensionality reduction and color the points according to their semantic class.
For clearerinterpretability, we limit the semantic classes to the commonly seen and important classes of road,person, and car .
The points are subsampled from the validation set.
We see that the embeddings produced using DepthContrast and our approach are both able to separate the three classes in feature space to a much stronger degree than the BYOL baseline, as well asthe random initialization which serves as a sanity check.
In comparison with DepthContrast, our embeddings have larger variance within the feature space while still being able to separate the semanticclasses, suggesting that they can possibly preserve more distinctive geometric information while extracting meaning.
This hypothesis is supported by the added performance gain of pretraining usingour method compared with DepthContrast as seen in Table 2.
Ablation study
In this section, we demonstrate the importance of each component in our training framework.
Weconduct our ablation studies with models pretrained on the widely accepted SemanticKITTI datasetwith complex and large-scale scenes in autonomous driving.
5.1 Volume cropping ratios
In the left side of Figure 4, we report the impact of applying different cropping ratios for the localcrop.
For the global crop, we randomly crop approximately 50% of the original volume so that mostof the geometry feature is still preserved in the momentum encoder.
We observe that smaller localcrops produce marginal improvements in finetuning, and that a 10% crop ratio is optimal.
5.2 Number of occupied volumes
For local volume reasoning, the number of occupied volumes plays an important role as it directlyaffects the number of negatives used for local contrastive loss.
Intuitively, by increasing the number of occupied volumes, the local contrastive loss forces the network to learn more distinct localfeatures.
However, with a fixed model size, the expressiveness of local features saturates after itforms a certain number of clusters, which directly correlates with number of occupied volumes inour training approach.
As shown in the middle of Figure 4, we see strong performance with 400occupied volumes with little gain for further increases. [Table] [Table]
[Table]
BYOL
[Table]
contrastive
[Table]
Instead of contrastive methods, Grill et al .
[19] proposed a new approach for self-supervised learning BYOL , which directly enforces feature consistency between a student network and a teachernetwork.
To better understand our framework, we reproduce the BYOL loss formulation to our besteffort to replace the PointInfoNCE in the local volume reasoning.
Formally, we use the followingmean square error between the normalized predictions and target projections:
[Equation]
Similar to Equation 2, M is the set of all occupied volumes, and z θ is a predictor for the studentnetwork.
Based on the results from Table 3, the features learned with BYOL performs significantlyworse than the contrastive approach in 1% SemanticKITTI finetuning.
We hypothesize that withinthe same instance, some volumes share very similar feature patterns.
Without explicitly encouraging dispersion within the feature space (e.g. negative examples), BYOLlearns less distinguishing features in our setting.
5.4 Different strategies for global volume reasoning
As mentioned in Section 3, a natural pretext task for global volume reasoning is classfication.
Givena pair of volume features f 1 , f 2 and their corresponding cluster labels Q v , we apply a linear reprojection h l to map the volume features to class scores, and optimize the classification loss below:
[Equation]
where P 1 j = softmax ( f 1 · h l ) and P 2 j = softmax ( f 2 · h l ) .
As shown in Table 4 where Cls represents the model pretrained on the classification loss, we see that it does not improve the finetuningperformance, compared to using local contrastive loss alone ( Local ).
We also compare differentnegative sampling strategies for the ClusterInfoNCE loss.
In our approach, after we find the positivefeature example for cluster i , we sample negative feature examples from other clusters.
In Table 4,we compare with only sampling negative examples from cluster i ( Hard ) (which can be seen as hardnegative sampling) as well as random sampling from all clusters ( Rand ).
Based on the results, ourapproach ( Our ) provides the largest performance boost.
5.5 Number of global clusters
We explore the impact of the number of global clusters used in our global feature queue to computethe contrastive loss (right plot of Figure 4).
This hyperparameter is shown to be impactful, where64 is the best setting for the SemanticKITTI LiDAR scans.
We believe that the optimal number isinfluenced by the complexities of the 3D scenes in question.
Limitations, future work, and broader impact
Lastly, we discuss the limitations of our model.
As the characteristics of 3D sensors vary widely(dense depth scans vs sparse LiDAR point clouds), our approach relies on in-domain unlabeleddata as opposed to producing a general purpose foundation model like those used in 2D images andlanguage.
Moreover, we do not make use of any temporal information, which can provide significantcues for feature consistency.
We believe that these are possible areas for future research.
We believethat increasing 3D perception capabilities of autonomous agents will allow them to operate moresafely and enrich our lives.
As self-supervised learning is able to extract useful representation fromunlabeled datasets, it can be less impacted by biases that may influence annotation processes, andpossibly improve the reliability and fairness of perception models.