PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
Abstract
PointNet++ is one of the most inﬂuential neural architectures for point cloud understanding.
Although the accuracy of PointNet++ has been largely surpassed by recent networks such as PointMLP and Point Transformer, we ﬁnd that a large portion of the performance gain is due to improved training strategies, i.e. data augmentation and optimization techniques, and increased model sizes rather than architectural innovations.
Thus, the full potential of PointNet++ has yet to be explored.
In this work, we revisit the classical PointNet++ through a systematic study of model training and scaling strategies, and offer two major contributions.
First, we propose a set of improved training strategies that signiﬁcantly improve
PointNet++ performance.
For example, we show that, without any change in architecture, the overall accuracy (OA) of PointNet++ on ScanObjectNN object classiﬁcation can be raised from 77 .
9% to 86 .
1% , even outperforming state-of-theart PointMLP.
Second, we introduce an inverted residual bottleneck design and separable MLPs into PointNet++ to enable efﬁcient and effective model scaling and propose PointNeXt , the next version of PointNets.
PointNeXt can be ﬂexibly scaled up and outperforms state-of-the-art methods on both 3D classiﬁcation and segmentation tasks.
For classiﬁcation, PointNeXt reaches an overall accuracy of 87 .
7% on ScanObjectNN, surpassing PointMLP by 2 . 3% , while being 10 × faster in inference.
For semantic segmentation, PointNeXt establishes a new state-of-theart performance with 74 .
9% mean IoU on S3DIS (6-fold cross-validation), being superior to the recent Point Transformer.
The code and models are available at https://github.com/guochengqian/pointnext
Introduction
Recent advances in 3D data acquisition have led to a surge in interest for point cloud understanding.
With the rise of PointNet [29] and PointNet++ [30], processing point clouds in their unstructured format using deep CNNs become possible.
Subsequent to “PointNets”, many point-based networks are introduced with the majority focusing on developing new and sophisticated modules to extract local structures, e.g . the pseudo-grid convolution in KPConv [43] and the self-attention layer in Point Transformer [56].
These newly proposed methods outperform PointNet++ by a large margin in a variety of tasks, leaving the impression that the PointNet++ architecture is too simple to learn complex point cloud representations.
In this work, we revisit PointNet++, the classical and widely used network, and ﬁnd that its full potential has yet to be explored, mainly due to two factors that were not present at the time of PointNet++: (1) superior training strategies and (2) effective model scaling strategies.
Through a comprehensive empirical study on various benchmarks, e.g ., ScanObjecNN [44] for object classiﬁcation and S3DIS [1] for semantic segmentation, we discover that training strategies, i.e ., data augmentation and optimization techniques, play an important role in the network’s performance.
In fact, a large part of the performance gain of state-of-the-art (SOTA) methods [46, 43, 56] over PointNet++ [30] is due to improved training strategies that are, unfortunately, less publicized compared to architectural changes.
For example, randomly dropping colors during training can unexpectedly boost the testing performance of PointNet++ by 5 .
9 % mean IoU (mIoU) on S3DIS [1], as demonstrated in Tab. 5.
In addition, adopting label smoothing [39] can improve the overall accuracy (OA) on
ScanObjectNN [44] by 1 . 3 %.
These ﬁndings inspire us to revisit PointNet++ and equip it with new advanced training strategies that are widely used today.
Surprisingly, as shown in Fig.
1, utilizing the improved training strategies alone improves the OA of PointNet++ by 8 . 2 % on ScanObjectNN (from 77 .
9 % to 86 .
, establishing a new SOTA without introducing any changes to the architecture (refer to Sec.
4.4.1 for details).
For the S3DIS segmentation benchmark, the mIoU evaluated in all areas by 6-fold cross-validation can increase by 13 .
6 % (from 54 .
5 % to 68 .
, outperforming many modern architectures that are subsequent to PointNet++, such as PointCNN [22] and DeepGCN [21].
Moreover, we observe that the current prevailing models [20, 43, 56] for point cloud analysis have employed many more parameters than the original PointNets [29, 30].
Effectively expanding PointNet++ from its original small scale to a larger scale is a topic worth studying because larger models are generally expected to enable richer representations and perform better [2, 19, 55].
However, we ﬁnd that the naive way of using more building blocks or increasing the channel size in PointNet++ only leads to an overhead in latency and no signiﬁcant improvement in accuracy (see Sec.
For effective and efﬁcient model scaling, we introduce residual connections [13], an inverted bottleneck design [36], and separable MLPs [32] into PointNet++.
The modernized architecture is named PointNeXt, the next version of PointNets.
PointNeXt can be scaled up ﬂexibly and outperforms SOTA on various benchmarks.
As demonstrated in Fig.
1, PointNeXt improves the original PointNet++ by 20 . 4 % mIoU (from 54 .
5 % to 74 .
9 %) on S3DIS [1] 6-fold and achieves 9 .
8 % OA gains on ScanObjecNN [44], surpassing SOTA Point Transformer [56] and PointMLP [28].
We summarize our contributions next:
• We present the ﬁrst systematic study of training strategies in the point cloud domain and show that PointNet++ strikes back ( +8 . 2 % OA on ScanObjectNN and +13 .
6 % mIoU on S3DIS) by simply adopting improved training strategies alone .
The improved training strategies are general and can be easily applied to improve other methods [29, 46, 28].
• We propose PointNeXt, the next version of PointNets.
PointNeXt is scalable and surpasses SOTA on all tasks studied, including object classiﬁcation [44, 49], semantic segmentation [1, 5]
and part segmentation [53], while being faster than SOTA in inference.
Preliminary: A Review of PointNet++
Our PointNeXt is built upon PointNet++ [30], which uses a U-Net [35] like architecture with an encoder and a decoder, as visualized in Figure 2.
The encoder part hierarchically abstracts features of point clouds using a number of set abstraction (SA) blocks, while the decoder gradually interpolates the abstracted features by the same number of feature propagation blocks.
The SA block consists of a subsampling layer to downsample the incoming points, a grouping layer to query neighbors for each point, a set of shared multilayer perceptrons ( MLPs ) to extract features, and a reduction layer to aggregate features within the neighbors.
The combination of the grouping layer, MLPs, and the reduction layer is formulated as:
[Equation]
where R is the reduction layer ( e.g . max-pooling) that aggregates features for point i from its neighbors denoted as { j : ( i, j ) ∈ N} .
p li , x li , x lj are the input coordinates, the input features, and the features of neighbor j in the l th layer of the network, respectively.
h Θ denotes the shared MLPs that take the concatenation of x lj and the relative coordinates ( p lj − p li ) as input.
Note that, since
PointNet++ with single-scale grouping that uses one SA block per stage is the default architecture used in the original paper [30], we refer to it as PointNet++ throughout and use it as our baseline.
Methodology: From PointNet++ to PointNeXt
In this section, we present how to modernize the classical architecture PointNet++ [30] into PointNeXt, the next version of PointNet++ with SOTA performance.
Our exploration mainly focuses on two aspects: (1) training modernization to improve data augmentation and optimization techniques, and (2) architectural modernization to probe receptive ﬁeld scaling and model scaling.
Both aspects have important impact on the model’s performance, but were under-explored by previous studies.
3.1 Training Modernization: PointNet++ Strikes Back
We conduct a systematic study to quantify the effect of each data augmentation and optimization technique used by modern point cloud networks [46, 43, 56] and propose a set of improved training strategies.
The potential of PointNet++ can be unveiled by adopting our proposed training strategies.
3.1.1 Data Augmentation
Data augmentation is one of the most important strategies to boost the performance of a neural network; thus we start our modernization from there.
The original PointNet++ used simple combinations of data augmentations from random rotation, scaling, translation, and jittering for various benchmarks [30].
Recent methods adopt stronger augmentations than those used in PointNet++.
For example, KPConv [43] randomly drops colors during training, Point-BERT [54] uses a common point resampling strategy to randomly sample 1 , 024 points from the original point cloud for data scaling, while RandLA-Net [15] and Point Transformer [56] load the entire scene as input in segmentation tasks.
In this paper, we quantify the effect of each data augmentation through an additive study.
We start our study with PointNet++ [30] as the baseline, which is trained with the original data augmentations and optimization techniques.
We remove each data augmentation to check whether it is necessary or not.
We add back the useful augmentations but remove the unnecessary ones.
We then systematically study all the data augmentations used in the representative works [46, 43, 32, 56, 28, 54], including data scaling such as point resampling [54] and loading the entire scene as input [15], random rotation, random scaling, translation to shift point clouds, jittering to add independent noise to each point, height appending [43] ( i.e ., appending the measurement of each point along the gravity direction of objects as additional input features), color auto-contrast to automatically adjust color contrast [56], and color drop that randomly replaces colors with zero values.
We verify the effectiveness of data augmentation incrementally and only keep the augmentations that give a better validation accuracy.
At the end of this study, we provide a collection of data augmentations for each task that allow for the highest boost in the model’s performance.
Sec. 4.4.1 presents and analyzes in detail the uncovered ﬁndings.
3.1.2 Optimization Techniques
Optimization techniques including loss functions, optimizers, learning rate schedulers, and hyperparameters are also vital to the performance of a neural network.
PointNet++ uses the same optimization techniques throughout its experiments: CrossEntropy loss, Adam optimizer [16], exponential learning rate decay (Step Decay), and the same hyperparmeters.
Owing to the development of machine learning theory, modern neural networks can be trained with theoretically better optimizers ( e.g .
AdamW [27] vs .
Adam [16]) and more advanced loss functions (CrossEntropy with label smoothing [39]).
Similarly to our study on data augmentations, we also quantify the effect of each modern optimization technique on PointNet++.
We ﬁrst perform a sequential hyperparameter search for the learning rate and weight decay.
We then conduct an additive study on label smoothing, optimizer, and learning rate scheduler.
We discover a set of improved optimization techniques that further
S e t A b s t r ac ti on S e t A b s t r ac ti on
S e t A b s t r ac ti on S e t A b s t r ac ti on
F ea t .
P r op a g a ti on
F ea t .
P r op a g a ti on
F ea t .
P r op a g a ti on
F ea t .
P r op a g a ti on
I nv R e s
M L P
I nv R e s
M L P
I nv R e s
M L P
I nv R e s
M L P
S ub s a m p l e
G r oup i ng
M L P s ( 64 )
R e du c ti on
M L P ( 1024 )
M L P ( 256 )
G r oup i ng
M L P ( 256 )
R e du c ti on
M L P
M L P s ( 128 )
I n t e r po l a t e [N,32] [N/4,64] [N/16,128] [N/64,256] [N/256,512] [N/64,256] concat *
The differences between the proposed
PointNeXt and the classical PointNet++ are highlighted in red borders.
[N/16,128] [N/4,64] [N,32] boost performance by a decent margin.
In general, CrossEntropy with label smoothing, AdamW, and
Cosine Decay can decently optimize models in various tasks.
See Sec.
4.4.1 for detailed ﬁndings.
3.2 Architecture Modernization: Small Modiﬁcations → Big Improvements
In this subsection, we modernize PointNet++ [30] into the proposed PointNeXt.
The modernization consists of two aspects: (1) receptive ﬁeld scaling and (2) model scaling.
3.2.1 Receptive Field Scaling
The receptive ﬁeld is a signiﬁcant factor in the design space of a neural network [38, 7].
There are at least two ways to scale the receptive ﬁeld in point cloud processing: (1) adopting a larger radius to query the neighborhood, and (2) adopting a hierarchical architecture.
Since the hierarchical architecture has been adopted in the original PointNet++, we mainly study (1) in this subsection.
Note that the radius of PointNet++ is set to an initial value r that doubles when the point cloud is downsampled.
We study a different initial value in each benchmark and discover that the radius is dataset-speciﬁc and can have signiﬁcant inﬂuence on performance.
This is elaborated in Sec.
Furthermore, we ﬁnd that the relative coordinates ∆ p = p lj − p li in Eq.
(1) make network optimization harder, leading to a decrease in performance.
Thus, we propose relative position normalization ( ∆ p normalization) to divide relative position by the neighborhood query radius:
[Equation]
[Equation]
Without normalization, values of relative positions ( ∆ p = p lj − p li ) are considerably small (less than the radius), requiring the network to learn a larger weight to apply on ∆ p .
This makes the optimization non-trivial, especially since weight decay is used to reduce the weights of the network and thus tends to ignore the effects of relative position.
The proposed normalization alleviates this issue by rescaling and in the meantime reduces the variance of ∆ p among different stages.
3.2.2 Model Scaling
PointNet++ is a relatively small network, where the encoder consists of only 2 stages in the classiﬁcation architecture and 4 stages for segmentation.
Each stage consists of only 1 SA block, and each block contains 3 layers of MLP.
The model sizes of PointNet++ for both classiﬁcation and segmentation are less than 2 M, which is much smaller compared to modern networks that typically use more than 10 M parameters [43, 28, 32].
Interestingly, we ﬁnd that neither appending more SA blocks nor using more channels leads to a noticeable improvement in accuracy, while causing a signiﬁcant drop in throughput (refer to Sec. 4.4.2), mainly due to vanishing gradient and overﬁtting.
Therefore, in this subsection, we study how to scale up PointNet++ in an effective and efﬁcient way.
We propose an Inverted Residual MLP (InvResMLP) block to be appended after the ﬁrst SA block, per stage, for effective and efﬁcient model scaling.
InvResMLP is built on the SA block and is illustrated at the bottom middle of Fig.
There are three differences between InvResMLP and SA.
(1) A residual connection between the input and the output is added to alleviate the vanishing gradient problem [13], especially when the network goes deeper.
(2) Separable MLPs are introduced to reduce computation and reinforce pointwise feature extraction.
While all 3 layers of MLPs in the original
SA block are computed on the neighborhood features, InvResMLP separates the MLPs into a single layer computed on the neighborhood features (between the grouping and reduction layers) and two layers for point features (after reduction), as inspired by MobileNet [14] and ASSANet [32].
The inverted bottleneck design [36] is leveraged to expand the output channels of the second MLP bytimes to enrich feature extraction.
Appending InvResMLP blocks is proven to signiﬁcantly improve performance compared to the appending of the original SA blocks (see Sec.
In addition to InvResMLP, we present three changes in the macro architecture.
(1) We unify the design of PointNet++ encoder for classiﬁcation and segmentation, i.e ., scaling the number of SA blocks for classiﬁcation from 2 to 4 while keeping the original number (4 blocks) for segmentation at each stage.
(2) We utilize a symmetric decoder in which its channel size is changed to match the encoder.
(3) We add a stem MLP, an additional MLP layer inserted at the beginning of the architecture, to map the input point cloud to a higher dimension.
In summary, we present PointNeXt, the next version of PointNets [29, 52], modiﬁed from PointNet++ by incorporating the proposed InvResMLP and the aforementioned macro-architectural changes.
The architecture of PointNeXt is illustrated in Fig.
We denote the channel size of the stem MLP as
C and the number of InvResMLP blocks as B .
A larger C leads to an increase in the width of the network ( i.e ., width scaling), while a larger B leads to an increase in the depth of the network ( i.e ., depth scaling).
Note that when B = 0 , only one SA block and no InvResMLP blocks are used at each stage.
The number of MLP layers in the SA block is set to 2 , and a residual connection is added inside each SA block.
When B (cid:54) = 0 , InvResMLP blocks are appended after the original SA block.
The number of MLP layers in the SA block in this case is set to 1 to save computation cost.
The conﬁguration of our PointNeXt family is summarized as follows:
• PointNeXt-S: C = 32 , B =• PointNeXt-B: C = 32 , B = (1 , 2 , 1 , 1) • PointNeXt-L: C = 32 , B = (2 , 4 , 2 , 2) • PointNeXt-XL: C = 64 , B = (3 , 6 , 3 , 3)
Experiments
We evaluate PointNeXt on ﬁve standard benchmarks: S3DIS [1] and ScanNet [5] for semantic segmentation, ScanObjectNN [44] and ModelNet40 [49] for object classiﬁcation, and ShapeNetPart [3] for object part segmentation.
Experimental Setups.
We train PointNeXt using CrossEntropy loss with label smoothing [39],
AdamW optimizer [27], an initial learning rate lr = 0 .
001 , weight decay 10 − 4 , with Cosine Decay, and a batch size of 32, with a 32G V100 GPU, for all tasks, unless otherwise speciﬁed.
The best model on the validation set is selected for testing.
For S3DIS segmentation, point clouds are voxel downsampled with a voxel size of 0 .
04 m following common practice [43, 32, 56].
PointNeXt is trained with an initial lr = 0 .
01 , for 100 epochs (training set is repeated by 30 times), using a ﬁxed number of points ( 24 , 000 ) per batch with a batch size of 8 as input.
During training, the input points are obtained by querying the nearest neighbors of a random point in each iteration.
Following Point Transformer [56], we evaluate PointNeXt using the entire voxel-downsampled scene as input.
For ScanNet scene segmentation, we follow the Stratiﬁed Transformer [17] and train PointNeXt with multi-step learning rate decay and decay at [70,90] epochs with a decay rate of 0.1 without label smoothing.
The voxel size is set to 0 .
02 m and input number of points in training is set to 64 , 000 .
We train the model forepochs (training set is repeated for 6 times) with a batch size of 2 per GPU with 8 GPUs.
For ScanObjectNN classiﬁcation, PointNeXt is trained with a weight decay of 0.05 for 250 epochs.
Following
Point-BERT [54], the number of input points is set to 1 , 024 , where the points are randomly sampled during training and uniformly sampled during testing (denoted as point resampled augmentation).
For
ModelNet40 classiﬁcation, PointNeXt is trained similarly as ScanObjectNN but for 600 epochs.
For
ShapeNetPart part segmentation, we train PointNeXt using a batch size of 8 per GPU with 4 GPUs, and Poly FocalLoss [18] as criterion, for 400 epochs.
Following PointNet++, 2,048 randomly sampled points with normals are used as input for training and testing.
The details of data augmentations used in S3DIS, ScanNet, ScanObjectNN, ModelNet40 and ShapeNetPart are detailed in Sec. 4.4.1.
[Table]
For all experiments except ShapeNetPart segmentation, we do not conduct any voting [23] 2 , since it is more standard to compare the performance without using any ensemble methods as suggested by
SimpleView [9].
However, we found that the performance in ShapeNetPart of nearly all models is quite close to each other, where it is hard to achieve state-of-the-art IoUs without voting.
We also provide model parameters (Params.) and inference throughput (instances per second) for comparison.
The throughput of all methods is measured using 128 × 1024 (batch size 128, number of points 1024) as input in ScanObjectNN and ModelNet40 and 64 × 2048 in ShapeNetPart.
In S3DIS, 16 × 15 ,points are used to measure throughput following [32], since some methods [46, 20] could not process the whole scene due to memory constraints.
The throughput of all methods is measured using an
NVIDIA Tesla V100 32GB GPU and a 32 core Intel Xeon @ 2.80GHz CPU.
4.1 3D Semantic Segmentation in S3DIS and ScanNet
S3DIS [1] (Stanford Large-Scale 3D Indoor Spaces) is a challenging benchmark composed of 6 large-scale indoor areas, 271 rooms, and 13 semantic categories in total.
The standard 6-fold cross-validation results in S3DIS are reported in Tab. 1.
Note that the ofﬁcial PointNet++ [30] did not conduct experiments in S3DIS.
Here, we use the results reported by PointCNN [22] for comparison.
Our PointNeXt-S, the smallest variant, outperforms PointNet++ by 13.5%, 6.4%, and 10.2% in terms of mean IoU (mIoU), overall accuracy (OA), and mean accuracy (mAcc), respectively, while being faster in terms of throughput.
The increased speed is due to the reduced number of layers in the SA block for PointNeXt-S (see Sec. 3.2.2).
With the proposed model scaling, the performance of PointNeXt can be gradually boosted.
For example, PointNeXt-L outperforms SOTA
Point Transformer [56] by 0.4% in mIoU while being 3 × faster.
Note that Point Transformer utilizes most of the improved training strategies of ours.
PointNeXt-XL, the extra large variant, achieves mIoU/OA/mAcc of 74.9%/90.3%/83.0%, while running faster than Point Transformer.
As a limitation, our PointNeXt-XL consists of more parameters and is more computationally expensive in terms of
FLOPs, mainly due to channel expansion ( × 4 ) in the inverted bottleneck and doubled initial channel size ( C = 64 ).
We also provide the results of PointNeXt in S3DIS area 5 in the Tab. 1 with mean ± std in three random runs, where PointNeXt achieves similar improvements as the 6-fold experiments.
ScanNet [5], another well-known large-scale segmentation dataset, contains 3D indoor scenes of various rooms with 20 semantic categories.
We follow the public training, validation, and test splits, with 1201, 312 and 100 scans, respectively.
For PointNet++, we use the results reported from the
Stratiﬁed Transformer [17] for comparison.
As shown in Tab. 1, we improve PointNet++ from 53 . 5% mIou to 57 .
2% mIoU in the validation set by adopting the improved training strategies (detailed in supplementary material).
PointNeXt-S further gains +11 .
in val mIoU over the original PointNet++ mostly due to the use of a smaller radius ( 0 . 1 m → 0 . 05 m) and relative position normalization.
The performance in ScanNet improves steadily with the increase in model sizes.
Our largest variant,
[Table]
PointNeXt-XL outperforms PointNet++ by 18 . 0% mIoU in validation and achieves 71 .
2% mIoU in testing, beating the recent methods Point Transformer [56] and CBL [41].
4.2 3D Object Classiﬁcation in ScanObjectNN and ModelNet
ScanObjectNN [44] contains about 15 , 000 real scanned objects that are categorized into 15 classes with 2 , 902 unique object instances.
Due to occlusions and noise, ScanObjectNN poses signiﬁcant challenges to existing point cloud analysis methods.
Following PointMLP [28], we experiment on PB_T50_RS, the hardest and most commonly used variant of ScanObjectNN.
As reported in
Tab. 2, the proposed PointNeXt-S surpasses existing methods by non-trivial margins in terms of both OA and mAcc, while using much fewer model parameters and running much faster.
Built upon PointNet++ [30], PointNeXt achieves signiﬁcant improvements over the originally reported performance of PointNet++, i.e. +9.8% OA and +10.4% mACC.
This demonstrates the efﬁcacy of the proposed training and model scaling strategies.
PointNeXt also outperforms SOTA PointMLP [28] ( i.e. +2.3% OA, +1.9% mACC), while running 10 × faster.
This shows that PointNeXt is a simple, yet effective, and efﬁcient baseline.
Note that we did not experiment with upscaled variants of PointNeXt on this benchmark, since we found that the performance had saturated using PointNeXt-S mostly due to the limited scale of the dataset.
ModelNet40 [49] was a commonly used 3D object classiﬁcation dataset, which has 40 object categories, each of which contains 100 unique CAD models.
However, recent works [12, 28, 34] show an increasing interest in the real-world scanned dataset ScanObejectNN compared to this synthesized dataset.
Following this trend, we mainly benchmarked PointNeXt in ScanObjectNN.
Here
[Table]
we also provide our results in ModelNet40.
Tab.
2 shows that advanced training strategies improve PointNet++ from 91 .
9 % OA to 92 .
8 % OA without any architecture change.
PointNeXt-S ( C = 32 ) outperforms the original reported PointNet++ by 1 . 3 % OA, while being faster.
Note that
PointNeXt-S with a larger width C = 64 can achieve a higher overall accuracy ( 94 . 0 %)
[Table]
4.3 3D Object Part Segmentation in ShapeNetPart
[Table]
ShapeNetPart [53] is a widely-used dataset for object-level part segmentation.
It consists of 16 ,models from 16 different shape categories, 2 - 6 parts for each category, and 50 part labels in total.
As shown in Tab. 3, our
PointNeXt-S with default width ( C = 32 ) obtains a performance comparable to that of the SOTA CurveNet [50] and outperforms a large number of representative networks, such as KPConv [43] and ASSANet [32] in terms of both instance mean IoU (ins.
mIoU) and throughput.
Due to the small scale of ShapeNetPart, the model would overﬁt after being depth scaled.
However, we ﬁnd by increasing the width from 32 to 64 instead, PointNeXt can outperform
CurveNet, while being over 4 × faster.
It is also worth highlighting that PointNeXt with an even larger width ( C = 160 ) reaches 87 .
Ins.
mIoU, whereas the performance of point-based methods has saturated below this value for years.
We highlight that we used voting only in ShapeNetPart by averaging the results of 10 randomly scaled input point clouds, with scaling factors equal to [0.8,1.2].
Without voting, we notice a performance drop around 0 .
5 instance mIoU.
4.4 Ablation and Analysis
Tab. 4 and Tab.
5 present additive studies for the proposed training and scaling strategies in
ScanObjectNN [44] and S3DIS [1], respectively.
We adopt the original PointNet++ as the baseline.
In ScanObjectNN, PointNet++ was trained by [44] with CrossEntropy loss, Adam optimizer, a learning rate 1e-3, a weight decay of 1e-4, a step decay of 0.7 for every 20 epochs, and a batch size of 16, for 250 epochs, while using random rotation and jittering as data augmentations.
The ofﬁcial PointNet++ did not conduct experiments in S3DIS dataset.
We refer to the widely used reimplementation [52], where PointNet++ was trained with the same settings as ScanObjectNN except that only random rotation was used as augmentation.
Note that for all experiments, we train all our models for 250 epochs in ScanObjectNN and for 100 epochs in S3DIS.
4.4.1 Training Strategies
Data augmentation is the ﬁrst aspect that we study to modernize PointNet++.
We draw four conclusions based on observations in Tab. 4 and 5.
(1) Data scaling improves performance for both classiﬁcation and segmentation tasks.
For example, point resampling is shown to boost the performance by 2 . 5 % OA in ScanObjectNN.
Taking the entire scene as input instead of using the block or sphere subsampled input as done in PointNet++ [30] and other previous works [43, 21, 32] improves the segmentation result by 1 . 1 % mIoU. (2)
Height appending improves performance, especially for object classiﬁcation.
Height appending makes the network aware of the actual size of the objects, thus leading to an increase in accuracy ( +1 . 1 % OA).
(3) Color drop is a strong augmentation that signiﬁcantly improves the performance of tasks where colors are available.
Adopting color drop alone adds 5.9% mIoU in S3DIS area 5.
We hypothesize that color drop forces the network to focus more on the geometric relationships between points, which in turn improves performance.
(4) Larger models favor stronger data augmentation.
Whereas random rotation drops the performance of PointNet++ by 0 .
3 % mIoU in S3DIS (2 nd row in Tab.
5 data augmentation part), it is shown to be beneﬁcial for larger-scale models ( e.g . raises 1 . 5 % mIoU on PointNeXt-B).
Another example in ScanObjectNN shows that the removal of random jittering also adds 1.1% OA.
In general, with the improved data augmentations, the OA of PointNet++ in ScanObjectNN and the mIoU in
S3DIS area 5 are increased by 5 . 8 % and 9 .
5 %, respectively.
[Table] [Table]
Optimization techniques involve loss functions, optimizers, learning rate schedulers, and hyperparameters.
As shown in Tab. 4 and 5, Label Smoothing, AdamW [27] optimizer, and Cosine Decay consistently boost performance in both classiﬁcation and segmentation tasks.
This reveals that the more developed optimization methods such as label smoothing and AdamW are generally good for optimizing a neural network.
Compared to Step Decay, Cosine Decay is also easier to tune (usually only the initial and minimum learning rates are required) and can achieve a performance similar to
Step Decay.
Regarding hyperparameters, using a learning rate greater than that used in PointNet++ improves the segmentation performance in S3DIS.
In general, our training strategies consisted of stronger data augmentation and modern optimization techniques can increase the performance of PointNet++ from 77 .
9 % to 86 .
1 % OA in ScanObjectNN dataset, impressively surpassing SOTA PointMLP by 0 .
The mIoUs in S3DIS area 5 and S3DIS 6-fold (illustrated in Fig. 1) are boosted by 11 . 7 and 13 .
6 absolute percentage points, respectively.
Our observations imply that a signiﬁcant portion of the performance gap between classical PointNet++ and SOTA is due to the training strategies.
[Table]
Generalize to other networks.
Although the training strategies are proposed for PointNet++ [30], we ﬁnd that they can be applied to other methods such as PointNet [29], DGCNN [46], and PointMLP [28], and also improve their performance.
Such generalizability is validated in
ScanObjectNN [44].
As shown in Tab. 6, the OA of the representative methods can all be improved when equipped with our training strategies.
4.4.2 Model Scaling
Receptive ﬁeld scaling includes both radius scaling and normalizing ∆ p deﬁned in Eqn.
(2), which are also validated in Tab. 4 and 5.
The radius is dataset speciﬁc, while down-scaling the radius from 0 .
2 to 0 .
15 improves 0 .
3 % OA in ScanObjectNN, keeping the radius the same as 0 .
1 achieves the best performance in S3DIS.
Regarding normalizing ∆ p , it improves the performance in ScanObjectNN and S3DIS by 0 .
3 OA and 0 .
4 mIoU, respectively.
Furthermore, in Tab. 7, we show that normalizing ∆ p has a larger impact ( 2 . 3 mIoU in S3DIS dataset) on the bigger model PointNext-XL.
Model scaling scales PointNet++ by the proposed InvResMLP and some macro-architectural changes (see Sec. 3.2.2).
In Tab. 4, we show that PointNeXt-S using the stem MLP, the symmetric decoder, and the residual connection in the SA block improves 1 . 0 % OA in ScanObjectNN.
Performance in the large-scale S3DIS dataset can be further unveiled (from 63 .
8 % to 70 .
5 % mIoU) by up-scaling PointNeXt-S using more blocks of the proposed InvResMLP, as demonstrated in Tab.
Furthermore, in Tab. 7, we ablate each component of the proposed InvResMLP
[Table]
block and different stage ratios in S3DIS area 5 using the best-performed model
PointNeXt-XL as the baseline.
As observed, each architectural change indeed contributes to increased performance.
Among all changes, the residual connection is the most essential, without which the mIoU will drop from 70 .
5 % to only 64 . 0 %.
The separable
MLPs increase 3 . 9 % mIoU while speeding up the network 3 times.
Removing the inverted bottleneck from the baseline leads to a drop of 1 . 5 % mIoU with less than a 1% gain in speed.
Adding more blocks inside each stage after removing inverted bottleneck can improve its performance to 69 .
7 ± 0 . 3 but is still lower than the baseline.
Tab. 7 also shows the performance of naive width scaling that increases the width of PointNet++ from 32 to 256 to match the throughput of PointNeXt-XL, naive depth scaling to append more SA blocks in PointNet++ to obtain the same number of blocks of PointNextXL whose B = (3 , 6 , 3 , 3) , and naive compound scaling to double the width of the naive depth scaled model to the same width as PointNeXt-XL ( C = 64 ).
Our proposed model scaling strategy achieves much higher performance than these naive scaling strategies, while being much faster.
Related Work
Point-based methods process point clouds directly using their unstructured format compared to voxel-based methods [10, 4] and multi view-based methods [37, 12, 9].
PointNet [29], the pioneering work of point-based methods, proposes to model the permutation invariance of points with shared
MLPs by restricting feature extraction to be pointwise.
PointNet++ [30] is presented to improve
PointNet by capturing local geometric structures.
Currently, most point-based methods focus on the design of local modules.
[46, 45, 31] rely on graph neural networks.
[51, 22, 43, 42] project point clouds onto pseudo grids to allow for regular convolutions.
[48, 23, 24] adaptively aggregate neighborhood features through weights determined by the local structure.
In addition, very recent methods leverage Transformer-like networks [56, 17] to extract local information through self-attention.
Our work does not follow this trend in local module design.
In contrast, we shift our attention to another important but largely under-explored aspect, i.e. , the training and scaling strategies.
Training strategies are studied recently in [2, 47, 26] on image classiﬁcation.
In the point cloud domain, SimpleView [9] is the ﬁrst work to show that training strategies have a large impact on the performance of a neural network.
However, SimpleView simply adopts the same training strategies as DGCNN [46].
On the contrary, we conducted a systematic study to quantify the effect of each data augmentation and optimization technique, and propose a set of improved training strategies that boost the performance of PointNet++ [30] and other representative works [29, 46, 28].
Model scaling can signiﬁcantly improve the performance of a network, as shown in pioneering works in various domains [40, 55, 21].
Compared to PointNet++ [30] that uses parameters less than 2 M, most current prevailing networks consist of parameters greater than 10 M, such as KPConv [43] ( 15 M) and PointMLP [28] ( 13 M).
In our work, we explore model scaling strategies that can scale up PointNet++ in an effective and efﬁcient manner.
We offer practical suggestions on scaling technologies that improve performance, namely using residual connections and an inverted bottleneck design, while maintaining throughput by using separable MLPs.
Conclusion and Discussion
In this paper, we demonstrate that with improved training and scaling strategies, the performance of
PointNet++ can be increased to exceed the current state of the art.
More speciﬁcally, we quantify the effect of each data augmentation and optimization technique that are widely used today, and propose a set of improved training strategies.
These strategies can be easily applied to boost the performance of PointNet++ and other representative works.
We also introduce the Inverted Residual MLP block into PointNet++ to develop PointNeXt.
We demonstrate that PointNeXt has superior performance and scalability over PointNet++ on various benchmarks while maintaining high throughput.
This work aims to guide researchers toward paying more attention to the effects of training and scaling strategies and motivate future work in this direction.
Limitation.
Even though PointNeXt-XL is one of the largest models among all representative pointbased networks [30, 43, 15, 56], its number of parameters ( 44 M) is still below that of small networks in image classiﬁcation such as Swin-S [25] ( 50 M), ConNeXt-S [26] ( 50 M), and ViT-B [8] ( 87 M), and is far from their large variants, including Swin-L ( 197 M), ConvNeXt-XL ( 350 M), and ViT-L ( 305 M).
In this work, we do not push the model size further, mainly due to the smaller-scale nature of point cloud datasets compared to their larger image counterparts, such as ImageNet [6].
Moreover, our work is limited to existing modules since the focus is not on introducing new architectural changes.
Acknowledgement
The authors would like to thank the reviewers of NeurIPS’22 for their constructive suggestions.
This work was supported by the King Abdullah University of Science and
Technology (KAUST) Ofﬁce of Sponsored Research through the Visual Computing Center (VCC) funding, as well as, the SDAIA-KAUST Center of Excellence in Data Science and Artiﬁcial Intelligence (SDAIA-KAUST AI).
A Detailed Description for Manuscript Tab.
Naive width scaling increases the channel size of PointNet++ from 32 to 256 to match the throughput of the baseline model, PointNeXt-XL.
Naive depth scaling refers to appending more SA blocks ( B = (3 , 6 , 3 , 3) , the same as PointNext-XL) in PointNet++.
Furthermore, naive compound scaling doubles the width of naive depth scaled model to the same as PointNeXt-XL ( C = 64 ).
Compared to the PointNet++ trained with improved training strategies ( 63 .
2 % mIoU, 186 ins./sec.), naive depth scaling ( 63 .
4 % mIoU, 53 ins. / sec.) and naive width scaling ( 59 .
4 % mIoU, 43 ins./sec.)
only lead to a large overhead in throughput with insigniﬁcant improvement in accuracy.
In contrast, our proposed model scaling strategy achieves much higher performance than the naive scaling strategies while being much faster.
This can be observed by comparing PointNeXt-XL ( 70 . 5 % mIoU, 45 ins./sec.) to the naive compound scaled PointNet++ ( 62 .
3 % mIoU, 24 ins./sec.).
B Training Strategies Comparison
In this section, we summarize the training strategies used in representative point-based methods such as DGCNN [46], KPConv [43], PointMLP [28], Point Transformer [56], Stratiﬁed Transformer [17],
PointNet++ [30], and our PointNeXt on S3DIS [1] in Tab.
I, on ScanObjectNN [44] in Tab. II, on
ScanNet [5] in Tab. III, and on ShapeNetPart [53] in Tab.
IV, respectively.
C Qualitative Results
We provide qualitative results of PointNeXt-XL for S3DIS (Fig. II) and PointNeXt-S ( C = 160 ) for ShapeNetPart (Fig. III).
The qualitative results of PointNet++ trained with the original training strategies are also included in the ﬁgures for comparison.
On both datasets, PointNeXt produces predictions closer to the ground truth compared to PointNet++.
More speciﬁcally, on S3DIS shown in (Fig. II), PointNeXt is able to segment hard classes, including doors ( 1 st , 3 rd , and 4 th rows), clutter ( 1 st and 3 rd rows), chairs ( 2 nd row), and the board ( 4 th row), while PointNet++ fails to segment properly to some extent.
On ShapeNetPart (Fig. III), PointNeXt precisely segments wings of an airplane ( 1 st row), microphone of an earphone( 2 nd row), body of a motorbike( 3 rd row), ﬁn of a rocket( 4 th row), and bearing of a skateboard ( 5 th row).
D Classiﬁcation Architecture
As illustrated in Fig.
I, the classiﬁcation architecture shares the same encoder as the segmentation one.
The output features of the encoder are passed to a global pooling layer ( i.e . global max-pooling) to acquire a global shape representation for classiﬁcation.
Note that the points are only downsampled by a factor of 2 in each stage, since the number of input points in classiﬁcation tasks is usually small, e.g . 1024 or 2048 points.
[Table]
[Table]
E Societal Impact
We do not see an immediate negative societal impact from our work.
We notice that the way we discover the improved training and scaling strategies may consume a little more computing resources and affect the environment.
Nevertheless, the improved training and scaling strategies will make researchers pay more attention to aspects other than architectural changes
[Table]
which in the long term makes research in computer vision more diverse and generally better
[Table]
[Table]
for ShapeNetPart segmentation.
Method
DGCNN
KPConv
PointNet++
PointNeXt (Ours)
Epochs
Batch size
Optimizer
Adam
SGD
Adam
AdamW
LR 3 × 10 −1 × 10 −1 × 10 −0 .
LR decay step step step multi step
Weight decay 0.
10 −0.
10 −
Label smoothing ε (cid:55) (cid:55) (cid:55) (cid:55)
Random rotation (cid:55) (cid:55) (cid:55) (cid:51)
Random scaling (cid:55) [0.9,1.1] (cid:55) [0.8,1.2]
Random translation (cid:55) (cid:55) (cid:55) (cid:55)
Random jittering (cid:55) 0.
(cid:51) 0.
Normal Drop (cid:55) (cid:55) (cid:55) (cid:51)
Height appending (cid:55) (cid:51) (cid:55) (cid:51) mIoU (%) 85.
S e t
A b s t r ac ti on
S e t
A b s t r ac ti on
S e t
A b s t r ac ti on
S e t
A b s t r ac ti on
I nv
R e s
M L P
I nv
R e s
M L P
I nv
R e s
M L P
I nv
R e s
M L P
M L P [N,32] [N/2,64]
[N/4,128] [N/8,256] [N/16,512]
G l ob a l P oo li ng
PointNet++
PointNeXt
Ground Truth
[Figure]
[Figure]