Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
Anonymous Author(s)
AffiliationAddress email
Abstract
Masked Autoencoders (MAE) have shown great potentials in self-supervised pretraining for language and 2D image transformers.
However, it still remains anopen question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds.
In this paper, we propose Point-M2AE , a strong
M ulti-scale M AE pre-training framework for hierarchical self-supervised learningof 3D point clouds.
Unlike the standard transformer in MAE, we modify theencoder and decoder into pyramid architectures to progressively model spatialgeometries and capture both fine-grained and high-level semantics of 3D shapes.
For the encoder that downsamples point tokens by stages, we design a multi-scalemasking strategy to generate consistent visible regions across scales, and adopta local spatial self-attention mechanism to focus on neighboring patterns.
Bymulti-scale token propagation, the lightweight decoder gradually upsamples pointtokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective.
Extensive experimentsdemonstrate the state-of-the-art performance of Point-M2AE for 3D representationlearning.
With a frozen encoder after pre-training, Point-M2AE achieves 92.9%accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods.
By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracyon ScanObjectNN, +3.36% to the second-best, and largely benefits the few-shotclassification, part segmentation and 3D object detection with the hierarchicalpre-training scheme.
Introduction
Learning to represent from unlabeled data without annotations, known as self-supervised learning,has attained great success in natural language processing [10, 31, 32, 5], computer vision [19, 7,8, 18] and multi-modality learning [30, 49, 21].
By pre-training on the large-scale raw data, thenetworks are endowed with robust representation abilities and can significantly benefit downstreamtasks with fine-tuning.
Motivated by masked language modeling [31, 10], MAE [18] and someother methods [45, 52, 3] adopt asymmetric encoder-decoder transformers [13] to apply maskedautoencoding for self-supervised learning on 2D images.
They represent the input image as multiplelocal patches, and randomly mask them with a high ratio to build the pretext task for reconstruction.
Specifically, the encoder aims at capturing high-level latent representations from limited visiblepatches, and the lightweight decoder is forced to reconstruct the RGB values of masked patches ontop.
Despite its superiority on grid-based 2D images, we ask the question: can MAE-style maskedautoencoding be adapted to irregular point clouds as a powerful 3D representation learner?
[Figure]
Masking
[Figure]
SkipConnections
MAE
Point-M2AE
Multi-scaleMasking
Reconstructed3DCoordinates
Reconstructed2DPixels
PlainEncoder
PlainDecoder
Hierarchical
Hierarchical Encoder
Decoder
U p s a m p l e D o w n s a m p l e
ConstantResolution
To tackle this challenge, we propose M ulti-scale M asked autoencoders for learning the hierarchicalrepresentations of point clouds via self-supervised pre-training, termed as Point-M2AE.
We representa point cloud as a set of point tokens depicting different spatial local regions, and inherit MAE’spipeline to first encode visible point tokens and then reconstruct the masked 3D coordinates.
Differentfrom 2D images, masked autoencoding for 3D point clouds has three characteristics to be considered. Firstly, it is critical to understand the relations between local parts and the overall 3D shapes, whichhave strong geometric and semantic dependence. As examples, the network can recognize an airplanestarting from its wing, or segment the wing’s part from the airplane’s global feature. Therefore, weregard the standard transformer with the plain encoder and decoder is sub-optimal for capturingsuch local-global spatial relations in 3D, which directly downsamples the input into a low-resolutionrepresentation as shown in Figure 1 (Top).
We modify both the encoder and decoder into multistage hierarchies for progressively encoding multi-scale features of point clouds, constructing anasymmetric U-Net [34] like architecture in Figure 1 (Bottom).
In detail, the shallower stages of theencoder contain a larger number of point tokens to focus on local patterns, while the deeper stagesmerge spatially adjacent tokens to acquire global understanding.
Secondly, as Point-M2AE encodesmulti-scale point clouds unlike the single-scale 2D images, the unmasked visible regions are requiredto be block-wise within one scale and consistent across scales, which are respectively for reservingmore complete local geometries and ensuring coherent feature learning for the network.
For this, weintroduce a multi-scale masking strategy, which generates random masks at the final scale with ahigh ratio (e.g., 80%), and back-projects the unmasked positions to all preceding scales.
Thirdly, tobetter capture the fine-grained 3D geometries, we adopt a local spatial self-attention mechanism withincreasing attention scopes for point tokens at different stages in the encoder, which refocus eachtoken within neighboring detailed structures.
Also, we utilize skip connections to complement thedecoder with fine-grained information from the corresponding stages of the encoder.
By the multi-scale pre-training, Point-M2AE can encode point clouds from local-to-global hierarchies and then reconstructs the masked coordinates from global-to-local perspectives, whichlearns powerful 3D representations and performs superior transfer ability.
After self-supervisedpre-training on ShapeNet [6], Point-M2AE achieves 92.9% classification accuracy for linear SVMon ModelNet40 [43] with the frozen encoder, which surpasses the runner-up CrossPoint [2] by+1.2% and even outperforms some fully supervised methods.
By fine-tuning on various downstreamtasks, Point-M2AE achieves 86.43% (+3.36%) accuracy on ScanObjectNN [37] and 94.0% (+0.8%)accuracy on ModelNet40 [43] for shape classification, 86.51% (+0.91%) instance mIoU on ShapeNetPart [47] for part segmentation, and 95.0% (+2.7%) accuracy on 10-way 20-shot ModelNet40 for few-shot classification.
Our multi-scale masked autoencoding also benefits the 3D object {+detectionon ScanNetV
+} [9] by +1.3% AP 25 and +1.3% AP 50 , which provides the detection backbone witha hierarchical understanding of the point clouds.
We summarize the contributions of our paper asfollows:
We propose Point-M2AE, a strong masked autoencoding framework, which conducts hierarchical point cloud encoding and reconstruction for better learning multi-scale spatialgeometries of 3D shapes.2.
We introduce a U-Net like transformer architecture for MAE-style pre-training on pointclouds, and adopt a multi-scale masking strategy to generate consistent visible regions acrossscales.3.
Point-M2AE achieves state-of-the-art performance for transfer learning on various downstream tasks, which indicates our approach to be a powerful representation learner for 3Dpoint clouds.
Related Work
Pre-training by Masked Modeling.
Compared to contrastive learning methods [19, 7, 8] that learnfrom inter-sample relations, self-supervised pre-training by masked autoencoding builds the pretexttasks to predict the masked parts of the input signals.
The series of GPT [31, 32, 5] and BERT [11]apply masked modeling to natural language processing and achieve extraordinary performanceboost on downstream tasks with fine-tuning.
Inspired by this, BEiT [4] proposes to match imagepatches with discrete tokens via dVAE [33] and pre-train a standard vision transformer [13, 48]by masked image modeling.
On top of that, MAE [18] directly reconstructs the raw pixel valuesof masked tokens and performs great efficiency with a high mask ratio.
The follow-up worksfurther improve the performance of MAE by momentum encoder [52], contrastive learning [3], andmodified reconstruction targets [41].
For self-supervised pre-training on 3D point clouds, the maskedautoencoding has not been widely adopted.
Similar to BEiT, Point-BERT [48] utilizes dVAE to map3D patches to tokens for masked point modeling, but heavily relies on constrastive learning [19],complicated data augmentation, and the costly two-stage pre-training.
In contrast, our Point-M2AEis a pure masked autoencoding method of one-stage pre-training, and follows MAE to reconstruct theinput signals without dVAE mapping.
Different from previous MAE methods adopting standard plaintransformer, we propose a hierarchical transformer architecture along with the multi-scale maskingstrategy to better learn a strong and generic representation for 3D point clouds.
Self-supervised Learning for Point Clouds.
3D representation learning without annotations hasbeen widely studied in recent years.
Mainstream methods mainly build the pretext tasks to reconstructthe transformed input point cloud based on the encoded latent vectors, such as rotation [27], deformation [1], rearranged parts [35] and occlusion [39].
From another perspective, PointContrast [44]utilizes contrastive learning between features of the same points from different views to learn discriminative 3D representations.
DepthContrast [50] further extends the contrast for depth maps of differentaugmentations.
CrossPoint [2] conducts cross-modality contrastive learning between point cloudsand their corresponding rendering images to acquire rich self-supervised signals.
Point-BERT [48]first introduces BERT-style pre-training for 3D point clouds with a standard transformer network andperforms competitively on various downstream tasks.
In this paper, we propose an MAE-style [18]pre-training framework, Point-M2AE, which reconstructs the highly masked 3D coordinates of theinput point cloud for self-supervised learning.
Point-M2AE with a hierarchical architecture achievesstate-of-the-art downstream performance by learning the multi-scale representation of point clouds.
Method
The overall pipeline of Point-M2AE is shown in Figure 2, where we encode and reconstruct the pointcloud by a hierarchical network architecture.
In Section 3.1, We first introduce the masking strategyof Point-M2AE with multi-scale representations of point clouds.
Then in Section 3.2 and Section 3.3,we present the details of our encoder and decoder with multi-stage hierarchies.
[Figure]
{𝐻 %& ,𝐻 %$ } {𝐻 !"
#& ,𝐻 !"
#$ }
[Figure] [Figure]
[Figure]
𝑃 %→#
[Figure] [Figure]
3.1 Multi-scale Masking
To build a U-Net [34] like masked autoencoder for hierarchical learning, we encode the point cloudby S scales with different number of points at each scale, and correspondingly modify the standardplain encoder into the S -stage architecture.
Following MAE, we embed the point cloud into discretepoint tokens and randomly mask them for reconstruction.
Importantly, for irregular-distributed pointsin the multi-scale architecture, the unmasked visible spatial regions are required to be consistentnot only within one scale, but also across different scales.
This is because the block-wise parts of3D shapes tend to preserve more complete fine-grained geometries, and the unmasked positions arebetter to be shared across all scales for coherent feature learning of the encoder.
Therefore, as shownin Figure 3, we first construct the S -scale coordinate representations of the input point cloud andback-project the random masks from the final S -th scale to the earlier scales to avoid fragmentedvisible parts.
S -scale Representations.
We denote the input point cloud as P ∈ R N × 3 and regard it as the 0 -thscale.
For the i -th scale, 1 ≤ i ≤ S , we utilize Furthest Point Sampling (FPS) to downsample thepoints from the ( i − 1) -th scale, which produces seed points P i ∈
R N i × 3 for scale i of N i points.
Then, we adopt k Nearest-Neighbour ( k -NN) to aggregate the neighboring k points for each seedpoint and obtain the neighbor indices I i ∈
R N i × k .
By successively downsampling and grouping, weacquire the S -scale representations { P i , I i } S i =1 of the input point cloud, where the number of points
N i gradually decreases and the inclusion relations between scales are recorded in I i .
Back-projecting Visible Positions.
For seed points P S at the final S -th scale, we randomly maskthem with a large proportion (e.g., 80%) and denote the remaining visible points as P vS ∈ R N vS × 3of N S points.
We then back-project the unmasked positions P vS to ensure the consistent visibleregions across scales.
For the i -th scale, 1 ≤ i < S , we retrieve all the k nearest neighbors of
P vi +1 from the indices
I i +1 to serve as the visible positions P vi , and mask the others.
By recursivelyback-projecting, we obtain the visible and masked positions of all S scales, denoted as { P vi , P mi } Si =1 ,where P vi ∈ R N vi × 3 , P mi ∈ R N mi × 3 and N i = N vi + N mi .
3.2 Hierarchical Encoder
Based on the multi-scale masking, we embed the initial tokens of visible points P v 1 for the 1 -st scaleand them into the hierarchical encoder with S stages.
Every stage is equipped with K stacked encoderblocks, and each block contains a local spatial self-attention layer and a Feed Forward Network (FFN)of MLP layers.
Between every two consecutive stages, we introduce spatial token merging modulesto aggregate adjacent visible tokens and enlarge receptive fields for downsampling the point clouds.
𝑆
[Figure]
[Figure]
FPS, 𝑘 - NN
Back - project …
Maskatthe
S - thS cale 𝑃 !"
𝑃 #"
FPS, 𝑘 - NN {𝑃 $ , 𝐼 $ } {𝑃 ! , 𝐼 ! } {𝑃 # , 𝐼 # } 𝐼 ! …
𝐼 "
[Figure]
Token Embedding and Merging.
Indexed by I 1 , we utilize a mini-PointNet [28] to extract and fuse the features of every seed point from P v 1 ∈ R N v 1 × 3 with its k nearest neighbors.
After that,we obtain the initial point tokens T v 1 ∈ R N v 1 × C 1 for the 1-st stage of the encoder, which embeds
N e 1 local patterns of the 3D shape.
Between the ( i − 1) -th
and i -th stages, i S , we merge
R N to to which
N
Due to
, the
Local Spatial Self-Attention.
For smaller i of shallower stages, we expect each token to mainlyfocus on finer-grained information and not to be disturbed by long-range signals.
Thus, we modifythe original self-attention layer by a local spatial constraint that only neighboring tokens within aball query [29] would be available for attention calculation.
As the point tokens are downsampledby stages, we set increasing radii { r i } Si =1 of multi-scale ball queries for gradually expanding theattention scopes, which fulfills the local-to-global feature aggregation scheme.
3.3 Hierarchical Decoder
Via the hierarchical encoder, we obtain the encoded visible tokens { T vi } Si =1 of all scales.
Startingfrom the highest S -th scale, we assign a shared learnable mask token to all the masked positions P mS ,and concatenate them with the visible tokens T vS .
We denote them as { H v 1 , H m 1 } with coordinates{ P vS , P mS } , which serve as the input of the hierarchical decoder.
We design the decoder to belightweight with S − 1 stages and only one decoder block for each stage, which enforces the encoderto embed more semantics of the point clouds.
Each decoder block consists of a vanilla self-attentionlayer and an FFN.
We do not apply the local constraint to the attention in the decoder, since a globalunderstanding between visible and mask tokens is crucial to the reconstruction.
Point Token Upsampling.
We upsample the point tokens between stages to progressively recoverthe fine-grained geometries of 3D shapes before reconstruction.
We regulate that the j -th stage ofthe decoder corresponds to the ( S + 1 − j ) -th stage of the encoder, both of which contain pointtokens of the same ( S + 1 − j ) -th scale with the feature dimension C S +1 − j .
Between the ( j − 1) -th and j -th stage, 1 < j ≤ S − 1 , we upsample the tokens { H vj − 1 , H mj − 1 } from the coordinates{ P vS +2 − j , P mS +2 − j } into { P vS +1 − j , P mS +1 − j } via the token propagation module.
Specifically, weobtain the k nearest neighbors of each point token in { H vj − 1 , H mj − 1 } indexed by I S +2 − j , and recovertheir neighbors’ features by weighted interpolation referring to PointNet++ [29], which generates thetokens { H vj , H mj } of the j -th stage.
Skip Connections.
To further complement the fine-grained geometries, we channel-wisely concatenate the visible tokens H vj ∈ R N S +1 − j × C S +1 − j of the decoder with T vS +1 − j ∈ R N S
+1 − j × C S +1 − jfrom the corresponding ( S + 1 − j ) -th stage of the encoder via skip connections, and adopt a linearprojection layer to fuse their features.
For the mask tokens H mj , we keep them unchanged, since theencoder only contains visible tokens without the masked ones.
[Table]
[Table]
Point Reconstruction.
After S − 1 stages of the decoder
[Table]
we acquire { H vS − 1 , H mS − 1 } with coordinates { P v 2 , P m 2 } and reconstruct the masked values from the mask tokens H mS − 1 .
Other thanpredicting values at the 0 -th scale of the input point cloud P , we reconstruct the coordinates of
P m 1 , namely, recovering the masked positions of the 1 -st scale P m 1 ∈ R N m 1 × 3 from the 2 -nd scale
P m 2 ∈ R N m 2 × 3 .
This is because { P v 1 , P m 1 } of the 1-st scale could well represent the overall 3Dshape and simultaneously preserve enough local patterns, which already constructs a comparativelychallenging pretext task for pre-training.
If we further upsample { H vS − 1 , H mS − 1 } into { H vS , H mS }and reconstruct the masked raw points from P m 1 , the extra spatial noises and computational overhead would adversely influence our performance and efficiency.
Therefore, for every token in
H mS − 1 ∈ R N m 2 × C 2 , we reconstruct its k nearest neighbors recorded in I 2 by a reconstruction head ofone linear projection layer and compute the loss by l 2 Chamfer Distance [14], formulated as,(cid:98)
[Equation]
Linear(
[Equation]
1 ) , where (cid:98)
[Equation]
where (cid:98) P m 2 → 1 and P m 2 → 1 denote the predicted and ground-truth reconstruction coordinates from the2 -nd scale to the 1 -st scale.
We only utilize L CD for supervision without contrastive loss to conducta pure masked autoencoding for self-supervised pre-training.
Experiments
In Section 4.1 and Section 4.2, we introduce the pre-training experiments of Point-M2AE and reportthe fine-tuning performance on various downstream tasks.
We also conduct ablation studies in
Section 4.3 to validate the effectiveness of our approach.
4.1 Self-supervised Pre-training
Settings.
We pre-train our Point-M2AE on ShapeNet [6] dataset, which contains 57,448 synthetic3D shapes of 55 categories.
We set the stage number S as 3, and construct a 3-stage encoder and a2-stage decoder for hierarchical learning.
We adopt 5 blocks in each encoder stage, but only 1 blockper stage for the lightweight decoder.
For the 3-scale point cloud, we set the point numbers, tokendimensions, and radii of the local spatial attention layers respectively as {512, 256, 64}, {96, 192,384} and {0.32, 0.64, 1.28}.
We also set different k for the k -NN at different scales, which are {16,8, 8}.
We mask the highest scale of point clouds with a high ratio of 80% and set 6 heads for all theattention modules.
The detailed training settings are in Appendix.
[Table]
Linear SVM.
After pre-training on ShapeNet, we test the 3D representation capability of PointM2AE via linear evaluation on ModelNet40 [43].
We sample 1,024 points from each 3D shapeof ModelNet40 and utilize our frozen encoder to extract their features.
On top of that, we traina linear SVM and report the classification accuracy in Table 1.
As shown, Point-M2AE achievesthe best performance among all existing self-supervised methods for point clouds, and surpassesthe second-best CrossPoint [2] by +1.7%.
Point-M2AE also exceeds Point-BERT [48] by +5.5%,which is a masked point modeling method with a MoCo loss [19] but adopts a standard transformerand conducts single-scale learning.
It is worth noting that even if we freeze all our parameters,
Point-M2AE with 92.9% accuracy still outperforms many fully trained methods on ModelNet40, e.g.,90.5% by PointNet++ [29], 92.8% by DensePoint [24], etc.
The experiments fully demonstrate thesuperior 3D representation capacity of our Point-M2AE.
4.2 Downstream Tasks
For fine-tuning on downstream tasks, we discard the hierarchical decoder in pre-training and appenddifferent heads onto the hierarchical encoder for different tasks.
Shape Classification.
We fine-tune Point-M2AE on two shape classification datasets: the widelyadopted ModelNet40 [43] and the challenging ScanObjectNN [37].
We follow Point-BERT to usethe voting strategy [25] for fair comparison on ModelNet40, which tests the model for several timeswith different point cloud augmentation and ensembles the predictions.
To handle the noisy spatialstructures, we increase k of k -NN into {32, 16, 16} for ScanObjectNN to encode local patterns withlarger receptive fields.
As reported in Table 2, Point-M2AE achieves 94.0% accuracy on ModelNet40with 1024 points per sample, which surpasses Point-BERT fine-tuned with 1024 points by +0.8%and 8192 points by +0.2%.
For ScanObjectNN in Table 3, our Point-M2AE outperforms the secondbest Point-BERT by a significant margin, +3.79%, +0.69%
and +3.36%, respectively for the threesplits, indicating our great advantages under complex circumstances by multi-scale encoding.
As
ScanObjectNN of real-world scenes has a large semantic gap with the pre-trained synthetic ShapeNet,
Point-M2AE also exerts strong transfer ability to understand point clouds of another domain.
Part Segmentation.
We evaluate Point-M2AE for part segmentation on ShapeNetPart [47], whichpredicts per-point part labels and requires detailed understanding for local patterns.
We adoptan extremely simple segmentation head to validate the effectiveness of our pre-training for wellcapturing both high-level semantics and fine-grained details.
By the hierarchical encoder, we obtain3 -scale
point tokens of {512, 256, 64} points, and perform feature propagation in PointNet++ [29] toindependently upsample the tokens into 2048 points of the input point cloud.
Then, we concatenatethe upsampled 3 -scale features for each point and predict the part label by stacked linear projectionlayers.
As reported in Table 4.2, Point-M2AE achieves the best 86.51% instance mIoU with the simplesegmentation head, surpassing the second-best Point-BERT by +0.91%.
Note that Point-BERT [48]and other methods [28, 29, 40] adopt hierarchical segmentation heads to progressively upsample the
[Table] [Table]
+0.
We report the performance (%) of
-supervised methods
[Table]
point features from intermediate layers, while our head contains no hierarchical structure and onlyrelies on the pre-trained encoder to capture the multi-scale information of point clouds.
The resultsfully demonstrate the significance of Point-M2AE’s multi-scale pre-training to segmentation tasks.
Few-shot Classification.
We conduct experiments for few-shot classification on ModelNet40 [43]to evaluate the performance of Point-M2AE with limited fine-tuning data.
As reported in Table 4.2,
Point-M2AE achieves the best performance for all four settings, and surpasses Point-BERT by +2.2%,+2.0%, +1.3%, and +2.7%, respectively.
Our approach also shows smaller deviations than othertransformer-based methods, which indicates Point-M2AE has learned to produce more universal 3Drepresentations for well adapting to downstream tasks under low-data regimes.3D Object Detection To further evaluate our hierarchical pre-training on 3D object detection, weapply Point-M2AE to serving as the feature backbone on the indoor ScanNetV2 [9] dataset.
Weselect 3DETR-m [26] as our baseline, which consists of a 3-block encoder and a transformer decoder.
Considering the quite different dataset statistics, e.g., 2k input points for ShapeNet [6] and 50k inputpoints for ScanNetV2, we adopt the same encoder architecture with that of 3DETR-m, and keep ourhierarchical decoder with skip connections unchanged for self-supervised pre-training on ScanNetV2.
More details of models and training are in Appendix.
As reported in Table 4.2, compared to trainingfrom scratch, our hierarchical pre-training boosts the performance of 3DETR-m by +1.34% AP 25 and+1.29% AP 50 .
The experiments demonstrate the effectiveness of Point-M2AE to learn multi-scalepoint cloud encoding for object detection and its potential to benefit a wider range of 3D applications.
4.3 Ablation Study
We conduct ablation study by modifying one of the components at a time to test their effectivenessand explore the best masking strategy for self-supervised pre-training.
We report the classificationaccuracy on ModelNet40 [43] by linear SVM to evaluate the pre-trained representations.
For
[Table] [Table]
downstream tasks
, we compare the performance between fine-tuning and training from scratch tovalidate the significance of our hierarchical pre-training.
Hierarchical Modules.
As reported in Table 7, on top of our final solution of Point-M2AE in thefirst row, we respectively experiment with removing the hierarchical encoder, hierarchical decoder,skip connections, and local spatial self-attention layers from our framework.
Specifically, we replaceour encoder and decoder with 1-stage plain architectures similar to MAE, which contains 15 and 2blocks of vanilla self-attention layers, respectively.
We observe the absence of multi-stage structureseither in encoder or decoder would hurt the performance, and the hierarchical encoder plays a betterrole than the decoder.
Also, the skip connections and local spatial attention can well benefit thenetwork by providing complementary information and local inductive bias.
Masking Strategy.
In Table 8, we report Point-M2AE with different mask settings.
Without themulti-scale masking, we randomly generate masks at each scale, which leads to fragmented visibleregions for all scales.
With this strategy, the network would ‘peek’ different parts of the point cloudat different stages, which disturbs the representation learning and harms the performance by 4.5 %accuracy.
For different mask ratios, we find the 80 % ratio performs the best to build a properlychallenging pretext task for self-supervised pre-training.
[Table]
on
[Table] [Table]
[Table]
With and without Pre-training.
We report
Point-M2AE on downstream tasks with andwithout the pre-training in Table 9.
For ‘w/o’,we randomly initialize our network and adoptthe same training settings with fine-tuning.
Asshown, the hierarchical pre-training can largelyboost the performance on four datasets respectively by +1.5 % , +2.5 % , +3.8 % , and +1.1 % , [Table] indicating the significance of our pre-training [Table]
scheme.
[Table]
Conclusion
We propose Point-M2AE, a multi-scale masked autoencoder for self-supervised pre-training on3D point clouds.
With a hierarchical architecture, Point-M2AE learns to produce powerful 3Drepresentations by encoding multi-scale point clouds and reconstructing the masked coordinatesfrom a global-to-local upsampling scheme.
Extensive experiments have shown the state-of-the-artperformance of Point-M2AE on downstream tasks and our superiority to be a strong 3D representation learner.
Limitations.
Although we have experimented Point-M2AE on various 3D tasks, itsperformance on open-world 3D object detection and scene segmentation has yet not been discussed.
Our future work will focus on this direction to apply Point-M2AE for wider 3D applications.
Societal
Impact
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB)
approvals, if applicable
N
/A](c
Did you
include the estimated hourly wage paid to participants and the total
amountspent
on participant compensation?
[N/A