Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis
Abstract
Although recent point cloud analysis achieves impressive progress, the paradigm of representation learning from a single modality gradually meets its bottleneck.
In this work, we take a step towards more discriminative 3D point cloud representation by fully taking advantages of images which inherently contain richer appearance information, e.g., texture, color, and shade.
Speciﬁcally, this paper introduces a simple but effective point cloud cross-modality training ( PointCMT ) strategy, which utilizes view-images, i.e. , rendered or projected 2D images of the 3D object, to boost point cloud analysis.
In practice, to effectively acquire auxiliary knowledge from view images, we develop a teacher-student framework and formulate the crossmodal learning as a knowledge distillation problem.
PointCMT eliminates the distribution discrepancy between different modalities through novel feature and classiﬁer enhancement criteria and avoids potential negative transfer effectively.
Note that PointCMT effectively improves the point-only representation without architecture modiﬁcation.
Sufﬁcient experiments verify signiﬁcant gains on various datasets using appealing backbones, i.e. , equipped with PointCMT, PointNet++ and
PointMLP achieve state-of-the-art performance on two benchmarks, i.e., 94.4% and 86.7% accuracy on ModelNet40 and ScanObjectNN, respectively.
Code will be made available at https://github.com/ZhanHeshen/PointCMT .
Introduction
As the fundamental 3D representation, point clouds have attracted increasing attention for various applications, e.g. , self-driving [2, 33, 34], robotics perception [7, 13, 6] etc .
Generally, a point cloud consists of sparse and unordered points in the 3D space, which is signiﬁcantly different from a 2D image with a dense and regular pixel array.
Prior studies treat the understanding of 2D images and 3D point clouds as two separate problems, and both have their own merits and drawbacks.
Concretely, rich color and ﬁne-grained texture are easily obtained in 2D images, but they are ambiguous in depth and shape sensing.
Previous works extract features on images through convolution neural networks (CNN).
In contrast, point clouds are superior in providing spatial and geometric information but only preserve sparse and textureless features.
Several prior studies process features on unstructured point clouds through local aggregation operators [34, 43].
It is natural to raise a question: Could we use the rich information hidden in 2D images to boost 3D point cloud shape analysis ?
To address the above issue, one straightforward way is to leverage the beneﬁts of both images and point clouds, i.e. , fusing information from two complementary representations with task-speciﬁc design [58, 24, 10, 32, 42].
However, utilizing additional image representation requires designing a multi-modal network, which takes the extra image inputs in both training and inference phases.
DGCNN
RS-CNN
PointNet++
Trained with point clouds only
Trained with PointCMT
[Figure]
Cross
[Figure]
“Airplane” “Sofa” P o i n t C l oud s
I m a g e s (
[Figure]
Rendered from CAD
Perspective Projection
Moreover, the exploiting extra-images is usually computation-intensive and paired-images are usually unavailable during inference. Thus, multi-modal learning meets its bottleneck in many aspects. This paper tries to ease the barrier of cross-modal learning between images and point clouds. Inspired by knowledge distillation (KD) that achieves knowledge transfer from a teacher model to a student one, we formulate the cross-modal learning as a KD problem, conducting alignment between sample representations learned by images and point clouds.
However, previous KD approaches usually assume that the training data used by the teacher and student are from the same distribution [17].
Thus, since sparse and disordered point clouds represent visual information different from images, naive feature alignment between two representations appeals to cause limited gains or negative transfer for the cross-modal scenario.
To this end, we design a novel framework for cross-modal KD and propose the point cloud cross-modal training strategy, i.e. , PointCMT in Figure 1 (a), which distills features derived from images into the point cloud representation.
Speciﬁcally, multiple view images for each 3D object can be generated through either rendering the CAD model or conducting perspective projection on the point cloud from different viewpoints.
These free auxiliary images are fed into the image network to obtain the global representations for the object.
Besides, feature and classiﬁer enhancements are conducted between the point cloud and image features, in which the newly proposed criteria effectively avoid negative transfer between different modalities, i.e. , directly applying [17] hampers the performance on ModelNet40.
After training, the model gains higher performance, only taking point clouds as input without architecture modiﬁcation.
Compared with multi-modal approaches, our solution has the following preferable properties: 1) Generality : It can be easily integrated with arbitrary point cloud analysis models without structural modiﬁcation.
Effectively : It signiﬁcantly boosts the performance upon several baseline approaches, e.g. , PointNet++ [34] achieves state-of-the-art 94.4% from 93.4% overall accuracy on ModelNet40, as shown in Figure 1 (b).
3) Efﬁciency : Our PointCMT only utilizes auxiliary image data in the training stage.
After training, the enhanced 3D model can be deployed without image inputs.
4) Flexibility :
The extensive experiments illustrate that PointCMT performs superior even without colorized and dedicated rendered images, i.e. , it can also greatly improve the performance when using images directly projected by sparse and textureless point clouds.
Thus, it provides an alternative solution to enhance the point cloud shape analysis when the additional rendered images are not accessible.
In summary, our contributions are: 1) This paper formulates cross-modal learning on point cloud analysis as a knowledge distillation problem, where we utilize the merits of texture and color-aware 2D images to acquire more discriminative point cloud representation.
We propose point cloud cross-modal training, i.e., PointCMT, strategy with corresponding criteria to boost point cloud models during the training stage.
3) Extensive experiments on several datasets verify the effectiveness of our approach, where PointCMT greatly boosts several baseline models even on the state-of-the-art, e.g. , PointNet++ [34] trained with PointCMT gains 1.0% and 4.4% accuracy improvements on
ModelNet40 [48] and ScanObjectNN, respectively.
Even based upon PointMLP [31], it increases its accuracy by 1% to 86.7% on ScanObjectNN dataset.
Related Works
3D Shape Recognition Based on Point Clouds.
These stream methods directly process raw point clouds as input (also called point-based methods ).
They are pioneered by PointNet [33], which approximates a permutation-invariant set function using a per-point Multi-Layer Perceptron (MLP) followed by a max-pooling layer.
Later on, point-based methods aim at designing local aggregation operators for local feature extraction.
Speciﬁcally, they generally sample multiple sub-points from the original point cloud, and then aggregate neighboring features of each sub-point through local aggregation operators, in which point-wise MLPs [34, 31], adaptive weight [44, 46, 27] and pseudo grid based methods [39, 20] are proposed.
More recently, there are some attempts to utilize non-local operator [54], or Transformer [60, 14] to mine the long distance dependency.
This paper also follows the paradigm of point-based methods to conduct point cloud shape analysis. 3D Shape Recognition Based on Images.
Since point clouds are irregular and unordered, some works consider projecting the 3D shapes into multiple images from different viewpoints (also called view-based methods ) and then leverage the well-developed 2D CNNs to process 3D data.
One seminal work of multi-view learning is MVCNN [38].
It extracts per-view features with a shared
CNN in parallel, then aggregates via a view-level max-pooling layer.
Most follow-up works propose more effective modules to aggregate the view-level features.
For instance, some of them enhance the aggregated feature by considering similarity among views [11, 56] while others focus on the viewpoint relation [45, 22].
The above methods usually utilize ad-hoc rendered images for each 3D shape, including shade and texture for the surface mesh.
Therefore, they generally achieve higher performance than point-based methods using sparse point clouds as input.
Recently, [12] proposes a simple but effective method ( SimpleView ) through directly projecting sparse point clouds onto image planes, achieving comparable performance with point-based methods.
Inspired by view-based methods, this paper takes advantage of extracted image features from the view-based method, which are utilized as prior knowledge to boost point cloud shape analysis.
Knowledge Distillation.
Knowledge distillation (KD) aims at compressing a large network (teacher) to a compact and tiny one (student), and boost the performance of the student at the same time.
The concept was ﬁrst shown by Hinton et al.
[17], which trains a student by using the softened logits of a teacher as targets.
Over the past few years, several subsequent approaches [23, 1, 5, 40, 9, 61] use different criteria to align the sample representations between the teacher and student.
However, almost all the existing works assume that the training data used by the teacher and student networks are from the same distribution.
Our experiment illustrates that new biases and negative transfer will be introduced in the distillation process if cross-modal data from different distribution ( e.g. , features extracted from unordered point cloud and regular grid image) is utilized directly on previous KDs.
Cross-Modal Knowledge Transfer.
Cross-modal knowledge transfer in computer vision is a relatively emerging ﬁeld that aims to utilize additional modalities at the training stage and enhance the model’s performance on the target modality at the inference.
Recently, there are 3D-to-2D knowledge transfer approaches, which adopt geometric aware 3D features from point clouds to enhance the performance of 2D tasks through a contrastive manner [18] or feature alignment [30].
Later on, approaches attempt to transfer priors in images to enhance 3D point cloud-related tasks, and some are designed for speciﬁc tasks.
Concretely, [28, 26] propose the images-to-point contrastive pre-training, [50] inﬂats 2D convolution kernels to the 3D ones and [57, 59, 53] independently apply cross-modal training for visual grounding, captioning and semantic segmentation.
Inspired but different from the above, we are the ﬁrst to conduct image-to-point knowledge distillation for point cloud analysis.
Methodlogy 3.1 Problem Statement
Let P ∈ R N × 3 and y ∈ R 1 be the point cloud and ground-truth label of the 3D object.
Its corresponding view-image counterparts can be denoted as I ∈ R V
×
H
×
W × 3 , where N , V and ( H, W ) are the number of points, number of view-images and image size, respectively.
View images can be gained by rendering the 3D CAD model [38] or perspective projecting the raw point cloud [12].
We denote T and S as the image and point cloud analysis networks, respectively, and regard them as the teacher and student in traditional knowledge distillation (KD).
For these networks, we split each of them into two parts: (i) Encoders ( i.e. , feature extractors Enc img ( · ) and Enc pts ( · ) ), the output of
V i e w - i m a g e s
P o i n t
C l oud ...
Pre-trained
ImageEncoder
Point Cloud Encoder 3D Classifier Feature Enhancement
ClassifierEnhancement
Training Only
Training & Inference
Supervised by Ground Truth
Features (a) PointCMT
Fixed Cross-Modal Point Generator
Cross-modalFeatures
Point Clouds Reconstruction (c) Feature Enhancement (b) Training of CMPG
Cross-ModalPoint Generator
Image Features
Supervised by Ground Truth
L CE
L Feature
L EMD
I
P
F img
F pts
F img F img
F pts
P pts
P img which at the last layer are global feature representations F img ∈ R D and F pts ∈ R D .
(ii) Classiﬁers , which project the feature representation F img and F pts into class logits through Cls img ( F img ) and
Cls pts ( F pts ) , where Cls( · ) denotes the classiﬁer.
Since we formulate the cross-modal learning as a KD problem, its goal in the learning process is to distill the priors knowledge from the image into point cloud features, obtaining an image-enhanced ideal feature F KD .
During the knowledge distillation, we parameterize the teacher and student networks with θ T and θ S , and denote the knowledge of the teacher network as K T .
From the
Bayesian perspective, a neural network can be viewed as a probability model, e.g. , P ( y |P , θ S ) for point cloud analysis model as an example: given an input point cloud P , the network assigns an output probability with the parameters θ S .
Therefore, if we want the student network guided by image knowledge on the input sample, our goal can be further reformulated as maximizing the probability
P ( F KD |P , y ; θ S , K T ) , which can also be used to measure the ability of student network extracting the feature with cross-modal information.
To ﬁnd the lower bound of the above probability, we deﬁne the discrepancy g between theoretically discriminative features F img ∗ and F pts ∗ as
[Equation]
where F img ∗ and F pts ∗ are ideal features in speciﬁc modalities.
In Lemma 1, we give the lower bound of cross-modal learning as a KD problem.
The proof is provided in the supplementary material.
Lemma 1 : By the deﬁnition above, P ( F KD |P , y ; θ S , K T ) is bounded below by P ( F KD |I , y ; θ T ) + λ − g , where λ is
[Equation]
In Lemma 1, λ measures the compatibility of knowledge distillation of the image networks, and can be viewed as a constant when the architectures are determined.
P ( F KD |I , y ; θ T ) is also a constant when the parameters θ T and the architecture of the point cloud analysis model are ﬁxed, e.g., using a pre-trained image network.
Therefore, the Lemma ensures that during the knowledge distillation, one can maximize P ( F KD |P , y ; θ S , K T ) through minimizing g , which gives a theoretical guarantee for the KD problem.
If we adopt previous KD studies in cross-modal scenario, they minimize g through making student directly approximate the teacher’s features [23, 1, 5] or predict logits [17].
However, in a common
KD problem, the teacher and student are generally trained on the same dataset with an identical distribution [17].
Moreover, the teacher generally achieves better performance than the student.
In contrast, the image and point cloud analysis models tend to learn different feature representations and logits distribution which are generally complementary.
Direct alignment may cause a limited gain or even negative transfer.
Moreover, previous KD approaches treat encoders and classiﬁers as a whole architecture since the teacher and student networks generally have the same components.
However, the point cloud convolution in encoder is signiﬁcantly different from the 2D CNN, but they have the same classiﬁer design.
To solve the cross-modal KD problem we propose PointCMT, which effectively solves the crossmodal learning problem.
The workﬂow of PointCMT is demonstrated in Figure 2 (a) and can be summarized as Algorithm 1.
Speciﬁcally, there are three stages exist in PointCMT.
In Stage I (Section 3.2), we train the image encoder and classiﬁer using view-images and ground-truth labels.
In Stage II (Section 3.3), we train the cross-modal point generator (CMPG) through image features, and we independently align features and logits of two modalities in stage III (Section 3.4), where the encoder and classiﬁer of point cloud analysis network are both enhanced.
3.2 Learning Image Priors
For each 3D object, we use multiple view-images ( i.e. , rendered color images or projected images from raw point cloud) as additional data, and the whole process can be described as:
[Equation]
Inspired by view-based learning approaches [38], V view-images from I ﬂow into a shared-weights image feature extractor CNN( · ) to obtain a series of vectors.
By aggregating the view-level vectors via an aggregation function A{·} , we obtain a global feature representation F img from all images, which integrates shape information from multiple views.
Finally, an image classiﬁer maps the above global feature to gain a prediction logits through Cls img ( F img ) , which is supervised by the ground truth label y through cross-entropy loss L CE .
3.3 Cross-Modal Point Generator
A cross-modal point generator (CMPG) can be seen as a nonlinear transformation R D
→ R N ×that maps the global feature representation F img acquired from images into the Euclidean space.
Thus, it can avoid potential negative transfer effectively by directly aligning cross-modal features from different distributions.
In order to better learn image priors in the point cloud analysis network, we pre-train the CMPG through the image feature F img ∈ R D , and ﬁx it in the distillation stage.
Figure 2 (b) illustrates the pre-training stage of CMPG.
It takes the image feature as input and reconstruct a point cloud ˆ P img ∈ R N × 3 , which is supervised by the original point cloud P through Earth Mover’s distance (EMD) [37]:
[Equation]
where |P| =
|P img | and φ : P → ˆ P img is a bijection, i.e. , for each point p ∈ P , φ ( · ) ﬁnds a sole point correspondence in ˆ P img .
After pre-training, CMPG reconstructs a point cloud through an image-relative feature representation.
3.4 Image Priors Assisted Training
During the training stage, three objectiveness should be optimized:
Classiﬁcation Loss.
In our PointCMT, arbitrary point cloud analysis models can be assembled.
Generally, it should be designed through a point encoder and a classiﬁer, where the point encoder takes a point cloud as input, generates the point cloud feature representation F pts and feeds it into the classiﬁer Cls pts ( · ) to obtain a class logits.
Finally, the cross-entropy loss L CE is used as criteria.
Feature Enhancement Loss.
Unlike previous KD methods that directly align features of teacher and student, we ﬁrst transform the cross-modal features into Euclidean space.
As shown in Figure 2 (c), the pre-trained CMPG independently transform F img and F pts , obtaining two point clouds ˆ P img and ˆ P pts , respectively.
After that, EMD loss is conducted on the two-point clouds as an objectiveness:
[Equation]
where | ˆ P pts | =
|P img | and φ : ˆ P pts → ˆ P img is a bijection.
Compared with traditional L 2 loss, the
EMD distance is natural for solving an assignment problem for permutation-invariant point sets.
For all but a zero-measure subset of point set pairs, the optimal bijection φ is unique and invariant under the inﬁnitesimal movement of the points.
Thus, EMD is differentiable almost everywhere.
Classiﬁer Enhancement Loss.
In addition to supervising the point feature extractor through the above Feature Enhancement, we further propose constraints conducted on the point classiﬁer (as
Figure 2 (a)). Speciﬁcally, the image feature generated by the teacher network is fed into the point classiﬁer, in which the gradient only back-propagates to the point classiﬁer.
Besides, classiﬁer enhancement is proposed to enable the point classiﬁer to handle the image feature during the distillation.
It aligns outputs logits of the point classiﬁer by using image and point features as inputs.
This constraint is modiﬁed based on the distillation loss in Hinton et al.
[17] as in Equation (6), but in this case, two sets of logits come from the same classiﬁer.
[Equation]
where D KL ( ·||· ) is KL divergence.
In contrast, our proposed classiﬁer enhancement is more suitable for the cross-modal scenario where a great discrepancy exist between image and point cloud features.
Concretely, the loss for classiﬁer enhancement can be written as
[Equation]
Final Loss.
The ﬁnal loss is a weighted sum of the above three losses L =
L CE + α L Feature + β L Classiﬁer , where α = 30 and β = 0 .
3 are the weights to adjust the ratios of each loss, respectively.
Experiment 4.1 Shape Classiﬁcation on ModelNet
[Figure]
We ﬁrstly evaluate our PointCMT on synthetic dataset ModelNet40 [48], which is a large-scale 3D CAD model dataset.
Dataset Description and Processing.
ModelNet40 is composed of 9,843 train models and 2,468 test models in 40 classes.
For the input of the 3D network, we use point clouds provided by the ofﬁcial dataset, which is the same as PointNet [33].
For the input of the image network, we use 20 rendered view images from CAD models utilized in RotationNet [22].
These images have a resolution of 224 × 224 .
Since they consider both the mesh surfaces and illumination, they can provide more information to the 3D network.
Selected samples of point clouds and corresponding multi-view images are shown in Figure 3.
Implementation.
For image network, we use ResNet-18 [16] pre-trained on ImageNet [8] as the feature extractor.
Following MVCNN [38], we obtain the global shape feature by applying viewwise max-pooling to the view-level features. Finally, a fully-connected layer is used to output the
[Table]
classiﬁcation logits.
During the training process of the above network, we use SGD as our optimizer with a learning rate of 0.01.
The batch size is set to 128 for 50 epochs.
After that, we ﬁx the image network and train the CMPG with Adam and 32 batch size for 50 epochs.
In practice, CMPG consists of three-layer MLP.
For the point cloud analysis models, DGCNN [43] and RS-CNN [27] are independently trained with the training strategies provided in their ofﬁcial codes.
PointNet++ [34] is trained with strategy of RS-CNN [27] as [12] for a better performance.
Comparison with State-of-the-arts.
The classiﬁcation results on ModelNet40 are shown in Table 1, where the overall accuracy (OA) and class mean accuracy (mAcc) are compared.
The upper part of the table illustrates the results of current state-of-the-art methods, in which we use PointNet++ [34], RS-CNN [27] and DGCNN [43] as our baselines.
We do not use PointMLP [31] as our baseline since it cannot robustly reproduce the highest results on ModelNet40, where the issue is mentioned in their open-sourced codes.
For models trained from scratch, PointMLP [31] achieves the highest accuracy.
As shown in the lower part of the table, after training with PointCMT, the performance of all baselines is greatly boosted, i.e., 1.0% improvement upon PointNet++ and 0.9% for RS-CNN and 0.6% for DGCNN.
We also compare our method to several open-sourced methods and report the parameters and testing speed.
As shown in the last two columns of the table, though PointMLP gains 0.1 higher overall accuracy, its network consists of about 7 .
7 × parameters and only achieves 46% speed of PointNet++.
In contrast, PointCMT performs well on light-weighted models, which shows its great potential for real-time applications, e.g. , scene parsing in autonomous driving.
4.2 Shape Classiﬁcation on ScanObjectNN
Though ModelNet40 is the widely used benchmark for point cloud analysis, it may not meet the realistic requirement due to its synthetic nature.
To this end, we also conduct experiments on the
ScanObjectNN benchmark [41], which is a real-world dataset.
Dataset Description and Processing.
ScanObjectNN collects 2,902 objects from real-world indoor scenes ScanNet [7] and SceneNN [19], categorizing into 15 categories.
Several variants are provided in the dataset, where the most challenging one is PB_T50_RS, i.e. , introducing perturbation objects (11,416 and 2,882 data for training and test) via random translation, shift, rotation and scaling.
Due to background, noise, and occlusions, this benchmark poses signiﬁcant challenges to existing point cloud
[Table]
analysis methods.
Furthermore, since the PB_T50_RS dataset only preserves the spatial coordinates (XYZ) for each object while the other information, such as RGB, is discarded, we also compared the original 2,902 objects (OBJ_ONLY), which includes additional RGB information.
On both above datasets, we only use depth images through conducting perspective projection on raw point cloud as additional inputs, as shown in the last column in Figure 3.
Section 4.6 will discuss more results using projection with additional color information.
Implementation.
All view-images in ScanObjectNN are gained by the projection of raw point clouds, and we follow the structure of [12] only generate six images for PB_T50_RS and OBJ_ONLY.
We train the image network from scratch with batch size 32 and SGD optimizer for longer epochs of 1,000.
The training strategy of CMPG is the same as ModelNet40.
For both point cloud models trained from scratch and with PointCMT, we use SGD optimizer for 1,000 epochs with batch size 32.
Comparison with State-of-the-arts.
The results are shown in Table 2, where PointNet++ [34] and current state-of-the-art PointMLP [31] are chosen as our baselines.
PointCMT signiﬁcantly improves the performance on both class mean accuracy (mAcc) and the overall accuracy (OA), even on state-of-the-art methods.
Speciﬁcally, although background, noise, and occlusions exist on
PB_T50_RS dataset, PointCMT still improves the overall accuracy of PointNet++ by 3.9%.
Moreover,
PointCMT also achieves state-of-the-art results on OBJ_ONLY dataset.
Note that there is no auxiliary information provided in images, and all view images are generated through the perspective projection of points coordinates.
Nevertheless, PointCMT still dramatically increases the mAcc of PointMLP from 89.4% to 92.0% (+2.6%).
4.3 Data Efﬁcient Learning
[Table]
We evaluate our approach under limited data scenarios in Table 3.
Here, we only sample a small amount of training data in each category on ModelNet40, and evaluate the entire testing data.
Our
PointCMT shows an even more signiﬁcant gap when using a small subset of the training data, again compared to PointNet++ which trained from scratch.
Especially when facing only 2% and 10% of the training data, we achieve about 1.9% and 2.8% improvements, respectively.
This result illustrates that PointCMT provides more vital guidance for point cloud models in the low data regime.
[Table]
4.4 Ablation study
The ablation results on three datasets are summarized in Table
[Table]
in which we use PointNet++ as our baselines.
We ﬁrst test the effectiveness of feature enhancement (FE) and classiﬁer enhancement (CE) in PointCMT.
The results demonstrate that only using FE already signiﬁcantly boosts the performance on both datasets,
i
[Table]
e., increases the overall accuracy by 0.4% and 3.1% on ModelNet and ScanObjetNN.
Only using classiﬁer enhancement (CE) improves the accuracy by around 0.6% and 2.9%.
Finally, it is surprising that when we use both FE and CE during the training phase, it achieves the best result of 94.4% and 83.3%, respectively.
4.5 Comparison with Knowledge Distillation Methods
[Table]
To further verify the effectiveness of our proposed method compared with typical teach-student architecture and other knowledge distillation manners, we compare PointCMT with typical approaches of knowledge transfer in Table 1.
Among all the methods, Hinton et al.
[17] is a pioneer study for knowledge distillation, while Huang et al.
[21] and Yang et al.
[55] are recent works.
As shown in the table, directly aligning features as [17] between two modalities will cause a negative transfer on ModelNet40.
This phenomenon does not appear on ScanObjectNN, since view images projected via point clouds may have a smaller gap than rendered images of CAD models.
Nevertheless, other KD techniques only achieve marginal improvement compared with PointCMT.
4.6 Different View-image Generation
[Table]
In this section, we compare results with different view-image generation strategies.
As shown in Figure 3, multiple view-image types can be applied in our framework, and we compare the results in Table 6.
As illustrated in the table, images rendered from the CAD model improve more compared with only using projection since the former provides additional shade and texture information.
In contrast, we ﬁnd out that using additional colors in the OBJ_ONLY dataset cannot boost performance.
The reason is that OBJ_ONLY dataset only contains 2,902 objects, and image networks are easier to overﬁt when using the color information.
Conclusion
In this work, we propose a point cloud cross-modal training strategy named PointCMT.
By exploiting some sophisticated architecture and reasonable criteria function, our PointCMT can boost the performance signiﬁcantly for point cloud analysis methods on several benchmarks, outperforming previous methods by a large margin.
We believe that our work can be applied to a broader range of other scenarios in the future, such as 3D semantic segmentation and object detection.
Meanwhile, our method provides an alternative solution to the comprehension of 3D scenes with severe texture details missing.
It can improve performance through image priors and knowledge transfer.
Acknowledgment
This work was supported in part by NSFC-Youth 61902335, by Key Area R&D Program of Guangdong Province with grant No.2018B030338 001, by the National Key R&D Program of China with grant No.2018YFB1800 800, by the Basic Research Project No. HZQB-KCZYZ-2021067 of Hetao Shenzhen HK S&T Cooperation Zone, by Shenzhen Outstanding Talents Training Fund, by
Guangdong Research Project No.2017ZT07X152, by Guangdong Regional Joint Fund-Key Projects 2019B1515120039, by the NSFC 61931024&8192 2046, by zelixir biotechnology company Fund, by Tencent Open Fund and by High-Performance Computing Portal under the administration of the
Information Technology Services Ofﬁce (ITSO) at CUHKSZ
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]