Learning Generalizable Part-based Feature Representation for 3D Point Clouds
Abstract
Deep networks on 3D point clouds have achieved remarkable success in 3D classification, while they are vulnerable to geometry variations caused by inconsistent data acquisition procedures.
This results in a challenging 3D domain generalization (3DDG) problem, that is to generalize a model trained on source domain to an unseen target domain.
Based on the observation that local geometric structures are more generalizable than the whole shape, we propose to reduce the geometry shift by a generalizable part-based feature representation and design a novel part-based domain generalization network (PDG) for 3D point cloud classification.
Specifically, we build a part-template feature space shared by source and target domains.
Shapes from distinct domains are first organized to part-level features and then represented by part-template features.
The transformed part-level features, dubbed aligned part-based representations, are then aggregated by a part-based feature aggregation module.
To improve the robustness of the part-based representations, we further propose a contrastive learning framework upon part-based shape representation.
Experiments and ablation studies on 3DDA and 3DDG benchmarks justify the efficacy of the proposed approach for domain generalization, compared with the previous state-of-the-art methods.
Our code will be available on http://github.com/weixmath/PDG .
Introduction
The 3D shape understanding and reasoning play a critical role in wide applications such as automatic drive, archaeology, virtual reality / augmented reality, etc .
Point cloud is a popular representation of 3D shape attributed to its simpleness and effectiveness.
Recently, with the thriving of deep learning, numerous deep architectures [1–10] have been proposed for 3D point cloud analysis.
Although these methods have achieved impressive results on the 3D shape classification task, all of them strongly rely on the i.i.d assumption on the source and target data but ignore the out-of-distribution situation in real-world practice.
For example, the performance of these methods drops dramatically when trained on CAD datasets [11, 12] and tested on real scanned datasets [13, 14].
Therefore, it is crucial and valuable to investigate how to learn a 3D classification model that generalizes well on a related test domain with domain shift to the training domain [15].
Domain shift refers to the existence of significant divergence between the distributions of the training and test datasets [15].
The domain shift may degrade the performance of network trained on a training dataset when generalizing to the test dataset.
Domain generalization methods for improving domain robustness of deep network have been extensively investigated for 2D images [16–21].
However, little work [22, 23] addresses the domain generalization (DG) problem for point cloud deep networks.
For point clouds, the major cause of domain shift is the geometry variations generated by inconsistent data collection processes, e .
g ., the realistic sensor noises, the non-uniform density of point clouds, and self-occlusion.
These domain shifts specific for 3D point clouds hinder the idea that directly adopts 2D image-oriented DG methods to point clouds, and this inspires us to design a 3D domain generalization model for addressing the domain shifts of point clouds.
[Figure]
To improve the model robustness to geometry variations, a straightforward way is to simulate geometry shift during training.
MetaSets [22] designs three point transformation tasks of simulating occlusions, missing parts and changes in scanning density, which have been proven to be effective as data augmentations.
They further learn feature representations by meta-learning using these tasks.
Instead of improving model’s generalization ability using globally pooled point cloud features, in this paper, we tackle the domain generalization problem for 3D point clouds based on our observation that the local geometric structures of point clouds are more likely to be shared across distinct domains, therefore being more generalizable to the geometry variations.
Parsing objects into parts is crucial for humans to understand the world.
For the task of object recognition, the visual system of people decomposes shapes into parts and recognizes objects based on the description and spatial relations of parts [24].
It is hard for machines to understand the chair on the right of Fig.
1 when it overfits on a set of chairs similar to the left one in Fig.
However, humans can easily interpret them by decomposing two chairs and inferring their labels from some similar parts ( e .
g ., legs and a plain), even when having broken parts of the right chair in Fig.
This motivation seems to be intuitive, and we will provide more experimental evidence in Sect. 2.
Based on this observation, we are interested in learning generalizable representation of 3D point clouds at the part-level instead of the global shape-level.
Along this idea, in this work, we propose a novel part-based domain generalization network for 3D point cloud classification.
We build a part-template feature space that is shared to source and target domains.
Shapes from distinct domains are first organized as part-level features and then aligned to part-template features by a cross attention mechanism.
Aligned part-based features are then aggregated by a part-based feature aggregation module for each point cloud.
To improve the robustness of part-based representation, we further propose a contrastive learning framework to enforce that the feature representations of a point cloud under different transformations are consistent in part-level and shape-level.
Extensive experiments conducted on 3DDA [25] and 3DDG [22] benchmarks demonstrate the effectiveness of our approach, and our method outperforms the compared methods by a notable margin.
Our contribution can be summarized in three folds.
First , we empirically observe that the geometry shift induced domain gap of point clouds could be reduced by part-level representation, thus we propose to learn part-based 3D feature representation to improve the generalization ability of point cloud classification models.
Second , we propose a novel part-based domain generalization network for 3D point cloud classification.
A contrastive learning framework upon part-based shape representation is further designed to improve the robustness of learned representations.
Third , our method achieves the best performance by comparisons on 3D benchmarks for domain generalization.
Reducing Geometry Shift by Part-based Feature Representation
Problem definition.
Let X and Y be input and label spaces.
A domain is defined by ( D , g D ) where
D is a probability distribution on X and g D : X → Y is a function mapping input to its ground-truth label.
The objective of domain generalization is to train a classifier F on the source domain D S that predicts well on target domain D T when the target samples are not available during training.
In general, we assume the label space is shared by source and target domains.
In 3D domain generalization, the input is a point cloud P ∈ R N × 3 , where N is the number of points.
We consider a point cloud classification model F composed of a feature extractor f θ :
[Figure]
P → Z and a classifier C ψ : F → R C , where Z ∈ R N × d is point-wise features, F ∈ R d is a global representation of P , and C is the number of classes.
The final prediction is given by ˆ y = softmax ( C ψ ( pooling ( f θ ( P )))) , where the pooling operation is determined by F .
The parameters ( θ, ψ ) are optimized with respect to the cross entropy loss L ( y, ˆ y )
= − (cid:80) C
i =
y i · log(ˆ y i ) .
Directly minimizing the cross entropy loss L will produce a discriminative network in the source domain, however, it may overfit the source domain and degeneralize on the target domain.
Comparison of generalization and discrimination abilities of shape-level and part-level features.
To understand a shape, people may process it in various granularities.
The fine-grained parts represent local geometric structures while coarsen parts contain more global semantic information.
Thus global shape-level features may be more discriminative than part-level features.
However, this advantage drops sharply when encountering a large distribution discrepancy between training and test domains for shape-level features.
On the contrary, part-level features encode the local geometric structures which are shared across different shapes in distinct distributions, while they are short of semantic information.
We next experimentally verify the above analysis.
We use A -distance as a measure to evaluate distribution discrepancy [26, 27].
It is defined as dist A
=
2(1 − 2 ϵ ) , where ϵ is the test error of a classifier trained to discriminate the source and the target domain data.
We train a PointNet [5] on source domains M (ModelNet dataset [12]) and test on target domain SO
(ScanObjectNN [14]).
We first get point-wise features Z ∈ R N × d for each shape that has N points.
Then point-wise features are max-pooled on all points to produce a global shape-level feature Z s .
We also split each shape into some overlapped parts in distinct scales, i .
e ., 16 parts (256 points in each part) andparts (512 points in each part) and max-pooled on each part to get part-level features { Z spi } 16 i =1 and { Z lpi } 8 i =1 .
Fig.
2 (a) shows the A -distance on each class of M → SO with shape-level features
Z s and part-level features { Z spi } 16 i =1 and { Z lpi } 8 i =1 .
We observe that dist A of part-level features is smaller than dist A of shape-level features and dist A decreases as the size of parts becomes smaller, indicating that part-level features are able to better reduce domain gap.
We also train a linear SVM on source domain training data and test on source domain test data and target domain test data to evaluate the discrimination abilities.
For part-level features, the prediction of a shape is by voting on the predictions of parts.
As shown in Fig.
2 (b), the discrimination abilities of features are weakened when the scale decreases.
Though part-level features could reduce the geometry shift between source and target domains, aggregating them simply by voting could not improve the discrimination ability.
Based on the above observations, we are inspired to represent a point cloud by part-level features to reduce the domain discrepancy.
We build a part-template feature space that is shared with source and target domains.
Part-level features in distinct domains are aligned to the part-template features, resulting in part-based feature representations with better generalization.
To improve the discrimination of the part-level features, the features of parts are fused with a part-based feature aggregation module to achieve a global representation for each point cloud.
Learning Part-based Representation of Point Clouds
We first introduce our proposed part-based representation of point clouds, taken as the main operation of our part-based domain generalization network for 3D point cloud classification presented in Sect. 4.
F e a t u re E n c o d
er
F e a t u re E n c o d
er
Source
[Figure]
𝒁 𝑸 𝟏 𝑺 𝒁 𝑸
𝑺
P a r t-
l e v e l
F e a t u re
A gg re ga t
i o n
P a r t-
l e v e l
F e a t u re
A gg re ga t i o n 𝒁 𝑸
𝑺 𝒁 𝑸 𝟏 𝑺 𝒁 𝑸
𝑺 𝒁 𝑸
𝑺
Part-level Features 𝒁 𝑸 𝟏 𝑻 𝒁 𝑸 𝟏 𝑻 𝒁 𝑸 𝟏 𝑻 𝒁 𝑸 𝟏 𝑻 𝒁 𝑸 𝟏 𝑻 𝒁 𝑸 𝟏 𝑻
Aligned Part- based Features
Global Representation 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑺 𝑭 𝑸 𝟏 𝑻 𝑭 𝑸 𝟏 𝑻 𝑭 𝑸 𝟏 𝑻 𝑭 𝑸 𝟏 𝑻 𝑭 𝑸 𝟏 𝑻 𝑭 𝑸 𝟏 𝑻
[Figure]
The major objective of this representation is to transform part-level features of shapes in distinct domains into a common space, i .
e ., part-template feature space.
All these part-level features are aligned to a set of learnable part-template features in the part-template feature space.
The transformed features are dubbed as the aligned part-based representations of point clouds.
As shown in Fig 3, taking two point clouds from source and target domains, they are first organized to part-level features, and then transformed to aligned part-based representations in part-template feature space by cross-attention.
In order to aggregate the aligned part-based representations to a global representation, we further propose a part-based feature aggregation module to aggregate features according to their importance.
3.1 From point clouds to part-level features
Given a point cloud P = { x 1 , x 2 , ..., x N } ∈ R N × 3 , point-wise features Z = { z 1 , z 2 , ..., z N } ∈
R N × d are first extracted by a feature extractor f θ , i .
e .
, Z = f θ ( P ; θ f ) .
We represent a point cloud by a union of M overlapped parts P = Q 1 ∪ Q 2 ∪ ... ∪ Q M , where each part Q i
= { x i 1 , ..., x ik } ∈ R k ×is defined as a center point x i 1 with its k nearest neighbor points { x i 1 , ..., x ik } .
We use farthest point sampling (FPS) to sample M center points for constructing M parts since FPS has better coverage of the entire point cloud.
For each part Q i , corresponding part-level feature Z Q i ∈
R
d is derived by maxpooling on point-wise features of part Q i :
[Equation]
3.2 Representing part-level features using part-template features
As discussed in Sect.
2, part-level features reflect the local geometric structures of 3D shapes, which could reduce geometry shifts more effectively.
Thus we are inspired to use this general information to improve the generalization ability of the classification model.
Given part-level features in the source domain, part-level geometry priors, modeled as part-template features, are extracted from the source domain and provide references for the target domain.
Specifically, we could construct a part-template space H , where H =
{ H i } N H i =1 ∈ R N H
×
d is a set of part-template features.
H serves as a basis set and each basis vector could encode a local geometric prior.
Our proposed part-template features are similar to the dictionary of local geometric priors and each input part-level feature is represented as a combination of them.
A common choice to get a dictionary is to apply k-means clustering on part-level features of the entire training data in each training step.
However, the dictionary computed by k-means can only reflect local geometric prior in the current step.
Instead, we propose to use a set of learnable part-template features to represent various shapes in the whole training process, which can be updated automatically by gradient propagation.
Leveraging this simple yet effective representation, we could capture rich part-level geometry priors.
Given part-level features Z Q = { Z Q i } Mi =1 ∈ R M × d and part-template features H =
{ H i } N H i =1 ∈
R N H
×
d , we want to find a feature transformation T : R N H ×
d → R M × d for representing part-level features with the part-template features, defined as
[Equation]
where T ∈ R M × N H is a linear transform map implementing T and F Q ≜ { F Q i } Mi =1 is transformed features, dubbed aligned part-based features.
Briefly speaking, each aligned part-based feature is a linear combination of part-template features, and the combination weight is decided by the relation between part-level features and part-template features.
Recently, some researchers [28–31] have adopted cross-attention to align features from different modalities.
[32, 33] further apply cross attention module to solve the unsupervised domain adaptation problem based on its great power in feature alignment and robustness to noises.
In our case, we use cross-attention module to learn dependencies between part-level features and part-template features.
The transformation map T is defined as
[Equation]
where the softmax function is applied on the rows of the scaled similarity matrix.
The i -th row of T is a normalized similarity vector between part-level feature Z Q i and all the part-template features { H i } N H i =1 .
Part-template features which are more similar to part-level feature Z Q i will contribute more to the combination, while the less similar part-template features make little impact.
Optimal transport [34] is another possible approach to distribution transport and alignment.
We can use optimal transport to learn T for transporting the part-template features such that the transported feature distribution is the same as that of part-level feature distribution.
We conduct the experiment that PDG with OT for feature alignment in the ablation study.
In this work, we employ cross-attention for aligning part-level features to part-template features because of its better performance and lower computation cost.
Once the T is derived, the corresponding aligned part-based features are derived by F Q = T H .
For each part-level feature Z Q i , we can observe that the aligned part-based feature F Q i is a weighted sum of part-template features { H i } N H i =1 .
3.3 Part-level features aggregation
After getting a set of aligned part-based features, a global representation is often obtained by pooling these features.
However, various parts of a shape may have different contributions for shape recognition.
Given the part-level representation F Q = { F Q i } Mi =1 , we define the score of each part as
[Equation]
) for i = 1 , ..., M . S ( · ) is a part score module with parameters θ P s , outputting the vector of probabilities of a part belonging to C shape classes, and max operation returns the maximal value of this vector.
P s ( · ) is implemented as the same structure as classifier C ψ ( · ) .
In a word, the score s i indicates the confidence score for classification of part F Q i .
Then each part is weighted by this part score, and the global representation of shape is derived by
[Equation]
Contrastive Learning Upon Part-based Shape Representation
We build a contrastive learning framework upon part-based shape representation to improve the robustness of the learned aligned part-based features.
The main idea is to maximize the agreement between a same shape under different augmentations in part-level and shape-level via contrastive loss.
We randomly sample a mini-batch of N point clouds { P k } Nk =1 and the contrastive learning task is defined on 2 N point clouds which are the pairs of samples under different augmentations.
We use three point cloud transformations including non-uniform density, dropping, and self-occlusion.
For more details, please refer to [22].
For each point cloud P k , we denote ˜ P k as the same point cloud with another augmentation.
Augmented shapes are encoded to derive the aligned part-based representation
F Q ∈ R 2 N × M × d and global representation F ∈ R 2 N × d .
Following [35], they are further processed by a projection head Proj ( · ) , resulting in ˆ F Q and ˆF .
Proj ( · ) is implemented as a two-layer MLPs with 4 d hidden dimensions.
We adopt the contrastive loss as the form of InfoNCE [36], which learns representation by attracting positive samples and dispelling negative samples.
The shape-level contrastive loss is defined as a supervised contrastive loss [37]:
[Equation]
where sim ( · , · ) denotes the cosine similarity of two inputs, and τ is a temperature hyper-parameter, empirically set to 0.07.
ˆF k is the anchor, ˆF + k denotes ˆF k ’s positively paired sample (excluding anchor itself) with the same label as ˆF k , and the negative sample ˆF − k is with different label to ˆF k .
By this shape-level contrastive loss, the global representation of same class under different transformations will cluster together while those from different classes will be pushed apart.
Part-level contrastive loss L Cpart is defined in a self-supervised form:
[Equation]
For the i -th part of point cloud P k , whose aligned part-based feature is ˆ
[Equation]
i , the positive sample ˆ F + Q k,i is the aligned part-based feature of i -th part of ˜ P k .
The positive part has the same center point as the anchor part.
The negative samples are the other aligned part-based features in this batch.
Part-level contrastive loss L Cpart encourages learned aligned part-based feature of a part of shape under different local transformations to be consistent in the feature space.
By the contrastive tasks in both part-level and shape-level, aligned part-based representations are robust to varying local point transformations and the combination of these aligned part-based representations could be discriminative for shape classification.
4.1 Overall loss
The overall training loss consists of shape loss L shape , part loss L part , shape-level contrastive loss
L Cshape , and part-level contrastive loss L Cpart .
Given global shape representation F , it is sent to a classifier C ψ followed by a softmax layer.
The total training loss is
[Equation]
where y is the class label and L shape is a cross entropy loss based on global shape representation F .
L part is the cross-entropy loss defined on aligned part-based feature S ( F Q i ) .
L p enforces the part score module S ( · ) could discriminate the shape category based on the aligned part-based features.
Experiments 5.1 Datasets
Sim-to-Real [22].
Real-to-Sim [22] is a 3DDG benchmark consisting of three domains:
ModelNet [12], ShapeNet [11] and ScanObjectNN [14].
(a) ModelNet ( M ) contains 12, clean 3D CAD models from 40 categories, with 9,483 training models and 2,468 test models.
(b) ShapeNet ( S ) contains 51,162 3D models categorized into 55 classes and its objects have larger structure variances compared with ModelNet.
(c) ScanObjectNN ( SO ) is a recently proposed real-world 3D object classification dataset with scanned indoor scene data.
It contains 2,902 object instances from 15 categories.
ScanObjectNN offers more practical challenges including background occurrence, object partiality, and different deformation variants.
Two versions of ScanObjectNN, respectively with background (ScanObjectNN-BG, denoted as
SO B ) and with hardest perturbations (ScanObjectNN-T50-RS, denotes as SO H ), are also utilized for testing.
Then six synthetic-to-real point cloud domain generalization tasks are built.
M → SO , M → SO B , and M → SO H refer to training network on ModelNet and test on
ScanObjectNN, ScanObjectNN-BG, and ScanObjectNN-T50-RS where they have 11 shared classes, while S → SO , S → SO B and S → SO H are defined as generalizing network from
ShapeNet to ScanObjectNN, ScanObjectNN-BG, and ScanObjectNN-T50-RS within 9 shared classes.
We follow the data preparation and experiment setting in [22].
Specifically, we use the official training and test split strategy for each dataset.
Each point cloud from three domains contains 2, points and is normalized within a unit ball.
We report two versions of results, i .
e ., “Best” that is the best achievable results among the epochs in training a network, while “Last five” indicating the average test results of the last five epochs of the training procedure.
Note that the reported “Best” is unreasonable since the labels of test data are inaccessible, but it provides an upper-bound of performance for a domain generalization approach.
Thus we mainly report the “Last five” as the comparison measure in this paper.
We calculate the average of “Last five” of four DG tasks to evaluate the performance of methods.
We conduct each experiment three times and report the average top-1 classification accuracy in tables.
PointDA [25].
PointDA
[25] dataset is a widely used point cloud domain adaptation benchmark, which collects shapes of 10 shared classes from ModelNet [12] (M), ShapeNet [11] (S), and ScanNet [13] ( S ⋆ ).
Six point cloud domain adaptation tasks including M → S , M → S ⋆ , S → M ,
S → S ⋆ , S ⋆ → M , and S ⋆ → S are built upon PointDA [25].
We follow the data preparation, dataset splitting, experiment setting in [25].
Each point cloud in PointDA [25] has 1,024 points.
We implement our method on this benchmark under the setting of domain generalization.
5.2 Compared methods
For Sim-to-Real, we first compare our method with three state-of-the-art point classification methods including PointNet [5], DGCNN [6] and PointMLP [4].
We find that these methods achieve better results than the results reported in [22], if a random jittering [5] is performed on the training data which is widely used in 3D classification model training.
We further compare our method with stateof-the-art 3DDG method MetaSets [22].
We report their results by directly running their published codes of http://github.com/thuml/Metasets .
For PointDA [25], we compare PDG with current 3D domain adaptation methods which use target domain data in training procedure, including DANN [38], PointDAN [25], RS [39], DefRec +
PCM [40], GAST [41] and GLRV [42].
We also compare it with state-of-the-art 3DDG method, i .
e .,
MetaSets [22], in which target domain data is inaccessible during training.
5.3 Implementation details
For our PDG, we adopt PointNet and DGCNN as backbones of feature extractor f θ and classifier
C ψ .
For fair comparison, we do not change the architecture of backbone and train all methods except
MetaSets for 160 epochs with batch-size 32 on one NVIDIA V100 GPU.
MetaSets is trained forepochs with batch-size 32 on two NVIDIA V100 GPUs.
We use Adam as the optimizer.
The initial learning rate and weight decay are 10 − 3 , 10 − 4 .
The learning rate reduced to 10 − 5 following a cosine quarter-cycle.
The part number M , points number k in each part, and the number of part-template features N H are respectively set to 8, 512, 384.
λ p and λ C in training loss are 0.05 and 0.01.
5.4 Experimental Results
Comparison with single domain state-of-the-art methods.
As shown in Table 1, PDG with
DGCNN as backbone outperforms previous state-of-the-art 3D classification methods by more than 5.5% in average accuracy of four tasks.
PointMLP [4] is a powerful method and achieves state-of-the-art results on ModelNet40 (94.1%) and ScanObjectNN (83.9%) for single domain point cloud classification, as shown in their paper.
Compared with it, our PDG (DGCNN) shows better generalization ability and outperforms them by 5.5%.
Compared with PointNet [5] and
DGCNN [6], PDG with PointNet and DGCNN as backbone networks surpass them on every 3DDG task, improving average accuracy by 4.2% and 6.2% respectively.
We also find that DGCNN [6]
[Table]
[Table]
outperforms PointNet [5] by only 0.1% while PDG (DGCNN) outperforms PDG (PointNet) by 2.1%.
This demonstrates that, given the part-level features better encoding local geometric structures, our aligned part-based representations could further improve the robustness and discrimination ability.
These significant performance gains (4.2% and 6.2%) of PDG over the baseline prove the effectiveness of our designed part-based domain generalization network, which improves both the generalization and discrimination abilities of point cloud classification models.
Comparison with 3DDA and 3DDG methods.
For Sim-to-Real [22], MetaSets [22] is a stateof-the-art 3D domain generalization model which meta-learns point cloud representations from some point transformation tasks.
As shown in Table 1, with the same backbone network PointNet and data augmentation, PDG outperforms MetaSets by 6.8%.
When equipped with DGCNN, the improvement is 8.2%.
In contrast with MetaSets which aims to improve the generalizable ability of shape-level features, our proposed PDG reduces geometry shift more effectively by the part-based feature representation.
For PointDA [25], from Table 2, PDG improves the baseline methods by 5.0% in the average accuracy of all tasks and outperforms 3DDG method Metasets [22] by 2.7%.
Specifically, PDG beats MetaSets in four tasks M → S ⋆ , S → M , S → S ⋆ , and S ⋆ → M .
For two synthetic-to-real tasks M → S ⋆ and S → S ⋆ , the improvements are 5.6% and 7.9%.
We can also find that PDG exceeds some 3DDA methods including DANN [38], PointDAN [25], and RS [39].
5.5 Ablation study
In this section, we take a closer look at the effects of components of our network, including crossattention module for feature alignment, part-level aggregation module and contrastive learning framework upon part-based shape representation.
We choose PointNet [5] as backbone network referred as “baseline” and experiments are conducted on M → SO task on Real-to-Sim [22] dataset.
We also investigate the effects of the scale of the parts and visualize the both part-level features and shape-level features of the baseline and our PDG.
Results of various architectures of PDG are presented in Table 3.
[Figure]
[Table]
Effectiveness of each component.
We first replace cross-attention module with optimal transport to find the transformation map T for feature alignment as discussed in Sect. 3.2.
Table 3 shows the results of
PDG employing optimal transport for feature alignment, which is denoted as PDG-OT.
Compared with
PDG-OT, PDG with cross-attention improves the performance by 2.4%.
As mentioned in Sect.
3.3, we design a part-based feature aggregation module to aggregate the aligned part-based features according to their confidence score for classification.
We compare it with two baseline pooling operations, i
[Table]
e ., max-pooling and average-pooling.
As shown in Table 1, PDG with part-level aggregation improves PDG-max and PDGavg by more than 2.2%.
This shows that our part-based feature aggregation module is a superior way to aggregate the aligned part-based features and brings a significant performance boost to our network.
We finally examine the effects of the contrastive learning framework defined in Sect.
As shown in
Table 3, with the contrastive learning framework, our method achieves better results than PDG (w/o
CL)
with 1.2% improvement.
We also remove the shape-level contrastive loss in PDG, denoted as
PDG (w/o SCL), which performs slightly worse than PDG by 0.5% (0.1%), while still outperforming baseline by 7.3% (7.8%).
This proves the effectiveness of our contrastive learning framework to improve the robustness of the aligned part-based representations and the major performance gain of
PDG is derived from the design of part-based feature representation.
Selection the scale of part.
As discussed in Sect.
2, the scale of the part-level features will influence the generalization and discrimination abilities.
Specifically, the discrimination ability of part-level features decreases when the scale reduces, while the generalization ability improves.
We show the results of PDG with 8 parts (512 points in each part) and 16 parts (256 points in each part) in Table 3,
PDG with 8 parts (i.e., PDG in Table 3) outperforms PDG-16parts by 2.2%.
Thus we choose 8 parts with 512 points in each part for a better balance of discrimination and generalization abilities.
Feature visualization and analysis.
We first visualize the learned shape-level and part-level features of PointNet [5] of class “chair” and “sofa” from source domain ( M ) and target domain ( S ⋆ ) in Fig.
We can observe that in Fig.
4 (a), the shape-level features in source domain display a high inter-class variance, while the features of “sofa” in target domain are closer to the features of “chair” in source domain due to the geometry shift between two domain data.
This implies better discrimination ability of shape-level representation in source domain while worse in target domain.
Conversely, the part-level features of distinct domains and classes cluster together as shown in Fig.(c), confirming our idea that the local geometric structures represented by part-level features are shared across different domains and classes and could reduce geometry shift effectively.
However, the less discriminative part-level features are hard to be aggregated into a global representation for classification.
Our PGD solves this problem effectively as shown in Fig.
4 (b)(d).
Our aligned part-based representations in Fig.
4 (d) improve the discrimination ability of part-level features while keeping the generalization ability.
And in Fig.
4 (b), we observe both lower inter-domain variance and higher inter-class variance in the global representations of PDG compared with shape-level features of PointNet in Fig.
4 (a), resulting in the improvement in classification accuracy (67.6% vs. 59.8%).
Related Work
Deep classification on point clouds.
The early work PointNet [5] processes points independently using MLP and aggregates them by a max-pooling to obtain the permutation invariant representation, ignoring the vital local geometric structure of point clouds.
DGCNN [6] models point clouds as dynamical graphs and proposes a dynamic edge convolution for classification of point clouds.
Succeeding methods [1–4, 6–10] achieve improved performance for 3D shape recognition.
However, all these methods assume that the training and test data are from the same domain thus can be weak at generalizing learned representations to new domains.
A method that could improve the domain generalization ability of point cloud representation is desirable.
Domain Generalization and Adaptation for Point Cloud classification.
How to represent 3D point clouds in different domains is an old but unsolved problem.
[44] is one of the first methods to explore the 3D domain adaptation problem, which leverages synthetic scans of 3D scenes from
Google 3D Warehouse to train an object detection system for 3D point clouds real-scan data.
[45] and [46] applied hough transform to the problem of robust 3D shape feature learning and evaluated on point cloud classification and retrieval tasks of 3D domain generalization problem.
[47] factored low-dimensional deformations and pose variations of the 3d shapes and recognized them in the scanned cluttered indoor scene, which is also a 3D domain generalization problem.
[48] focused on the CAD-to-scan retrieval task of the 3D domain generalization problem and introduced a method called CAD-Deform to obtain accurate CAD-to-scan fits by non-rigidly deforming retrieved CAD models.
Recently, domain adaptation or generalization problem on deep point cloud classification network [22, 25, 40–42, 49–52] has drawn more attention.
PointDAN [25] proposes a 3D domain adaptation network for point cloud classification by minimizing an MMD loss to align local features across domains.
[40–42, 49–52] further improve the results by designing a series of self-training tasks.
However, all these methods are designed for 3D domain adaptation with known target domain data distribution.
MetaSets [22] explores 3D domain generalization problem that the target domain data is inaccessible when training, and proposes to meta-learn point cloud representations from a group of point cloud transformations.
Compared with them, we tackle the domain generalization problem by reducing geometry shifts using aligned part-based representation.
Experiments on 3DDA and 3DDG benchmarks show that our PDG outperforms Metasets method by a large margin.
Contrastive self-supervised learning.
Contrastive learning [53] which learns representation by contrasting positive pairs and negative pairs has achieved increasingly success for self-supervised learning [35, 54–59].
The learned representation could capture information shared between different augmentations of the same input.
In this paper, our motivation is that aligned part-based representation should be similar under different local point transformations.
To this end, we propose a contrastive learning framework upon part-based shape representation and improve the robustness of aligned part-based representation effectively.
Conclusion
In this work, we propose a novel part-based domain generalization network for 3D point cloud classification.
We propose to learn generalizable part-based feature representations of point clouds to reduce the geometry gap of distinct domains.
A contrastive learning framework is further designed to improve the robustness of the part-based feature representation.
Extensive experiments justify its effectiveness.
The current method is designed for point cloud classification, and we plan to extend it to point cloud segmentation or detection in the future work.
Acknowledgment This work was supported by NSFC with grant numbers of 11971373, 12125104,
U20B2075, 12090021, 61721002.