C ONTRASTIVE L EARNING OF 3D S HAPE D ESCRIPTOR WITH D YNAMIC A DVERSARIAL V IEWS
Anonymous authors
Paper under double-blind review
A BSTRACT
View-based deep learning models have shown the capability to learn 3D shape descriptors with superior performance on 3D shape recognition, classiﬁcation and retrieval.
Most popular techniques often leverage class labels to train deep neural networks under supervision to learn to extract 3D deep representation by aggregating information from a static and pre-selected set of different views used for all shapes.
Those approaches, however, often face challenges posed by the requirement of a large amount of annotated training data and the lack of a mechanism for the dynamic mining of shape-instance-dependent views towards the learning of more informative 3D shape representation.
This paper addresses those two challenging issues by introducing the concept of adversarial views and developing a new mechanism to generate 2D views for the adversarial training of a selfsupervised contrastive model for 3D shape descriptor, denoted as CoLAV.
In particular, compared to the recent advances in multi-view approaches, our proposed
CoLAV gains advantages by leveraging the contrastive learning techniques for self-supervised learning of 3D shape representations without the need for labeled data.
In addition, CoLAV introduces a novel mechanism for the dynamic generation of shape-instance-dependent adversarial views as positive pairs to adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation.
Comprehensive experimental results on 3D shape classiﬁcation demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other existing view-based descriptor learning techniques, even though CoLAV is completely self-trained with unlabeled 3D datasets (e.g., ModelNet40)
I NTRODUCTION
The impressive performance of Deep Neural Networks (DNNs) for 2D visual feature learning naturally motivates 3D computer vision researchers to develop deep learning models for 3D representation for downstream 3D tasks (e.g., shape classiﬁcation (Qi et al., 2017b), shape correspondence (Li et al., 2019), shape retrieval (Han et al., 2018), shape registration (Miao et al., 2016).
To this end, researchers developed the ﬁrst family of 3D deep learning models that operate directly on 3D raw data represented by point clouds (Qi et al., 2017b; Shi et al., 2020), triangulated meshes (Hanocka et al., 2019), or 3D regular voxels (Choy et al., 2019).
The recent advancement of approaches in this category has signiﬁcantly contributed to the shape classiﬁcation, correspondence, segmentation, registration, and detection.
The second important family of 3D deep learning models starts with rendering multiple 2D projections of a 3D object from a static and pre-selected set of different views, followed by learning the compact 3D representation via fusing the deep feature maps of these 2D views.
As they can straightforwardly leverage the latest developments of convolutional neural networks, the view-based methods have shown a very attractive capability to learn 3D shape descriptors with superior performance on 3D shape recognition, classiﬁcation, and retrieval (Li et al., 2020;
Yuan & Fang, 2020; Shi et al., 2020).
Particularly, recent advances in view-based approaches lead to impressive state-of-the-art performances in various 3D vision tasks such as 3D shape classiﬁcation, retrieval, correspondence, and segmentation (Wei et al., 2020).
The most popular techniques often leverage the ground truth labels to train deep neural networks and learn to extract deep representations by aggregating information from a static and pre-selected set of different views.
Those approaches, however, often face challenges posed by the requirement of a large amount of annotated training data and the lack of a mechanism for the adaptive selection of shape-instance-dependent views towards the learning of more informative 3D shape representation.
This paper addresses those two challenging issues by introducing the concept of adversarial views (Xiang et al., 2019) and developing a new mechanism to generate views for the adversarial training of a self-supervised model for 3D representation learning, denoted as CoLAV.
Speciﬁcally, different from the recent advances in multi-view approaches (Kanezaki et al., 2018), CoLAV gains advantages by learning the 3D shape representation without the need for labeled data.
In addition, in contrast to using a static set of views as input during the training of a multiview-based 3D shape representation learning framework, we take a further step to investigate the effect of using dynamically rendered views as input during the 3D shape representation learning.
To this end, we introduce an adversarial view generator that enables the dynamic generation of shape-instance-dependent adversarial views, which can fool the deep learning model to misidentify its instance-wise identity, as positive pairs to adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation.
Our pipeline can be found in Figure 1.
Speciﬁcally, our model takes a 3D shape instance x as input, and we ﬁrst generate an augmented 3D shape t ( x ) by adopting a set of augmentation operations t sampled from a pool of predeﬁned augmentations.
Then, we use a randomly initialized a rotation matrix (characterized by α , β , and γ ) to generate rotated 3D shape x adv 0 .
For each 3D instance x , t ( x ) , and x advwe render k views using the same differentiable renderer r τ ( · ) .
The rendered views are fed to a descriptor learning module to extract shape descriptors.
After we obtained the descriptors, we use a contrastive loss to measure the mutual information of descriptors.
Two key components are employed in our 3D shape unsupervised descriptors learning process.
The ﬁrst one is Adversarial view .
We use the contrastive loss to calculate the gradient with respect to α , β , and γ and update the rotation matrix using the updating rule which introduce in the method section.
This rotation matrix updating is represented by the purple path.
The second component is the contrastive learning module with a shape-instance-dependent adversarial view attack , represented in the red path, which is adopted towards the learning of more informative and robust 3D shape representations.
In this component, a new 3D instance x advi , denoted as adversarial 3D instance, can be obtained using the updated rotation matrix.
We adopt the same differentiable renderer r τ ( · ) to render adversarial views for the adversarial training of the Descriptor learning module in order to produce robust and informative 3D shape descriptor.
We summarized our contribution as follows: 1) We introduce a novel concept of adversarial views of a 3D shape and develop a new mechanism to adaptively and dynamically render shape-instancedependent views towards the learning of more informative 3D shape representation via CoLAV. 2)
The proposed CoLAV is capable of learning 3D shape representations from multiple 2D views without requiring ground truth annotations.
Our comprehensive experimental results on 3D shape classiﬁcation and retrieval demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other techniques, even though CoLAV is completely self-trained with unlabeled 3D datasets (e.g., ModelNet40).
R ELATED W ORKS 2. M ULTI -V IEW L EARNING
In recent research, multi-view-based methods have become a popular topic in 3D computer vision.
Basically, these methods start with rendering projections of 3D objects from different view perspectives, generate representation features for each projection, and merge them into a global descriptor of the 3D objects.
The compact descriptor can be used to improve performance and robustness in 3D object learning tasks such as classiﬁcation and retrieval.
One of the earliest methods in multi-viewbased learning is called multi-view convolutional neural network (MVCNN)
(Su et al., 2015), which fuses feature representations extracted from multiple views into one compact descriptor by a maxpooling layer.
Ruizhongtai Qi et al.
(2016) advances the progress by incorporating a multi-resolution component into MVCNN to boost the classiﬁcation accuracy.
However, using a max-pooling layer to fuse representation features from multiple views can only retain the maximum value on each dimension leading to possible losses of information.
The group-view convolutional neural network (GVCNN) (Feng et al., 2018) is proposed to solve this problem, which generates a descriptor via a grouping strategy that keeps more information from different views.
There is also research attaching attention-based models to the framework, which leverages spatial relationship among different views’ feature representation (Han et al., 2018; 2019).
Yang & Wang (2019) also leverages the relationship between views and constructs a model that connects corresponding regions on each projection and strengths the representation learned from each individual viewpoint.
The aforementioned view-based methods have achieved promising performance on various tasks.
However, their training process requires a large amount of annotated data.
To alleviate the dependency of annotated data, Ho et al.
(2020) proposed a self-supervised method that randomly selects a view feature out of several pre-selected static views of a 3D object as a prototype to learn an invariant stochastic object embedding which is further used to generalize the method (Ho et al., 2020) on test data.
Different from all previous view-based methods, we investigate the effect of using dynamic views.
Instead of using pre-selected static views, in our work, dynamically rendered views are used to learn an informative multi-view-based 3D object representation, meanwhile, no annotated class labels are given.
This encourages the proposed model to better generalize on various unseen views.
2. A DVERSARIAL L EARNING
In the concept of adversarial learning, adversarial examples are treated as hard positive samples in the model training process to improve the robustness of neural networks (Szegedy et al., 2013).
Goodfellow et al.
(2014) introduced the fast gradient sign method (FGSM) which increases the robustness of the model by perturbing input samples along with gradient directions.
Following this direction, numerous methods (Kurakin et al., 2016; Papernot et al., 2016; Shafahi et al., 2019; Xie & Yuille, 2019; Xie et al., 2020) were proposed and have obtained impressive robustness against adversarial attack in image space.
Recently, Zeng et al.
(2019) claimed that instead of generating adversarial samples in 2D space, ﬁnding adversarial 3D objects behind the projected 2D images is crucial to improve the robustness.
In their work (Zeng et al., 2019), FGSM is used to perturb the input sample in 3D space governed by a supervised loss, and ground truth class labels are required to generate adversarial 3D objects.
Unlike Zeng et al.
(2019), in this work, we introduce a new approach to generate adversarial views based on 3D objects without using ground truth labels.
2. C ONTRASTIVE L EARNING
The concept of contrastive learning has been extensively used in recent studies, especially in selfsupervised (SSL) learning scenarios.
In SSL 3D object learning, contrastive learning can help retrieve an invariant descriptor for each 3D object by applying random augmentations to each object and minimizing the contrastive loss by measuring the similarity between the feature vectors repre- senting original and augmented shapes.
There have been a certain amount of augmentation strategies that came up by previous studies in the 2D computer vision ﬁeld (Wu et al., 2018; Ye et al., 2019;
Tian et al., 2019), implementing data augmentation by rotation, cropping, and other transformation methods.
Such augmentations can force the network to learn sophisticated descriptors to represent input instances while training, thus boosting robustness.
By comparing different methods of augmentation, Chen et al.
(2020) indicates that the effectiveness of contrastive learning can be enhanced by employing a stronger data augmentation.
On the other hand, although we have various augmentation methods to boost performance, it is arduous for contrastive learning models to overcome the hard negative mining problem mentioned in Sohn (2016); Wu et al.
; Schroff et al.
One of the possible solutions to that problem is to select negative samples from a distribution of noise that treats all negative instances equally, which is proposed by the contrastive noise estimation (NCE) (Gutmann & Hyv¨arinen, 2010), by which contrastive learning is directly inspired Bose et al.
; Bose et al.;
Ho & Vasconcelos (2020).
M ETHODS
The pipeline of our proposed module is illustrated in Fig.
The CoLAV is a generic module that generates instance-wise adversarial views to adversarially train a contrastive learning model towards the goal of learning a more informative and robust shape descriptor for a given task.
3. A DVERSARIAL V IEWS
We introduce our novel approach to obtain adversarial views while alleviating the dependency of annotated data.
As we illustrated in Section 2, although many multi-view-based methods have been proposed, their performances rely on contents of input views which are controlled to camera positions and annotated data.
As we observed from existing works (Su et al., 2015; Feng et al., 2018), camera positions are static across the entire 3D shape dataset, which leads the trained model vulnerable to unseen views.
To address this issue, we introduce the concept of adversarial views as hard positive training samples during the model training, which aims to fool the predictor to make wrong predictions by maximizing the objective function.
Adversarial Views Generation.
Concretely, the generation of dynamic adversarial views depends on the generation of an adversarial 3D sample x adv .
To generate adversarial 3D samples, existing researches focus on perturbing 3D point positions within a certain range in 3D space.
This perturbation is almost indistinguishable by human eyes, i.e., a perturbed 3D sample is visually identical to the original 3D object.
Different from the existing adversarial approaches, we propose a method to generate visually different adversarial views of each 3D instance with the aim to fake the model to treat the generated views as new instances.
Given a 3D shape x from the dataset D , a differentiable renderer r τ ( · ) : x → I , a rotation matrix M controlled by three randomly initialized parameters α , β , γ , a descriptor learning model f θ ( · ) and a task-speciﬁc predictor g φ ( · ) , the generation of x adv is deﬁned as:
[Equation]
where i denotes the generation iterations and ˆ M denotes the updated rotation matrix controlled by ˆ α, ˆ β, ˆ γ which are formulated as follow:
[Equation]
where L is the loss for training the neural network f θ ( · ) and g φ ( · ) with parameter φ and θ , the sign ( · ) gives the sign of gradients, and (cid:15) represents the step size of one gradient attack.
We employ a function p ( · ) as projection function to clip three angles ˆ α, ˆ β, ˆ γ with in [ − 180 ◦ , 180 ◦ ] .
The adversarial views I adv is rendered by:
[Equation]
To train a neural network using adversarial samples generated from Equation 12, we can directly minimize the L over I adv .
This adversarial training process can be formulated as follow:
[Equation]
The computation of loss term L φ,θ , such as cross-entropy loss requires the ground truth labels, which prevents us from applying this approach to unannotated datasets.
To address this, we alleviate the dependency of ground truth annotations by adopting a self-supervised contrastive loss.
Self-Supervised Adversarial Views Generation.
Speciﬁcally, given a pool of augmentation operations T , such as random rotation, random jittering, random scaling, random noise, and random dropout, we randomly select a set of augmentation operations t ∼ T .
The speciﬁc data augmentation operation that we adopt in our method can be found in Section 4.2.
Our goal is to maximize the similarity between f θ ( r ( x )) and f θ ( t ( r ( x ))) , since r ( x ) and t ( r ( x )) have the same instance-level proprieties, and to minimize the similarity between f θ ( t ( r ( x ))) and f θ ( t ( r ( x (cid:48) ))) , where x (cid:48) is a different 3D shape instance.
For a clear illustration, we denote r ( x ) as I , t
( r ( x )) as I pos , and t ( r ( x (cid:48) ))
as I neg .
Then, the constrastive learning process can be formulated by minimizing the following loss function:
[Equation]
where we follow the SimCLR (Chen et al., 2020) to adopt Γ as a temperature parameter and the sim ( · , · ) function, which is deﬁned as follow:
[Equation]
By adopting this formulation, we can reformulate Equation 2 and make it self-supervised, which relaxes the constrain of requiring annotated datasets during adversarial view generation.
The selfsupervised version of adversarial view generation can be formulated as follow:
[Equation]
3. C ONTRASTIVE L EARNING WITH S HAPE -I NSTANCE -D EPENDENT A DVERSARIAL V IEW A TTACK
Once we have the adversarial views as our positive samples, generated from Equation 12, we can learn a robust and informative 3D shape descriptor through contrastive learning.
The learning process can now be formulated as follow:
[Equation]
The ﬁnal loss function L 3Dco can be described as:
[Equation]
[Equation] where { x pos } is the positive 3D sample set which is composed by { t ( x ) , t ( x adv ) } and { x neg } denotes the set of negative 3D samples { t ( x (cid:48) ) } .
The training of CoLAV can be found in Algorithm 1.
E XPERIMENTS
In this section, we conduct experiments to evaluate the performance of our proposed method on two benchmark datasets.
More speciﬁcally, we evaluate our module when adopting various multiview-based backbones, including GVCNN (Feng et al., 2018) and MVCNN
(Su et al., 2015), as our descriptor learning module.
Section 4.1 provides implementation details of our model and a description of datasets used in our experiments. In the following section, we carry out experiments to classify and retrieve 3D objects. In Section 4.3, we demonstrate that the descriptor extracted by our model is robust to rotation. In Section 4.4, we further investigate our model’s effectiveness by using different settings, such as different representations of 3D inputs, different numbers of views, etc.
Input: Original 3D shapes: x , augmented 3D shapes: t ( x ) , augmented negative 3D shapes: t ( x (cid:48) ) , descriptor learning model: f , parameter of model: θ , adversarial view generation iterations: i ; for all iteration ∈ training stage do for all x ∈ minibatch do for attack iteration ∈ [0 , i ) do update rotation matrix M .
end for generate adversarial views.
end for calculate objective function L 3Dco using Equation 9.
update model parameter θ end for
4. D ATASET , I MPLEMENTATION AND T RAINING D ETAILS
Dataset: We conduct experiments on the ModelNet40 (Wu et al., 2015) dataset.
ModelNet40 is one of the ﬁrst benchmark datasets that were acquired from CAD objects.
This dataset contains 12, high-resolution, watertight 3D shapes in the mesh format, and all 3D meshes are labeled intocategories.
In our experiment, we apply the standard splitting of the dataset.
We use 9,843 shapes for training and 2,468 shapes for testing.
In addition to ModelNet40 benchmark dataset, we also evaluate our method on ScanObjectNN (Uy et al., 2019) dataset.
Due to the page limit, we include this part in our supplementary materials.
Implementation and Training Details:
The entire model is implemented in Pytorch.
An Adam optimizer with a batch size of 16 is used to train the entire network.
The momentum was 0.9, and the weight decay rate is set to 10 − 5 .
The learning rate starts at 0.0005 and then decreases by a factor of 0.5 every 20 epochs. [Table] Shi. et.
al.
Shi et al. (2020) point cloud (cid:55) 84. VoxNet Maturana & Scherer (2015) voxel grid (cid:88) 83. SubvolumeSup Qi et al.
voxel grid (cid:88) 89.
Voxception Brock et al.
(2016) voxel grid (cid:88) 91.
- 3D-DescripNet Xie et al.
(2018) voxel grid (cid:55) 83.
- 3D-GAN Wu et al.
(2016) voxel grid (cid:55) 83.
T-L Network Shi et al.
(2020) image + voxel (cid:55) 74.
MVCNN Su et al.
(2015) 12 views (cid:88) 89.
MVCNN, metric Su et al.
(2015) 12 views (cid:88) 89.
RotationNet Kanezaki et al.
(2018) 12 views (cid:88) 90.
MHBN Yu et al.
(2018) 12 views (cid:88) 93.
- 3D2SeqViews Han et al.
(2019) 12 views (cid:88) 93.
MVCNN Su et al.
(2015) 12 views (cid:88) 92.
MVCNN (G.) Su et al.
(2015) 12 views (cid:88) 92.
GVCNN (G.) Feng et al. (2018) 12 views (cid:88) 92. Ours (MVCNN) dynamic views (cid:55) 93.
Ours (GVCNN) dynamic views (cid:55) 94.
and retrieval performance of our model with existing methods on the ModelNet40 dataset.
Our model is trained without any supervision of ground truth but still outperforms most of the supervised methods on classiﬁcation accuracy and retrieval mAP.
“(Go.)” represents the methods use GoogLeNet as backbone network.
4. R ESULTS ON M ODEL N ET3D Object Classiﬁcation:
To the best of our knowledge, our work is the ﬁrst to extract 3D shape descriptors from dynamic views without using ground truth labels.
To validate the generalization ability of our proposed model, we conduct experiments for the task of 3D object classiﬁcation on the ModelNet40 benchmark and compare our method against various multi-view baed 3D shape descriptor learning methods.
In this comparison, we compare not only with multi-view-based methods but also with other types of 3D representation, such as point clouds and voxel grids.
For the 3D object classiﬁcation task, as part of the inputs, we employ random rotation, random scaling, and random jittering as t ( · ) to generate t ( x ) .
Speciﬁcally, for a mesh object x , we randomly rotate the object around three axes (x, y, z) and randomly scale the mesh object in the range of [0 . 75 , 1 . 15] .
After random rotation and random scaling, we apply random jittering on each vertex of the 3D mesh.
For this augmentation operation, we add a random displacement to each point and clip this displacement in the range of [ − 0 .
After random jittering, the connection of all faces remains the same as origin 3D instance, which preserves the watertight property of 3D mesh.
We use the same data augmentation but apply on another 3D shape instance to produce t ( x (cid:48) ) as a negative sample.
In this experiment, we adopt Pytorch3D as the renderer r τ ( · ) to render 2D views for each 3D instance on-the-ﬂy.
We render k = 12 images of each 3D shape instance and use i = 5 to generate adversarial views.
For each attack iterations i , we use step size (cid:15)
= 0 .
015 to update rotation parameters α, β, γ .
We use
GVCNN and MVCNN as two different backbone networks to generate 3D shape representation z .
We evaluate the unsupervised learned 3D shape representation z by using a linear SVM with default parameters.
Experiment Results: Table 1 shows the classiﬁcation accuracy of our proposed model as well as other methods on the ModelNet40 dataset.
In spite of the fact that our method is completely selfsupervised, our MVCNN based model achieves 93.6% accuracy on 3D object classiﬁcation on the
ModelNet40 dataset, which surpasses the classiﬁcation accuracy of previously proposed supervised methods listed on the table.
Our GVCNN based model achieves 94.3% accuracy, which further improves the classiﬁcation accuracy and surpasses the supervised methods.
Compared with MVCNN,
Ours(MVCNN) has an improvement of 1.5% on accuracy.
Compared with GVCNN, Ours(GVCNN) outperforms GVCNN by 1.7% of accuracy.
Another notable difference is that in previous multiview-based methods on the table, views are all generated statically before training.
In our model, new sets of views can be generated for each training epoch.
Throughout the entire training process, different views can be dynamically rendered for the training of descriptor learning network to obtain more robust and informative 3D shape descriptors compared with existing methods.
To this end, an advantage of dynamically view generation raises, compared with static view-based multi-view methods.
3D Object Retrieval: Furthermore, we follow the same training setups that we presented above to conduct the experiment on the task of 3D object retrieval.
For each 3D object, its respective shape representation is used to perform the object retrieval.
We calculate the distance between every two latent representations to conduct retrieval.
Speciﬁcally, given a learned 3D shape representation z of a query shape x , we aim to ﬁnd similar shapes by comparing the distance between z and every other shape descriptors.
To evaluate the shape retrieval performance, we follow the stander 3D shape retrieval evaluation metric (mAP).
Experiment Results: As shown in the last column of Table 1, our MVCNN based self-supervised method obtains 85.3 mAP when performing 3D object retrieval tasks on the ModelNet40 dataset, and GVCNN based method achieves the mAP of 85.9, which exceeds the performance of most of the supervised methods proposed by previous studies.
Furthermore, compared to our baseline method
MVCNN and GVCNN, our methods improve retrieval mAPs by 2.3 and 4.6 respectively.
4. R OBUSTNESS TO R OTATION
In this section, we conduct further experiments to explore the resistance to rotation.
Since our model dynamically generates 2D views and performs adversarial attacks to force the network to take the worst views into consideration, our model is expected to be highly robust to the rotation.
This test is designed to measure our performance dealing with 3D objects which are rotated into different angles.
Experiment Setup:
In this experiment, we test the rotation robustness of our model on the ModelNet40 benchmark dataset.
We rotated the testing data around a random axis by either 90 ordegrees and reported the performance.
Experiment Results:
As shown in Table 2 our self-supervised network with both 6 views and 12 views conﬁguration achieves attractive robustness to the rotation.
Dealing with 90 degrees of rotation along the gravity axis, our 6 views model performs 90.8 accuracy of classiﬁcation, which is the best among all the previous methods, including all the supervised methods listed on the table.
For 180 degrees of rotation along the Y-axis, our 12 views model outperforms all the previous methods, obtaining 90.9 classiﬁcation accuracy. Method Modality Supervision 90 ◦ 180 ◦
PointNet (Qi et al., 2017a) point cloud (cid:88) 42.
PointNet++ (Qi et al., 2017b) point cloud (cid:88) 47. RSCNN (Hu et al., 2021) point cloud (cid:88) 90. MVCNN (Su et al., 2015) 12 views (cid:88) 84. ViewGCN (Wei et al., 2020)
12 views (cid:88) 86.
GVCNN (Feng et al., 2018)
12 views (cid:88) 88.
Ours(MVCNN) 6 views (cid:55) 89.
Ours(MVCNN) 12 views (cid:55) 90.
[Table]
4. A BLATION STUDY
In this section, We study the effect of the components of our model.
First, we investigate the effect of the number of views during training and testing, then analysis of the adversarial views generated by our network is presented.
Due to the page limitation, the in-depth experiment conducted to explore the effect of different augmentation operations we adopted during the model training process, the effectiveness of our proposed unsupervised loss and the effectiveness of adversarial view are included in the Appendix.
[Table]
4.4. E FFECT OF N UMBER OF V IEWS :
In this study
[Table]
we investigate the impact of different numbers of views on our model performing 3D object classiﬁcation.
We apply { 8, 12 } views to train our model and retrain the MVCNN and
GVCNN.
We adopt { 2, 4, 8, 12 } views during the testing phase.
As it is listed in Table 3, our model outperforms our baseline GVCNN and MVCNN in all the combinations of training and testing views selected.
Even when we employit two views for testing, which is arduous for other methods, our model can still achieve a high level of accuracy.
We attribute this success to the adversarial attack procedure that generates challenging positive samples for the network to learn from and causes the network to draw informative descriptors from the two views selected via contrastive learning.
4.4. A DVERSARIAL V IEW A NALYSIS :
We analyze the effectiveness of our adversarial attack procedure by observing the adversarial views generated by our model.
As shown in Figure 2, with i deﬁned in Equation 12 representing the number of iterations of adversarial attack, we can see that our model will gradually rotate the adversarial views to challenging angles which depict less informative details of the 3D object.
When i = 5 , the adversarial views of the sofa are highly ambiguous that even human eyes cannot recognize the object easily.
Moreover, with the help of the contrastive learning procedure in the following steps, the network is expected to draw a more sophisticated and invariant descriptor for each 3D object to enhance classiﬁcation accuracy on these challenging views.
C ONCLUSION
In this paper, we introduce a novel concept of adversarial views generated via a self-supervised contrastive learning mechanism that augments the positive training samples by performing shapeinstance-dependent adversarial attacks and introduce a multi-view pipeline that directly takes 3D instance as input.
More importantly, different from the previous multi-view-based methods using pre-rendered static views for the training of deep neural networks, our proposed method can dynamically generate views for a given 3D instance on the ﬂy without requiring additional network parameters.
Extensive experiments conducted on different benchmark datasets demonstrate the effectiveness of the newly introduced adversarial views.
Our self-supervised method boosts the 3D classiﬁcation accuracy and 3D retrieval mAP to outperform the supervised methods previously proposed.
In addition, ablation studies of our model prove the effectiveness and efﬁciency of the proposed components.
Our method highly improves robustness to rotation thanks to the adversarial views dynamically generated by our adversarial attack procedure.
A A PPENDIX A. D ETAILED I MPLEMENTATION
We use the Adam optimizer with a batch size of 16 is used to train the entire network.
The momentum was 0.9, and the weight decay rate is set to 10 − 5 .
The learning rate starts at 0.0005 and then decreases by a factor of 0.5 every 20 epochs.
We train the entire network in 180 epochs and we use a 1024-d latent vector to represent the learned descriptor.
To evaluate the shape retrieval scores, we extract 3D descriptors using the trained network.
We sort the most relevant shapes for each query by cosine distance and compute the mAP score.
A. 3D SHAPE RETRIEVAL TASK ON M ODEL N ET
We show examples of our 3D object retrieval in Figure 3.
For each group of 3D shapes, the shape to the left of the vertical line is the query object from the test set, and on the right side of the line, we show the retrieved objects from the training set.
[Figure]
In addition to ModelNet40 benchmark dataset, we also evaluate our method on ScanObjectNN Uy et al.
(2019) dataset.
ScanObjectNN is the newest and one of the most challenging datasets for 3D object recognition.
It is acquired from the real-world objects in 3D point cloud representation and three variants are proposed for different levels of difﬁculty.
1) Vanilla ScanObjectNN dataset contains clean 3D object point clouds.
2) Background ScanObjectNN dataset which includes real-world background information of each 3D object.
Perturbed ScanObjectNN dataset performs random [Table] perturbation on the Background ScanObjectNN dataset to increase the difﬁculty of 3D object recognition tasks.
In our experiment, we adopt the Vanilla ScanObjectNN dataset to evaluate our method.
The reason why we use the ”vanilla” version is that other versions of ScanObjectNN contain backgrounds which could lead the propped adversarial view only to reﬂect the background and occlude the 3D objects.
The ﬁrst variant is composed of 2902 3D point clouds and each 3D point cloud has its unique label from 1 to 15.
ScanObjectNN (Uy et al., 2019) randomly split into a training set (80%) and test set (20%) and we follow this protocol to conduct our experiment.
3D Object Classiﬁcation:
In this section, we conduct experiments for the task of 3D object classiﬁcation on the ScanObjectNN benchmark and compare our method with many other methods retrained on the ScanObjectNN dataset.
In this comparison, we consider multiple representation learning methods taking different 3D modalities as input, such as point clouds, into consideration.
Similar to Section 4.2, we use GVCNN and MVCNN as two different backbone modules f θ ( · ) .
Different from ModelNet40, ScanObjectNN is a point-cloud-based dataset.
To apply the multi-viewbased 3D representation learning method, we re-render 3D objects from multiple viewpoints.
Here, we adopt 12 viewpoints in this experiment.
An elevation angle of π/ 9 is applied for all viewpoints with azimuths sampled form [0 , π ) uniformly.
In our proposed model, we adopt Pytorch3D as a differentiable renderer and use its point cloud rendering utilities.
We also employ random rotation,random scaling, and random jittering operations as t/ ( · ) to generate t ( x ) as we speciﬁed in Section 4.2.
Furthermore, we add the random dropout in t ( · ) operation as the last augmentation.
In this experiment, we use i = 8 to perform the adversarial attack with step size (cid:15) = 0 .
005 to update rotation parameters α, β, γ .
We adopt the same default linear SVM from the scikit-learn library to perform the downstream task of 3D object classiﬁcation.
Experiment Results:
In Table 4, we compare the classiﬁcation accuracy of our model with other retrained methods, including PointNet (Qi et al., 2017a), PointNet++ (Qi et al., 2017b), KD-Networks (Klokov & Lempitsky, 2017), VoxNet (Maturana & Scherer, 2015), 3D-GAN (Wu et al., 2016),
MVCNN (Su et al., 2015) and GVCNN (Feng et al., 2018), on the ScanObjectNN dataset.
As listed in Table 4, Our two backbones achieve 84 .
1% and 85 .
4% of accuracy on 3D object classiﬁcation, which outperforms existing supervised methods.
This demonstrates our proposed method can effectively ﬁnd challenging views and hence increase the generalize ability of the backbone model.
Speciﬁcally, compared with MVCNN, our Ours(MVCNN) improves the performance by 2% on accuracy and our Ours(GCVNN) network outperforms GVCNN by 2.3% of accuracy.
3D Object Retrieval
In this subsection, we follow the protocol that was presented in Section 4.2 to evaluate our proposed method on 3D object retrieval.
Experiment Results: As shown in the last column of Table 4, our methods achieve the best results compared with other methods listed in the table.
Speciﬁcally, Ours(MVCNN) improves the mAP by 1.9 compared with MVCNN (GoogLeNet), metric (Su et al., 2015).
Ours(GVCNN) achieves the retrieval mAP of 76 . 8 which outperform the baseline result, GVCNN (GoogLeNet)
(Feng et al., 2018) 75.9, by 0.9.
This improvement validates the effectiveness of our proposed method in the 3D object recognition task.
A. E FFECT OF D IFFERENT A UGMENTATION O PERATION :
In section 3.3, we introduce our unsupervised method which leverages the power of contrastive learning which relies on different data augmentation operations.
In this section, we conduct various experiments on ModelNet40 to investigate the effect of different data augmentations.
As presented in Table 5, we adopt two different sets of data augmentation operations for mesh objects.
The ﬁrst set (Set 1) of operations contains random rotation, random scaling, and random jittering.
The second set (Set 2) of operations contains random scaling and random jittering.
The third set (Set 3) of operations only contains random jittering.
In addition to mesh objects, we also conduct experiments on point clouds, and three sets of augmentations are used.
Moreover, a random dropout of point cloud has also been added to t ( · ) .
To validate the performance on the point cloud, we ﬁrst randomly sample 2048 points from the ModelNet40 data set, and we use the point cloud rendering function from the
Pytorch3D library to render 3D point clouds.
From Table 5 we can see that our proposed method achieves the best result when adopting complex data augmentations.
Observing from the second and fourth rows and eighth and tenth rows of Table
we ﬁnd that the rotation operation contributes less during the learning process.
This can be attributed to our proposed adversarial attack which can generate rotated adversarial samples.
[Table] [Table]
A.4. E FFECT OF U NSUPERVISED L OSS :
In this section, we further investigate the effectiveness of our proposed adversarial attack from two aspects.
We train Ours(GVCNN) on the ModelNet40 dataset using 9,843 shapes and compare our proposed method with our method that depreciates the adversarial attack which is a self-supervised multi-view-based 3D representation learning method.
Speciﬁcally, we use random rotation, random scaling, and random jittering as t ( · ) , and we use
[Equation]
as the objective function to unsupervisedly train the network.
The experimental results can be found in Table 6.
Our adversarial attack achieved over 1.0% of improvement on classiﬁcation accuracy comparing with the self-supervised approach.
Moreover, our supervised methods also outperform
GVCNN and MVCNN by 0.4% and 0.1% respectively.
A. E FFECT OF A DVERSARIAL V IEWS :
we further conduct an experiment by applying random rotation (as opposed to the gradient-descent guided rotation) to input 3D instance to generate adversarial views to verify the effectiveness of our proposed adversarial views.
Speciﬁcally, we modify Equation 7 as:
[Equation]
and we follow
[Equation]
) to generate views by using random ∆ α, ∆ β, ∆ γ for each step of adversarial attack, where M is a random initialized rotation matrix, and ˆ M is a rotation matrix controlled by ˆ α, ˆ β, ˆ γ .
We use
GVCNN as the backbone network and follow the same experiment setting in our main paper to train the new module.
We report the classiﬁcation accuracy on ModelNet40, denoted as “Random rotation (multiple steps)”.
Moreover, we compare our model with the one without using adversarial views.
Speciﬁcally, we replace the adversarial view generation from our method with a random rotation operation, denoted as “No adversarial views generation”, to validate its effectiveness.
As
[Table]
presented in this table, our method outperforms other two methods by at least 2%, which proves the effectiveness of our proposed adversarial generation method.
In addition to the aforementioned experiment, we conduct an experiment on the number of backpropagation steps for the rotation matrix to answer this question.
We use i = 1 , 3 , 5 , 7 while the other experimental settings remain the same as we presented in Section 4 of the main submission to validate the effectiveness of our method.
As listed in Table 8, our method reaches the best performance when we set i=5 during the training process.
A. C OMPUTATIONAL O VERHEAD :
We conducted an additional experiment as follows.
We ﬁrst generated a collection of 2D view images ofﬂine through rending the 3D objects sampled from ModelNet40.
Then we compared the computational overhead for the training with the ofﬂine rendered view images and for the training with online view generation (our method).
We report the time of a forward pass for one input
[Table]
[Table]
sample. Due to the differentiable rendering process is implemented under the Pyotroch framework, the rendering process can be accelerated by GPU which can signiﬁcantly increase the rendering speed.
As we can see from the experimental results, our online adversarial views generation during the training does not introduce a signiﬁcant computational overhead.