Unsupervised Point Cloud Completion and Segmentation by Generative Adversarial Autoencoding Network
Abstract
Most existing point cloud completion methods assume the input partial point cloud is clean, which is not the case in practice, and are generally based on supervised learning.
In this paper, we present an unsupervised generative adversarial autoencoding network, named UGAAN, which completes the partial point cloud contaminated by surroundings from real scenes and cutouts the object simultaneously, only using artificial CAD models as assistance.
The generator of UGAAN learns to predict the complete point clouds on real data from both the discriminator and the autoencoding process of artificial data.
The latent codes from generator are also fed to discriminator which makes encoder only extract object features rather than noises.
We also devise a refiner for generating better complete cloud with a segmentation module to separate the object from background.
We train our UGAAN with one real scene dataset and evaluate it with the other two.
Extensive experiments and visualization demonstrate our superiority, generalization and robustness.
Comparisons against the previous method show that our method achieves the state-of-the-art performance on unsupervised point cloud completion and segmentation on real data.
Introduction
In recent years, point clouds have gained more popularity [11] as the standard outputs of 3D scanning devices [17, 39] and the fundamental data structure to represent and process 3D data [7, 6, 19].
However, clean and complete point clouds of objects, on which the downstream applications such as reconstruction [13] significantly rely, are hard to obtain in practice due to the nature of scanners and occlusions.
Point cloud completion which infers a complete object model given a partial point cloud thus has received considerable attention.
Recent supervised methods [42, 41, 31, 26, 36, 34] show remarkable performance on recovering the original shapes from incomplete point clouds, but generally assume that the input point cloud is clean and does not contain any noises and outliers.
Such an assumption does not always hold in practice, because the surroundings around the target object will be inevitably seen by the cameras during scanning real scenes.
Though semantic or instance segmentation on point clouds has been investigated in the literature, the state-of-the-art instance segmentation methods [21, 5] only report around 68% average precision on the public ScanNet [7] and [1] datasets.
That is to say, accurately cutting out the target object from the whole point cloud still remains challenging.
Therefore, it is important for the downstream applications to remove noises from point clouds or directly complete the point clouds with noises.
In this paper, we propose an unsupervised end-to-end network for completing the partial point cloud with noises from real scenes and outputting the object mask at the same time, dubbed UGAAN.
Previous supervised point cloud completion and segmentation works require the paired clean and complete point clouds, together with object-level point labels of real scenes, for training, which are hard and costly to obtain in practice [39, 22].
Most deep-learning-based methods heavily rely on such training data, which limits their applications in practice.
Clearly different from previous methods, our UGAAN only needs unlabeled real-scene objects and unpaired artificial CAD models for unsupervised training.
We employ the framework of GAN [10] which is widely used in unsupervised tasks.
Traditional framework of GAN is hard to converge due to the gap between real-scene data and artificial data.
To resolve this problem, we utilize an autoencoding generator to learn the basic shapes of objects from the artificial data for easier convergence.
Usually, the discriminator could easily distinguish the artificial data as real data and the prediction as fake data, leading to bad performance due to the data gap between predicted point clouds and artificial data.
Our discriminator, by contrast, accepts the prediction of artificial data rather than the artificial data itself as real data, which can reduce the negative effect of data gap.
Our discriminator also takes latent codes extracted from real-scene data and artificial data as the criterion, by which the generator can robustly learn a better latent space for prediction.
We employ a refiner to refine and upsample the first prediction result, and the final complete point clouds thus generated are more accurate and uniform.
Finally, the segmentation module predicts the labels according to the predicted point cloud and input point cloud.
Compared with usual point cloud completion datasets [42] whose objects are aligned by the yaw angles and scales, real-scene datasets such as ScanNet and S3DIS [1] contain objects that have different yaw angles and varied scales, making the prediction of complete point clouds challenging.
To accommodate real-scene data, we randomly rotate and scale the artificial point clouds for easier learning and convergence.
We train our UGAAN on ScanNet with ShapeNet as the artificial dataset without any pre-training or fine-tuning, and then directly evaluate it on the other two real-scene datasets, including S3DIS and
ScanObjectNN [28].
Comparisons with the state-of-the-art methods [22, 39] show the superiority of our method.
The main contributions of our work are as follows
• We propose, for the first time, an unsupervised end-to-end network for completion and segmentation of point clouds with outliers by combining GAN and an autoencoding network.
• Our method completes the real-scene object with outliers from the background without the need of any pre-training.
• We conduct extensive experiments for analysis and comparisons on three different real-scene datasets, the results of which show the superiority and generalization ability of our methods.
The rest of this paper is organized as follows.
Section 2 reviews briefly the works on point cloud completion and segmentation.
Section 3 introduces our UGAAN in detail.
Dataset generation, experiments and comparisons are shown in Section 4 and Section 5.
Section 6 concludes the whole paper and highlights future work.
Related Work 2.1 Point cloud completion and segmentation.
In the early stage, existing successful works [27, 8] utilize voxels as representations of 3D models on which 3D convolutional neural networks can be immediately applied.
PMP-Net [34] and
PMP-Net++ [33] generate complete point clouds by moving input points to appropriate positions iteratively with minimum moving distance.
SnowflakeNet [36] utilizes transformer and point-wise feature deconvolutional modules to refine the first-stage point could multiple times.
For semantic segmentation, Jiang et al.
[15] propose a multi-stage ordered convolution module to stack and encode the information from eight spaces, achieving orientation encoding and scale awareness.
Liang et al.
[21] split nodes of a pre-trained, intermediate, semantic superpoint tree for proposals of instance objects.
Point Transformer [43] introduces transformer in the encoding process of point cloud to learn the representation, which can be applied to different point cloud processing such as segmentation and classification.
2.2 Unsupervised point cloud learning.
Xiao et al.
[37] summarize the unsupervised point cloud representation learning works using DNNs.
Some works [20, 38, 30] propose point cloud unsupervised pre-training approaches, Others proposed unsupervised model [40, 12, 4] achieves excellent results on classification, segmentation and upsampling tasks.
Jiang et al.
[14] introduce two contrastive losses to respectively facilitate downstream classification and segmentation.
Coseg [22] presents a point cloud object co-segmentation task, aiming to segment the common 3D objects in a set of point clouds, and fathom a method with co-contrastive losses to minimize feature discrepancy inside estimated object points and maximize feature separation between the object and background points, which also leads to a weak generalization.
Chen et al.
[39] propose an unpaired point cloud completion method that can be trained without requiring explicit correspondence between partial and complete point clouds by employing an adaptive transform network in GAN, however, the training process of its network is complicated with several pre-training models.
Method 3.1 Problem Formulation
Let P r ∈ M n 0 × 3 represent the point cloud from real-scene data with distribution p r and P a denote artificial data with distribution p a , respectively.
P r contains points belonging to the object and from background (noises).
Obj ( P r ) represents the object points on P r .
Assuming that Obj ( P r ) ∼ p o , the goal of unsupervised point cloud completion and segmentation is to train a generator G satisfying the following two conditions:
G ( P r ; θ g ) ∼ p a , 2.
G ( P r ; θ g ) ∩ P r ∼ p o ,
where G ( P r ; θ g ) represents the point cloud predicted by G with input P r and parameters θ g , and the operation ∩ denotes the intersection of two point clouds 2 .
The two conditions are utilized to constrain completion and segmentation processes of point clouds, respectively.
Condition 1 encourages the predicted shapes to be similar to the shapes in the artificial dataset.
Condition 2 requires that the generator should predict shapes according to P r rather than generating random shapes, and the segmentation result which is the intersection of prediction and P r should match the object points.
3.2 Network Architecture
We employ Generative Adversarial Network (GAN) [10] to solve this unsupervised learning problem.
As shown in Figure 1, our UGAAN consists of an autoencoding generator and a discriminator that plays a minimax game for unsupervised completion and segmentation of real-scene object point cloud.
Furthermore, we also utilize a refiner to refine the predicted complete point cloud and a segmentation module for label prediction.
Autoencoding Generator.
We use PointNet++ [24] as the encoder E of the generator to extract the latent code z of input point cloud P i , and a multi-layer perceptron (MLP) as the decoder to predict the point cloud P o ∈ M n 1 × 3 .
The generator takes artificial data as input to learn basic shapes of point clouds.
Then, we employ a lightweight encoder-decoder refiner [42] R on P o to get the refined and upsampled output P o r ∈ M n 2 × 3 .
Different from [42], the upsampling module directly refines the complete point cloud P o .
Given an input point cloud P i , we can obtain the latent code, the complete prediction and the refinement by z =
E
( P i ; θ g ) , P o =
G ( P i ; θ g ) and P o r =
R ( P o ; θ r ) , respectively, where θ g and θ r are the parameters of generate G and refiner R .
fake / real ?
real
Encoder Encoder
Encoder
Decoder
MLP
Decoder
SegmentationModule
Latent code 𝑧𝑧
Global feature
Global feature
I npu t po i n t c l oud 𝑃𝑃 𝑖𝑖
C o m p l e t
e
po
i n t c l oud 𝑃𝑃 𝑜𝑜
R e f i n e d po i n t c l oud 𝑃𝑃 𝑜𝑜 𝑟𝑟
S e g m e n t a ti on l a b e l
C
AutoencodingGenerator
Discriminator
Refiner
UGAAN:
Unsupervised Generative Adversarial
Network
Generator
Discriminator 𝑃𝑃 𝑖𝑖 ∼𝑝𝑝 𝑎𝑎 𝑃𝑃 𝑖𝑖 ∼𝑝𝑝 𝑟𝑟
Discriminator.
We take advantage of the encoder proposed in [42] to extract the global features f of
P o .
Then the global feature f and the latent code z are combined and fed into a MLP for predicting the probability P :
[Equation]
where P i ∼ p a and θ d is the parameters of the discriminator.
To reduce the negative effect of the shape gap between the prediction and artificial data, we feed the predicted shapes rather than the artificial data into the discriminator.
The discriminator also takes latent codes as the criterion by which the generator can learn better latent codes for prediction.
Segmentation Module.
Finally, the segmentation results (i.e. the object points of p r ) are obtained under the guidance of the complete prediction P o .
Here, we directly calculate the intersection between
P o and P r to get the object points, rather than predicting with another network.
Due to the discrete nature of point clouds, the strict intersection between P o and P r is almost empty.
As a consequence, we make use of general intersection between the point clouds P 1 and P 2 which is indicated by the overlapping area:
[Equation]
where cd p → P 1 will be introduced in Equation 5 with d being the threshold.
The predicted segmentation label of p ∈ P r conditioned on P o with threshold d can be computed as:
[Equation]
Besides, the segmentation label of P r =
[ p T 1 , ..., p Tj , ..., p Tn ] T can be obtained by S d ( P r ; P o ) =
[ S d ( p 1 ; P r , P O ) , ..., S d ( p j ; P r , P O ) , ..., S d ( p n ; P r , P O )] T ∈ M n × 1 .
3.3 Optimization
We modify the optimization strategy introduced by GAN [10] to train our UGAAN on the unsupervised point cloud completion and segmentation problem.
Specifically, the optimization problem of the proposed UGAAN is formulated as:
[Equation]
Due to the data gap between artificial data and real-scene data [39] and the limited fitting ability of the decoder, the terms G ( P r ) and P a often exhibit huge differences as shown in the top left part of Figure1.
It makes D easier to optimize than G which makes network hard to converge and will result in poor completion performance.
Thus, different from the original GAN where the discriminator directly works on G ( P r ) and P a , the discriminator of UGAAN predicts according to G ( P r ) and G ( P a ) to reduce the negative effects caused by these gaps for easier convergence.
Besides, if we impose no other constraints on problem 4, the generator may just generate random points after the optimization process is complete.
Thus, we also impose two losses on problem 4 to restrict the prediction of the generator, namely the autoencoding loss between G ( P a ) and P a , and the reconstruction loss between G ( P r ) and P r .
We employ the well-known Chamfer Distance(CD) [9] to measure the similarity between point clouds.
For a point cloud P =
[ p T 1 , ..., p Tj , ..., p Tn ] T , where p j ∈ M 1 × 3 , and a point p ∈ M 1 × 3 , the
Chamfer Distance from p to P is:
[Equation]
For two point clouds P 1 = [ p T 1 , ..., p Tj , ..., p Tn ] T and P 2 , the Chamfer Distance from P 1 to P 2 is:
[Equation]
Besides, the Chamfer Distance between P 1 and P 2 is defined as:
[Equation]
The point cloud P 1 will be a part of the point cloud P 2 when the term cd P 1 → P 2 is minimized.
Moreover, the point clouds P 1 and P 2 will be identical when the term cd P 1 ↔ P 2 is minimized.
We use the Chamfer Distance between G ( P a ) and P a as the autoencoding loss.
Beside, the Chamfer
Distance from G ( P r ) to P r is utilized as the reconstruction loss, which makes the predicted shape be a part of the input and forces the generator to predict according to the input.
We also apply the
Chamfer Distance between R ( G ( P a )) and P a to optimize the refiner R .
In practice, we alternately optimize G with the optimization goal:
[Equation]
and D with optimization goal:
[Equation]
where α 1 ∼ 6 are the weights used for balancing the influences between each term.
Experiments 4.1 Datasets
We first generate the complete point clouds by sampling CAD models of ShapeNet [3] and ModelNet [35], serving as the artificial data.
We randomly rotate and scale them to simulate poses of real-scene data.
Then we follow a similar process as ScanObjectNN [28] to generate the input point clouds with surrounding points (background points) by cutting out the points in axis-aligned boxes whose origins are the center of objects in ScanNet [7].
The sizes of these axis-aligned boxes are s larger than the axis-aligned bounding boxes of objects, where we set s to 50% .
We also record labels of points which are 1 for object points and 0 for background points (noises) for evaluation.
We select four categories to establish four datasets and randomly split the data into train, valid and test sets.
The coordinates of all point clouds are normalized to [ − 1 , 1] .
Our UGAAN is trained on the train set, and evaluated on test set.
4.2 Evaluation Metrics
Generating completion ground truth manually for real-scene data is hard.
To overcome this, we utilize
Scan2CAD[2], a dataset containing the alignment information from models of ShapeNet to scenes of
[Table] [Table]
ScanNet, to replace the incomplete objects with complete models, and use [16, 23, 44] to remove the unseen points caused by occlusion for simulating the real scene data.
Then Chamfer Distance(CD) and F-Score@0.1%(F 0 .
1% score )[25] is employed to measure the difference between predicted complete point clouds and ground-truth models.
For the evaluation of segmentation, we employ IOU, following the previous work [22], to measure the difference between prediction and ground truth.
4.3 Implementation details
Our framework is implemented using PyTorch.
The point numbers n 0 ∼ 2 in Section 3 are set to 2048 , 512 and 2048 , and the weights α 1 ∼ 6 are set to 1 , 100 , 5 , 100 , 0 .
5 and 0 .
We train four categories separately with the Adam optimizer.
The learning rate of the optimizer and the batch size for our network are set to 1 .
× 10 − 5 and 4 , separately.
We train the model for 240 epochs.
Training our
UGAAN takes about 20 hours for convergence with a GTX 2080Ti GPU.
We will make our dataset and codes public in the future.
4.4 Comparison
We compare our UGAAN against existing representative methods on point cloud segmentation including unsupervised methods [22, 39, 45, 18] and supervised methods [43, 32, 24] and on unsupervised point cloud cloud completion [39] both quantitatively and qualitatively.
We simply modify [39] by replacing its reconstruction loss with ours and retrain it to fit our datasets, and then utilize our segmentation module to get the segmentation results for evaluation.
As shown in Table 1, we compare these methods on ScanNet and S3DIS with four categories, including the chair, table, sofa and bookshelf.
Our method is trained on the training set of ScanNet and tested on the test set of these two datasets.
The results are 6 .
16% and 4 .
51% higher than [39] and [22], showing the superiority of our method.
We also test our method trained with ScanNet with more categories on the test set of
ScanObjectNN, and the results are shown in the last second row of Table 3.
There are only seven common categories in these two datasets.
Thus, we can only conduct the comparison on these seven categories.
We also train our UGAAN on 15 categories of ScanObjectNN together with ShapeNet and ModelNet to evaluate the generalization ability across multiple categories.
The results are shown in the last row of Table 3.
The results are 8 .
73% and 4 .
46% higher than [22] on "Avg.
(part)" and "Avg.
(all)".
This verifies the generalization ability of our method with different categories.
For the quantitative comparison of completion, we build a test set using the method described in Section 4.
[Table]
on the categories containing chair, table, bookcase and sofa.
Table 2 shows that our UGAAN can predict more accurate complete point clouds.
Qualitative results on the three categories are shown in Figure 2.
The point clouds produced by our
UGAAN exhibit more accurate shapes and details, such as the legs and backrest of the chair and the legs of the table shown in Figure 2, leading to more accurate segmentation.
And the completion results of ours present more uniform spatial distributions, such as the sofa shown in Figure 2 which contains less noise, which could be attributed to our refiner.
Uniform point distribution would facilitate further applications including re-construction, simplification, and so on.
4.5 Ablation study
Ablation study on network architecture.
We conduct experiments on three ablated models to evaluate the necessity of important structures and reconstruction loss of our network.
The first model is the basic structure of GAN with a feature extractor and a shape generator.
The generator predicts a complete point cloud according to the input point cloud, and the discriminator distinguishes between the prediction and the artificial object.
The second model replaces the original generator with the autoencoding generator based on the first model.
The third model adds the reconstruction loss introduced in Section 3.3 to the second model.
The discriminator of the fourth model makes a distinction between the predictions with real-scene data and the artificial object as input separately.
And the discrimination of our full model also takes latent codes as the criterion.
As shown in Table 4, we train and test these ablated models with the ScanNet dataset on the chair category.
The evaluation shows the importance of these structures and reconstruction loss.
The comparison against the third and fourth models shows that there is a gap between the predictions and artificial point clouds, which explains the reason we apply discriminator to the predictions rather than the artificial point clouds directly.
[Table] [Table]
Refiner.
We also evaluate the importance of refiner both quantitatively and qualitatively.
As shown in Figure 3, the refiner can refine the prediction and make the prediction more uniform and reduce noises.
The mIOU of our methods without refiner and with refiner on the test sets of the S3DIS and
ScanObjNN are shown in Table 5.
These methods are trained with the train set of ScanNet.
The results show that even though the refiner takes no effect when the method is trained and evaluated on the same dataset, the refiner makes our method more robust when evaluated on new datasets.
The noise ratio s .
We also set the noise ratio s to 20% , 10% , 5% and 0% , and generate four more datasets with different noise ratios.
Then we repeat the experiments above on the chair category and the IOUs are shown in Figure 4.
The lower the noise ratio is, the easier the problem is, making the model perform better.
The results also show our generalization performance.
[Figure] [Figure]
I O U our method on chair category with different noises ratio.
[Figure]
The threshold d of intersection.
We also evaluate the influence of different threshold d of intersection on the IOU during the evaluation.
As Figure 5 shows, we apply different d to the categories chair and bag.
The results show that different object has different best d , leading to best IOU according to their volume in practice.
Since we normalize the coordinates of point clouds to [ − 1 , 1] , the point clouds of small objects such as bags contain more details in the same space than big objects such as chairs.
Extracted features of object points and background points.
We visualize the extracted features of object points and background points from the encoder for analysis.
As Figure 6(a) shows, every line indicates the feature of a point.
The lighter the pixel is, the larger the value of this channel is.
The figure shows that object points have larger values on some channels which can be passed to the latent code by max-pooling operation.
Figure 6(b) shows the visualization of the features of artificial points clouds, whose pattern is similar to Figure 6(a), since all the points of artificial data are object points.
Visualization of latent codes.
We use t-SNE [29] to visualize the latent code distribution of real-scene data and artificial data.
As shown in Figure 7 (c), the blue points represent the artificial data and red points indicate the real-scene data, and Figure 7 (e) shows the detailed point clouds of an area.
The real-scene data that contains the same shape as the artificial data are grouped together, which shows the robustness of our method with input contaminated by noises.
The point clouds having similar yaw angles are grouped together showing that our method could learn reasonable latent codes.
This could be attributed to the fact that our discriminator also takes latent codes as the criterion.
Visualization of the latent codes extracted from the method whose discriminator only takes predicted point cloud as the criterion is shown in Figure 7 (d).
It is obviously observed that these two kinds of data are separated clearly.
We also train our method with the dataset that mix up the real-scene data with different noise ratios.
As shown in Figure 7 (a) and (b), similar object with different noise ratios are grouped together, which shows the robustness of our method on real-scene data with different noise ratios.
[Figure]
Conclusion, Discussion, and Future Work
We have presented UGAAN, an end-to-end network specifically designed for unsupervised completion and segmentation of point clouds contaminated by noises or containing outliers.
Benefiting from our proposed autoencoding generator and latent-code-based discriminator, our method is able to generate complete point cloud accurately and segment the input real-scene point cloud in the meanwhile.
Extensive experiments show the superiority and generalization of our method on different categories of different datasets.
We also conduct experiments verifying the robustness of our method on noises.
Our method needs artificial datasets during the unsupervised training, which limits our method to apply to those categories without artificial correspondences.
The encoder and decoder of our generator and the overall structure of GAN can be improved for better prediction and the segmentation module can be replaced with a learning-based module, which are the future work of our method.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant numbers 62032011, 61972194 and Natural Science Foundation of Jiangsu Province award numbers
BK20211147.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[No] There is no theoretical results in this paper.
(b) Did you include complete proofs of all theoretical results?
[No] There is no theoretical results in this paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the pseudo code of our method in our supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See our supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] See the supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See the supplemental material.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] There is no crowdsourcing or conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] There is no crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No] There is no crowdsourcing or conducted research with human subjects.