Unsupervised Point Cloud Completion and Segmentation by Generative Adversarial Autoencoding Network
Abstract
Most existing point cloud completion methods assume the input partial point cloudis clean, which is not the case in practice, and are generally based on supervisedlearning.
In this paper, we present an unsupervised generative adversarial autoencoding network, named UGAAN, which completes the partial point cloudcontaminated by surroundings from real scenes and cutouts the object simultaneously, only using artificial CAD models as assistance.
The generator of UGAANlearns to predict the complete point clouds on real data from both the discriminatorand the autoencoding process of artificial data.
The latent codes from generatorare also fed to discriminator which makes encoder only extract object featuresrather than noises.
We also devise a refiner for generating better complete cloudwith a segmentation module to separate the object from background.
We trainour UGAAN with one real scene dataset and evaluate it with the other two.
Extensive experiments and visualization demonstrate our superiority, generalizationand robustness.
Comparisons against the previous method show that our methodachieves the state-of-the-art performance on unsupervised point cloud completionand segmentation on real data.
Introduction
In recent years, point clouds have gained more popularity [11] as the standard outputs of 3D scanningdevices [17, 39] and the fundamental data structure to represent and process 3D data [7, 6, 19].
However, clean and complete point clouds of objects, on which the downstream applications suchas reconstruction [13] significantly rely, are hard to obtain in practice due to the nature of scannersand occlusions.
Point cloud completion which infers a complete object model given a partial pointcloud thus has received considerable attention.
Recent supervised methods [42, 41, 31, 26, 36, 34]show remarkable performance on recovering the original shapes from incomplete point clouds, butgenerally assume that the input point cloud is clean and does not contain any noises and outliers.
Suchan assumption does not always
In this paper, we propose an unsupervised end-to-end network for completing the partial point cloudwith noises from real scenes and outputting the object mask at the same time, dubbed UGAAN.
Previous supervised point cloud completion and segmentation require the paired clean and completepoint clouds, together with object-level point labels of real scenes, for training, which are hardand costly to obtain in practice [39, 22].
Most deep-learning-based methods heavily rely on suchtraining data, which limits their applications in practice.
Clearly different from previous methods,our UGAAN only needs unlabeled real-scene objects and unpaired artificial CAD models from forunsupervised training.
We employ the framework of GAN [10] which is widely used in unsupervised tasks.
Traditionalframework of GAN is hard to converge due to the gap between real-scene data and artificial data.
To resolve this problem, we utilize an autoencoding generator to learn the basic shapes of objectsfrom the artificial data for easier convergence.
Usually, the discriminator could easily distinguishthe artificial data as real data and the prediction as fake data, leading to bad performance due tothe data gap between predicted point clouds and artificial data.
Our discriminator, by contrast,accepts the prediction of artificial data rather than the artificial data itself as real data, which canreduce the negative effect of data gap.
Our discriminator also takes latent codes extracted fromreal-scene data and artificial data as the criterion, by which the generator can robustly learn a betterlatent space for prediction.
We employ a refiner to refine and upsample the first prediction result,and the final complete point clouds thus generated are more accurate and uniform.
Finally, thesegmentation module predicts the labels according to the predicted point cloud and input point cloud.
Compared with usual point cloud completion datasets [42] whose objects are aligned by the yawangles and scales, real-scene datasets such as ScanNet and S3DIS [1] contain objects that havedifferent yaw angles and varied scales, making the prediction of complete point clouds challenging.
To accommodate real-scene data, we randomly rotate and scale the artificial point clouds for easierlearning and convergence.
We train our UGAAN on ScanNet with ShapeNet as the artificial dataset without any pre-training orfine-tuning, and then directly evaluate it on the other two real-scene datasets, including S3DIS and
ScanObjectNN [28].
Comparisons with the state-of-the-art methods [22, 39] show the superiority ofour method.
The main contributions of our work are as follows.
• We propose, for the first time, an unsupervised end-to-end network for completion andsegmentation of point clouds with outliers by combining GAN and an autoencoding network.
• Our method completes the real-scene object with outliers from the background without the
need of any pre-training.
• We conduct extensive experiments for analysis and comparisons on three different real-scene
datasets, the results of which show the superiority and
generalization
ability of our methods.
The rest of this paper is organized as follows.
Section 2 reviews briefly the works on point cloudcompletion and segmentation.
Section 3 introduces our UGAAN in detail.
Dataset generation,experiments and comparisons are shown in Section 4 and Section 5.
Section 6 concludes the wholepaper and highlights future work.
Related Work2.1 Point cloud completion and segmentation.
In the early stage, existing successful works [27, 8] utilize voxels as representations of 3D models on which 3D convolutional neural networks can be immediately applied.
PMP-Net [34] and
PMP-Net++ [33] generate complete point clouds by moving input points to appropriate positionsiteratively with minimum moving distance.
SnowflakeNet [36] utilizes transformer and point-wisefeature deconvolutional modules to refine the first-stage point could multiple times.
For semanticsegmentation, Jiang et al.
[15] propose a multi-stage ordered convolution module to stack and encodethe information from eight spaces, achieving orientation encoding and scale awareness.
Liang et al.
[21] split nodes of a pre-trained, intermediate, semantic superpoint tree for proposals of instanceobjects.
Point Transformer [43] introduces transformer in the encoding process of point cloud to learnthe representation, which can be applied to different point cloud processing such as segmentation andclassification.
2.2 Unsupervised point cloud learning.
Xiao et al.
[37] summarize the unsupervised point cloud representation learning works using DNNs.
Some works [20, 38, 30] propose point cloud unsupervised pre-training approaches, Others proposedunsupervised model [40, 12, 4] achieves excellent results on classification, segmentation and upsampling tasks.
Jiang et al.
[14] introduce two contrastive losses to respectively facilitate downstreamclassification and segmentation.
Coseg [22] presents a point cloud object co-segmentation task,aiming to segment the common 3D objects in a set of point clouds, and fathom a method withco-contrastive losses to minimize feature discrepancy inside estimated object points and maximizefeature separation between the object and background points, which also leads to a weak generalization.
Chen et al.
[39] propose an unpaired point cloud completion method that can be trainedwithout requiring explicit correspondence between partial and complete point clouds by employingan adaptive transform network in GAN, however, the training process of its network is complicatedwith several pre-training models.
Method
3.1 Problem Formulation
Let P r ∈ M n 0 × 3 represent the point cloud from real-scene data with distribution p r and P a denoteartificial data with distribution p a , respectively.
P r contains points belonging to the object and frombackground (noises).
Obj ( P r ) represents the object points on P r .
Assuming that Obj ( P r ) ∼ p
o , thegoal of unsupervised point cloud completion and segmentation is to train a generator G satisfying thefollowing two conditions:
G ( P r ; θ g ) ∼ p a ,2.
G ( P r ; θ g ) ∩ P r ∼ p o ,
where G ( P r ; θ g represents the point cloud predicted by G with input P r and parameters θ g , and theoperation ∩ denotes the intersection of two point clouds 1 .
The two conditions are utilized to constraincompletion and segmentation processes of point clouds, respectively.
Condition 1 encourages thepredicted shapes to be similar to the shapes in the artificial dataset.
Condition 2 requires that thegenerator should predict shapes according to P r rather than generating random shapes, and thesegmentation result which is the intersection of prediction and P r should match the object points.
3.2 Network Architecture
We employ Generative Adversarial Network (GAN) [10] to solve this unsupervised learning problem.
As shown in Figure 1, our UGAAN consists of an autoencoding generator and a discriminator thatplays a minimax game for unsupervised completion and segmentation of real-scene object point cloud.
Furthermore, we also utilize a refiner to refine the predicted complete point cloud and a segmentationmodule for label prediction.
Autoencoding Generator. We use PointNet++ [24] as the encoder E of the generator to extract thelatent code z of input point cloud P i , and a multi-layer perceptron (MLP) as the decoder to predictthe point cloud P o ∈ M n 1 × 3 . The generator takes artificial data as input to learn basic shapes of pointclouds. Then, we employ a lightweight encoder-decoder refiner [42] R on P o to get the refined andupsampled output P o r ∈ M n 2 × 3 .
Different from [42], the upsampling module directly refines thecomplete point cloud P o . Given an input point cloud P i , we can obtain the latent code, the completeprediction and the refinement by z = E ( P i ; θ g ) , P o =
G ( P i ; θ g ) and P o r = R ( P o ; θ r ) , respectively,where θ g and θ r are the parameters of generate G and refiner R . fake / real ? real
Encoder
Encoder
Encoder
Decoder
MLP
Decoder
SegmentationModule
Latent code 𝑧𝑧
Global feature
Global feature
I npu t po i n t c l oud 𝑃𝑃 𝑖𝑖
C o m p l e t
e
po
i n t c l oud 𝑃𝑃 𝑜𝑜
R e f i n e d po i n t c l oud 𝑃𝑃 𝑜𝑜 𝑟𝑟
S e g m e n t a ti on l a b e l
C
AutoencodingGenerator
Discriminator
Refiner
UGAAN: Unsupervised Generative Adversarial Network
Generator
Discriminator 𝑃𝑃 𝑖𝑖 ∼𝑝𝑝 𝑎𝑎 𝑃𝑃 𝑖𝑖 ∼𝑝𝑝 𝑟𝑟
Discriminator.
We take advantage of the encoder proposed in [42] to extract the global features f of
P o .
Then the global feature f and the latent code z are combined and fed into a MLP for predictingthe probability P :
[Equation]
where P i ∼ p a and θ d is the parameters of the discriminator.
To reduce the negative effect of theshape gap between the prediction and artificial data, we feed the predicted shapes rather than theartificial data into the discriminator.
The discriminator also takes latent codes as the criterion bywhich the generator can learn better latent codes for prediction.
Segmentation Module.
Finally, the segmentation results (i.e. the object points of p r ) are obtainedunder the guidance of the complete prediction P o .
Here, we directly calculate the intersection between
P o and P r to get the object points, rather than predicting with another network.
Due to the discretenature of point clouds, the strict intersection between P o and P r is almost empty.
As a consequence,we make use of general intersection between the point clouds P 1 and P 2 which is indicated by theoverlapping area:
[Equation]
where cd p → P 1 will be introduced in Equation 5 with d being the threshold.
The predicted segmentation label of p ∈ P r conditioned on P o with threshold d can be computed as:
[Equation]
Besides, the segmentation label of P r =
[ p T 1 , ..., p Tj , ..., p Tn ] T can be obtained by S d ( P r ; P o )
=[ S d ( p 1 ; P r , P O ) , ..., S d ( p j ; P r , P O ) , ..., S d ( p n ; P r , P O )] T ∈ M n × 1 .
3.3 Optimization
We modify the optimization strategy introduced by GAN [10] to train our UGAAN on the unsupervised point cloud completion and segmentation problem.
Specifically, the optimization problem ofthe proposed UGAAN is formulated as:
[Equation]
Due to the data gap between artificial data and real-scene data [39] and the limited fitting abilityof the decoder, the terms G ( P r ) and P a often exhibit huge differences as shown in the top leftpart of Figure1.
It makes D easier to optimize than G which makes network hard to convergeand will result in poor completion performance.
Thus, different from the original GAN where the discriminator directly works on G ( P r ) and P a , the discriminator of UGAAN predicts accordingto G ( P r ) and G ( P a ) to reduce the negative effects caused by these gaps for easier convergence.
Besides, if we impose no other constraints on problem 4, the generator may just generate randompoints after the optimization process is complete.
Thus, we also impose two losses on problem 4 torestrict the prediction of the generator, namely the autoencoding loss between G ( P a ) and P a , and thereconstruction loss between G ( P r ) and P r .
We employ the well-known Chamfer Distance(CD) [9] to measure the similarity between pointclouds.
For a point cloud P =
[ p T 1 , ..., p Tj , ..., p Tn ] T , where p j ∈ M 1 × 3 , and a point p ∈ M 1 × 3 , the
Chamfer Distance from p to P is:
[Equation]
For two point clouds P 1 = [ p T 1 , ..., p Tj , ..., p Tn ] T and P 2 , the Chamfer Distance from P 1 to P 2 is:
[Equation]
Besides, the Chamfer Distance between P 1 and P 2 is defined as:
[Equation]
The point cloud P 1 will be a part of the point cloud P 2 when the term cd P 1 → P 2 is minimized.
Moreover, the point clouds P 1 and P 2 will be identical when the term cd P 1 ↔ P 2 is minimized.
We use the Chamfer Distance between G ( P a ) and P a as the autoencoding loss.
Beside, the Chamfer
Distance from G ( P r ) to P r is utilized as the reconstruction loss, which makes the predicted shapebe a part of the input and forces the generator to predict according to the input.
We also apply the
Chamfer Distance between R ( G ( P a )) and P a to optimize the refiner R .
In practice, we alternately optimize G with the optimization goal:
[Equation]
and D with optimization goal:
[Equation]
where α 1 ∼ 6 are the weights used for balancing the influences between each term.
Experiments
4.1 Datasets
We first generate the complete point clouds by sampling CAD models of ShapeNet [3] and ModelNet [35], serving as the artificial data.
We randomly rotate and scale them to simulate poses ofreal-scene data.
Then we follow a similar process as ScanObjectNN [28] to generate the input pointclouds with surrounding points (background points) by cutting out the points in axis-aligned boxeswhose origins are the center of objects in ScanNet [7].
The sizes of these axis-aligned boxes are slarger than the axis-aligned bounding boxes of objects, where we set s to 50% .
We also record labelsof points which are 1 for object points and 0 for background points (noises) for evaluation.
We selectfour categories to establish four datasets and randomly split the data into train, valid and test sets.
The coordinates of all point clouds are normalized to [ − 1 , 1] .
Our UGAAN is trained on the train set,and evaluated on test set.
Metrics
Generating completion ground truth manually for real-scene data is hard. To overcome this, we utilize Scan2CAD[2], a dataset containing the alignment information from models of ShapeNet to scenes of [Table]
[Table] ScanNet, to replace the incomplete objects with complete models, and use [16, 23, 44] to remove theunseen points caused by occlusion for simulating the real scene data. Then Chamfer Distance(CD)and F-Score@0.1%(F 0 .
1% score )[25] is employed to measure the difference between predicted completepoint clouds and ground-truth models. For the evaluation of segmentation, we employ IOU, followingthe previous work [22], to measure the difference between prediction and ground truth.
4.3 Implementation details
Our framework is implemented using PyTorch.
The point numbers n 0 ∼ 2 in Section 3 are set to 2048 ,512 and 2048 , and the weights α 1 ∼ 6 are set to 1 , 100 , 5 , 100 , 0 .
5 and 0 .
We train four categoriesseparately with the Adam optimizer.
The learning rate of the optimizer and the batch size for ournetwork are set to 1 .
× 10 − 5 and 4 , separately.
We train the model for 240 epochs.
Training our UGAAN takes about 20 hours for convergence with a GTX 2080Ti GPU.
We will make our datasetand codes public in the future.
4.4 Comparison
We compare our UGAAN against existing representative methods on point cloud segmentation including unsupervised methods [22, 39, 45, 18] and supervised methods [43, 32, 24] and on unsupervisedpoint cloud cloud completion [39] both quantitatively and qualitatively.
We simply modify [39]by replacing its reconstruction loss with ours and retrain it to fit our datasets, and then utilize oursegmentation module to get the segmentation results for evaluation.
As shown in Table 1, we comparethese methods on ScanNet and S3DIS with four categories, including the chair, table, sofa andbookshelf.
Our method is trained on the training set of ScanNet and tested on the test set of thesetwo datasets.
The results are 6 .
16% and 4 .
51% higher than [39] and [22], showing the superiority ofour method.
We also test our method trained with ScanNet with more categories on the test set of ScanObjectNN, and the results are shown in the last second row of Table 3.
There are only sevencommon categories in these two datasets.
Thus, we can only conduct the comparison on these sevencategories.
We also train our UGAAN on 15 categories of ScanObjectNN together with ShapeNetand ModelNet to evaluate the generalization ability across multiple categories.
The results are shownin the last row of Table 3.
The results are 8 .
73% and 4 .
46% higher than [22] on "Avg.
(part)" and"Avg.
(all)".
This verifies the generalization ability of our method with different categories.
For the
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
quantitative comparison of completion, we build a test set using the method described in Section 4.2on the categories containing chair, table, bookcase and sofa.
Table 2 shows that our UGAAN canpredict more accurate complete point clouds.
Qualitative results on the three categories are shown in Figure 2.
The point clouds produced by our
UGAAN exhibit more accurate shapes and details, such as the legs and backrest of the chair and thelegs of the table shown in Figure 2, leading to more accurate segmentation.
And the completion resultsof ours present more uniform spatial distributions, such as the sofa shown in Figure 2 which containsless noise, which could be attributed to our refiner.
Uniform point distribution would facilitate furtherapplications including re-construction, simplification, and so on.
4.5 Ablation study
Ablation study on network architecture.
We conduct experiments on three ablated models toevaluate the necessity of important structures and reconstruction loss of our network.
The firstmodel is the basic structure of GAN with a feature extractor and a shape generator.
The generatorpredicts a complete point cloud according to the input point cloud, and the discriminator distinguishesbetween the prediction and the artificial object.
The second model replaces the original generatorwith the autoencoding generator based on the first model.
The third model adds the reconstructionloss introduced in Section 3.3 to the second model.
The discriminator of the fourth model makes a distinction between the predictions with real-scene data and the artificial object as input separately.
And the discrimination of our full model also takes latent codes as the criterion.
As shown in Table 4,we train and test these ablated models with the ScanNet dataset on the chair category.
The evaluationshows the importance of these structures and reconstruction loss.
The comparison against the thirdand fourth models shows that there is a gap between the predictions and artificial point clouds, whichexplains the reason we apply discriminator to the predictions rather than the artificial point cloudsdirectly.
[Table] [Table]
Refiner.
We also evaluate the importance of refiner both quantitatively and qualitatively.
As shownin Figure 3, the refiner can refine the prediction and make the prediction more uniform and reducenoises.
The mIOU of our methods without refiner and with refiner on the test sets of the S3DIS and
ScanObjNN are shown in Table 5.
These methods are trained with the train set of ScanNet.
Theresults show that even though the refiner takes no effect when the method is trained and evaluated onthe same dataset, the refiner makes our method more robust when evaluated on new datasets.
The noise ratio s .
We also set the noise ratio s to 20% , 10% , 5% and 0% , and generate four moredatasets with different noise ratios.
Then we repeat the experiments above on the chair category andthe IOUs are shown in Figure 4.
The lower the noise ratio is, the easier the problem is, making themodel perform better.
The results also show our generalization performance.
[Figure]
[Figure]
The threshold d of intersection.
We also evaluate the influence of different threshold d of intersectionon the IOU during the evaluation.
As Figure 5 shows [Figure] we apply different d to the categories chair andbag.
The results show that different object has different best d , leading to best IOU according to theirvolume in practice.
Since we normalize the coordinates of point clouds to [ − 1 , 1] , the point clouds ofsmall objects such as bags contain more details in the same space than big objects such as chairs.
Extracted features of object points and background points.
We visualize the extracted features ofobject points and background points from the encoder for analysis.
As Figure 6(a) shows, every line indicates the feature of a point.
The lighter the pixel is, the larger the value of this channel is.
Thefigure shows that object points have larger values on some channels which can be passed to the latentcode by max-pooling operation.
Figure 6(b) shows the visualization of the features of artificial pointsclouds, whose pattern is similar to Figure 6(a), since all the points of artificial data are object points.
Visualization of latentcodes.
We use t-SNE [29]to visualize the latent codedistribution of real-scenedata and artificial data.
Asshown in Figure 7 (c), theblue points represent theartificial data and red pointsindicate the real-scene data,and Figure 7 (e) showsthe detailed point cloudsof an area.
The real-scenedata that contains the sameshape as the artificial dataare grouped together, whichshows the robustness of our method with input contaminated by noises.
The point clouds havingsimilar yaw angles are grouped together showing that our method could learn reasonable latent codes.
This could be attributed to the fact that our discriminator also takes latent codes as the criterion.
Visualization of the latent codes extracted from the method whose discriminator only takes predictedpoint cloud as the criterion is shown in Figure 7 (d).
It is obviously observed that these two kinds ofdata are separated clearly.
We also train our method with the dataset that mix up the real-scene datawith different noise ratios.
As shown in Figure 7 (a) and (b), similar object with different noise ratiosare grouped together, which shows the robustness of our method on real-scene data with differentnoise ratios.
[Figure]
[Equation]
Conclusion, Discussion, and Future Work
We have presented UGAAN, an end-to-end network specifically designed for unsupervised completionand segmentation of point clouds contaminated by noises or containing outliers.
Benefiting from ourproposed autoencoding generator and latent-code-based discriminator, our method is able to generatecomplete point cloud accurately and segment the input real-scene point cloud in the meanwhile.
Extensive experiments show the superiority and generalization of our method on different categoriesof different datasets.
We also conduct experiments verifying the robustness of our method on noises.
Our method needs artificial datasets during the unsupervised training, which limits our method toapply to those categories without artificial correspondences.
The encoder and decoder of our generatorand the overall structure of GAN can be improved for better prediction and the segmentation modulecan be replaced with a learning-based module, which are the future work of our method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[No] There is notheoretical results in this paper.
(b) Did you include complete proofs of all theoretical results?
[No] There is no theoreticalresults in this paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
pseudo code of our method in our supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.3.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See our supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.3.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See the supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See the supplemental material.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots,
if applicable?
[No] There is no crowdsourcing or conducted research with human
subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No] There is no crowdsourcing or conducted
research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] There is no crowdsourcing or conducted
research with human subjects.