L EARNING R EPRESENTATIONS AND G ENERATIVE M ODELS FOR 3D P OINT C LOUDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.
In this paper, we look at geometric data represented as point clouds.
We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.
The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.
We also perform a thorough study of different generative models including: GANs operating on the raw point clouds, signiﬁcantly improved GANs trained in the ﬁxed latent space our AEs and Gaussian mixture models (GMM).
Interestingly, GMMs trained in the latent space of our AEs produce samples of the best ﬁdelity and diversity.
To perform our quantitative evaluation of generative models, we propose simple measures of ﬁdelity and diversity based on optimally matching between sets point clouds
I NTRODUCTION
Three-dimensional (3D) representations of real-life objects are a core tool for vision, robotics, medicine and augmented / virtual reality applications.
Popular recent encodings include view-based projections , volumetric grids and graphs ; those complement more traditional shape representations such as 3D meshes, level set functions, curve-based CAD models and constructive solid geometry.
These encodings, while effective in their respective domains (e.g. acquisition or rendering), are often poor in semantics.
For example, naïvely interpolating between two different cars in a view-based representation does not yield a representation of an ’intermediate’ car.
Furthermore, these raw, high-dimensional representations are typically not well suited for the design of generative models via classic statistical methods.
As such, editing and designing new objects with such representations frequently involves the construction and manipulation of complex, object-speciﬁc parametric models linking the semantics to the representation.
This may require signiﬁcant expertise and effort.
Recent advances in deep learning bring the promise of a data-driven approach .
In domains where data is plentiful, deep learning tools have eliminated the need for hand-crafting features and models.
Deep learning architectures like autoencoders (AEs)
(Rumelhart et al., 1988; Kingma & Welling, 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014;
Radford et al., 2015; Denton et al., 2015; Che et al., 2016) are successful at learning complex data representations and generating realistic samples from complex underlying distributions.
Recently, deep learning architectures for view-based projections (Su et al., 2015; Wei et al., 2016; Kalogerakis et al., 2016), volumetric grids (Qi et al., 2016b; Wu et al., 2015; Hegde & Zadeh, 2016) and graphs (Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016;
Yi et al., 2016b) have appeared in the 3D machine learning literature.
In this paper we focus on point clouds: a relatively unexplored 3D modality.
Point clouds provide a homogeneous, expressive and compact representation of surface geometry, easily amenable to geometric operations.
These properties make them attractive from a learning point of view.
In addition, they come up as the output of common range-scanning acquisition pipelines used in devices like the Kinect and iPhone’s recent face identiﬁcation feature.
Only a handful of deep architectures for 3D point clouds exist in the literature: PointNet (Qi et al., 2016a; 2017) successfully tackled classiﬁcation and segmentation tasks;
Kalogerakis et al.
(2016) used point-clouds as an intermediate step in their pipeline.
We provide the ﬁrst results that use deep architectures to learn representations and generative models for point clouds.
Generative models have garnered increased attention recently in the deep learning community with the introduction of GANs (Goodfellow et al., 2014).
An issue with GAN-based generative pipelines is that training them is notoriously hard and unstable (Salimans et al., 2016).
More importantly, there is no universally accepted way to evaluate generative models .
In evaluating generative models one is interested in both ﬁdelity , i.e. how much generated points resemble the actual data, and coverage , i.e. what fraction of the data distribution a generated sample represents.
The latter is especially important given the tendency of certain GANs to exhibit mode collapse.
In particular, it assigns the same penalty to models that miss isolated examples interspersed all over, and models that miss whole contiguous pieces of the data manifold.
We provide simple methods to deal with both issues (training and evaluation) in our target domain.
Our speciﬁc contributions are:
• We design a new AE architecture –inspired by recent architectures used for classiﬁcation (Qi et al., 2016a)– that is capable of learning compact representations of point clouds with excellent reconstruction quality even on unseen samples.
The learned representations are (i) good for classiﬁcation via simple methods (SVM), improving on the state of the art (Wu et al., 2016); (ii) suitable for meaningful interpolations and semantic operations.
• We create the ﬁrst set of generative models which (i) can generate point clouds measurably similar to the training data and held-out test data; (ii) provide good coverage of the training and test dataset.
We argue that jointly learning the representation and training the GAN is unnecessary for our modality.
We propose a workﬂow that ﬁrst learns a representation by training an AE with a compact bottleneck layer, then trains a plain GAN in that ﬁxed latent representation.
We point to theory (Arjovsky & Bottou, 2017) that supports this idea, and verify it empirically.
Latent GANs are much easier to train than monolithic (raw) GANS and achieve superior reconstruction with much better coverage.
Somewhat surprisingly, GMMs trained in the latent space of ﬁxed AEs achieve the best performance across the board.
• To support our qualitative evaluation, we perform a careful study of various old and new metrics, in terms of their applicability (i) as objectives for learning good representations; (ii) for the evaluation of generated samples.
We propose a simple metric based on an optimal matching that can be used as a measure of coverage.
We ﬁnd that Chamfer distances fail to discriminate certain pathological cases from good examples.
• We show that training multi-class GANs works almost on par with dedicated GANs trained per-object-category, as long as the GANs are trained the latent space.
The rest of this paper is organized as follows: Section 2 outlines the necessary background and building blocks for our work and introduces our evaluation metrics.
Section 3 introduces our models for latent representations and generation of point clouds.
In Section 4, we evaluate all of our models both quantitatively and qualitatively, and analyze their behaviour.
Further results and evaluation can be found in the appendix.
The code for all our models will be made publicly available on github.
B ACKGROUND
[Equation]
Autoencoders.
Autoencoders (AE - inset) are deep architectures that aim to reproduce their input.
They are especially useful, when they contain a narrow bottleneck layer between input and output.
Upon successful training, the bottleneck layer corresponds to a lowdimensional representation, a code for the dataset.
The Encoder (E) learns to compress a data point x into its latent representation, z .
The Decoder (D) can then reproduce x from its encoded version z .
G x
DATA
D z x
Generative Adversarial Networks.
GANs are state-of-the-art generative models.
The basic architecture (inset) is based on a adversarial game between a generator (G) and a discriminator (D).
The generator aims to synthesize samples that look indistinguishable from real data (drawn from x ∼ p data ) by passing a randomly drawn sample z ∼ p z through the generator function G .
The discriminator tries to tell synthesized from real samples.
The most commonly used losses for the discriminator and generator networks are:
[Equation] [Equation]
where θ ( D ) , θ ( G ) are the parameters for the discriminator and the generator network respectively.
In addition to the classical GAN formulation, we also use the Wasserstein GAN (Gulrajani et al., 2017), which has shown improved stability during training.
Challenges speciﬁc to point cloud geometry.
Point clouds as an input modality present a unique set of challenges when building a network architecture.
As an example, the convolution operator – now ubiquitous in image-processing pipelines – requires the signal (in our case, geometry) to be deﬁned on top of an underlying grid-like structure.
Such a structure is not available in raw point clouds, which renders them signiﬁcantly more difﬁcult to encode than e.g. images or voxel grids.
Recent classiﬁcation work on point clouds (PointNet – Qi et al.
(2016a))
bypasses this issue by circumventing 2D convolutions.
Another issue with point clouds as a representation is that they are unordered - any permutation of a point set still describes the same shape.
This complicates comparisons between two point sets, typically needed to deﬁne a loss function.
This unorderedness of point clouds also creates the need for making the encoded feature permutation invariant.
Point-set distances.
Two permutation-invariant metrics for comparing unordered point sets have been proposed in the literature (Fan et al., 2016).
On the one hand, the Earth Mover’s distance (EMD) (Rubner et al., 2000) is the solution of a transportation problem which attempts to transform one set to the other.
For two equally sized subsets S 1 ⊆ R 3 , S 2 ⊆ R 3 , their EMD is deﬁned by
[Equation]
where φ is a bijection.
Interpreted as a loss, EMD is differentiable almost everywhere.
On the other hand, the (still differentiable but more computationally efﬁcient) Chamfer (pseudo)-distance (CD) measures the distance between each point in one set to its nearest neighbor in the other set, d CH ( S 1 , S 2 )
=
(cid:80) x ∈ S 1 min y ∈ S 2 (cid:107) x
− y (cid:107) 22 + (cid:80)
y ∈ S 2 min x ∈ S 1 (cid:107) x
− y (cid:107) 22 .
Evaluation Metrics for representations and generative models.
In the remainder of the paper, we will frequently be needing to compare a given set (distribution) of points clouds, whether reconstructed or synthesized, to its ground truth counterpart.
For example, one might want to assess the quality of a representation model, in terms of how well it matches the training set or a held-out test set.
Such a comparison might be done to evaluate the faithfulness and/or diversity of a generative model, and measure potential mode-collapse.
To measure how well a point-cloud distribution A matches a ground truth distribution G , we use the following metrics:
Coverage.
For each point-cloud in A we ﬁnd its closest neighbor in G ; closeness can be computed using either CD or EMD, thus yielding two different metrics, COV-CD and COV-EMD.
Coverage is measured as the fraction of the point-clouds in G that were matched to point-clouds in A .
A high coverage score typically indicates that most of G is roughly represented within A .
Minimum Matching Distance (MMD).
Coverage is not representative of the ﬁdelity of A with respect to G as matched elements need not be close.
To capture ﬁdelity, we ﬁnd the matching between G and
A with the lowest sum of distances (MMD).
Either of the structural distances can be used, yielding
MMD-CD and MMD-EMD.
We report the average of distances in the matching.
MMD measures the distances in the pairwise matchings, so it correlates with how realistic the elements of A are.
Jensen-Shannon Divergence (JSD).
The Jensen-Shannon divergence of the empirical distribution of individual points in the pointclouds.
We measure this via estimated random variables deﬁned in the ambient 3D space; this is possible because our training data are pre-aligned in 3D space, hence also the decoded and generated data.
Speciﬁcally, we consider a 28 3 regularly-spaced voxel grid centered in the sphere of diameter 1.
We construct a 28 3 random variable vector for A by counting the number of 3D points lying within each voxel, across all point clouds of A ; similarly for G .
We report the JSD of these two random vectors.
R EPRESENTATION AND G ENERATIVE M ODELS
In this section we describe the architectures of our representation and generative models for point clouds, starting from our autoencoder design.
Later, we introduce a GAN architecture tailored to point-cloud data, followed by a more efﬁcient pipeline that ﬁrst trains an AE and trains a much smaller GAN in the learned latent space, and a simpler generative model based on Gaussian Mixtures.
3. L EARNING REPRESENTATIONS OF 3D POINT CLOUDS
The input to our AE network is a point cloud with 2048 points ( 2048 × 3 matrix), representing a 3D shape.
The encoder architecture follows the principle of Qi et al.
(2016a): MLP-layers with increasing feature size, ending with a "symmetric" function.
This approach encodes every point independently and uses a permutation-invariant (symmetric) function to make a joint representation.
In our implementation we use 5 MLP layers, each followed by a ReLU and a batch-norm layer.
The output of the last MLP layer is passed to a feature-wise maximum to produce a k -dimensional vector which is the basis for our latent space.
The decoder transforms the latent vector with 3 fully connected layers, the ﬁrst two having ReLUs, to produce a 2048 × 3 output.
For a permutation invariant objective, we explore both the efﬁcient EMD-distance approximation (Fan et al., 2016) and the Chamfer-Distance as our structural losses; this yields two distinct AE models, referred to as
AE-EMD and AE-CD (detailed architecture parameters can be found in Appendix A).
To determine an appropriate size for the latent-space, we constructed AEs differing only by k ∈ 4 , 8 . . . ,and trained them with point-clouds of a single object class, under the two losses.
We repeated this procedure with pseudo-random weight initializations three times (see appendix, Fig. 12) and found that k = 128 had the best generalization error on the test data, while achieving minimal reconstruction error on the the train split.
3. G ENERATIVE M ODELS FOR P OINT C LOUDS
Raw point cloud GAN (r-GAN).
The ﬁrst version of our generative model operates directly on the raw 2048 × 3 point
set input – to the best of our knowledge this work is the ﬁrst to present a GAN for point clouds.
The architecture of the discriminator is identical to the AE (modulo the ﬁlter-sizes and the number of neurons), without any batch-norm and with leaky ReLUs Maas et al.
(2013) instead or
ReLUs.
The output of the last fully connected layer is fed into a sigmoid neuron.
The generator takes as input a 128-dimensional noise vector and maps it to a 2048 × 3 output by 5 FC-ReLU layers.
G z
D r z
E x
GAN
D x
Latent-space GAN (l-GAN).
In our l-GAN, instead of operating on the raw point cloud input, we pass the data through our pre-trained autoencoder, trained separately for each object class with the EMD (or Chamfer) loss function.
Both the generator and the discriminator of the GAN then operate on the 128dimensional bottleneck variable of the AE.
Finally, once the GAN training is over, the output of the generator is decoded to a point cloud via the AE decoder.
The architecture for the l-GAN is signiﬁcantly simpler than the one of the r-GAN.
We found that very shallow designs for both the generator and discriminator (in our case, one hidden FC layer for the generator and two FC for the discriminator) are sufﬁcient to produce realistic results.
Gaussian Mixture Model.
In addition to the l-GANs, we also train a family of Gaussian Mixture
Models (GMMs) on the latent spaces learned by our AEs.
We ﬁtted GMMs with varying numbers of Gaussian components, and experimented with both diagonal and full covariance matrices for the
Gaussians.
The GMMs can be turned into point-cloud generators by ﬁrst sampling the latent-space from the GMM distribution and accordingly using the AE’s decoder, similarly to the l-GANs.
E VALUATION AND R ESULTS
Our source for shapes is the ShapeNet repository (Chang et al., 2015); we pre-center all shapes into a sphere of diameter 1.
Unless otherwise stated, we train speciﬁc per-class models, and split the models in each class into training/testing/validation set using a 85%-5%-10%
4. E VALUATING THE LATENT REPRESENTATION
Classiﬁcation.
A common technique for evaluating the quality of unsupervised representation learning algorithms is to apply them as a feature extractor on supervised datasets and evaluate the performance of linear models ﬁtted on top of these features.
We use this technique to evaluate the performance of the latent “features” computed by our AE.
For this experiment to be meaningful, the
AE was trained across all different shape categories: we used 57,000 models from ShapeNet fromcategories of man-made objects.
Exclusively for this experiment, we used a bigger bottleneck of 512 , increased the number of neurons and applied batch-norm to the decoder as well.
To obtain features for an input 3D shape, we feed forward to the network its point-cloud and extract the 512-dimensional bottleneck layer vector.
This feature is then processed by a linear classiﬁcation SVM trained on the de-facto 3D classiﬁcation benchmark of ModelNet Wu et al.
Table 1 shows comparative results.
Note that previous state of the art Wu et al.
(2016) uses several layers of a GAN to derive a 7168 -long feature; our 512 -dimensional feature is more intuitive and parsimonious.
[Table]
with features derived in an unsupervised manner.
Comparing to [1] Kazhdan et al.
(2003), [2] Chen et al.
(2003), [3] Girdhar et al.
(2016a), [4] Sharma et al.
(2016), [5] Wu et al.
The decoupling of latent representation from generation allows ﬂexibly choosing the AE loss, which can improve the learned feature.
On ModelNet10, which includes primarily larger objects than
ModelNet40, an AE trained with the EMD loss performs slightly better than one trained with the CD.
On the other hand, when the variation within the collection increases, CD produces better results.
This is perhaps due to its more local and less smooth nature, which allows it to understand rough edges and some high frequency geometric details.
Finally, note that since our AEs were not trained on ModelNet, this experiment also demonstrates the domain-robustness of our learned features.
Qualitative Evaluation.
To visually assess the quality of the learned representation, we show some reconstruction results in Fig.
Here, we use our AEs to encode samples from the test split of the ground truth dataset (the leftmost of each pair of images) and then decode them and compare them visually to the input (the rightmost image).
These results shows the ability of our learned representation to generalize to unseen shapes.
In addition to reconstruction, our learned latent representation enables a number of interesting shape editing applications, including shape interpolations (Fig. 2), part editing and shape analogies.
More results are showcases in Appendix E.
Generalization Ability.
Our AEs are able to reconstruct unseen shapes; this is highlighted not only in the results of Figure 1, but also in quantitative measurements of the ﬁdelity and coverage of the reconstructed ground truth datasets (see appendix, Table 5) and by the comparable reconstruction quality on the training vs. test splits (appendix, Figure 12).
4. E VALUATING THE GENERATIVE MODELS
We train and compare a total of ﬁve generative models on the data distribution of point-clouds of the chair category.
We begin by establishing the two AEs with the 128-dimensional bottleneck, trained with the CD or EMD loss respectively – referred to as AE-CD and AE-EMD.
Both AEs were stopped at the epoch at which the average reconstruction error with respect to our validation dataset was minimized (see Appendix, Figure 12).
We train an l-GAN in each of the AE-CD and AE-EMD latent spaces.
In the space associated only with the AE-EMD we train a further two models: an identical (architecture-wise) l-GAN that utilizes the Wasserstein objective with gradient-penalty (Gulrajani et al., 2017), and a family of GMMs.
Lastly, we also train an r-GAN directly on the point cloud data.
Model Selection.
All GANs are trained for maximally 2000 epochs; for each GAN, we select one of its training epochs to obtain the “ﬁnal” model, based on how well the synthetic results match the ground-truth distribution.
Speciﬁcally, at a given epoch, we use the GAN to generate a set of synthetic point clouds, and measure the distance between this set and the validation set (Section 2).
We avoid measuring this distance using MMD-EMD, given the high computational cost of EMD.
Instead, we use either the JSD or MMD-CD metrics to compare the synthetic dataset to the validation dataset.
To further reduce the computational cost of model selection, we only check every 100 epochs (50 for r-GAN).
The epochs at which the various models were selected using the JSD criterion are shown in Table 3.
Using the same criterion, we also select the number and covariance type of Gaussian components for the GMM, and obtain the optimal value of 32 components.
GMMs performed much better with full (not diagonal) covariance matrices, suggesting strong correlations between the latent dimensions; see appendix (Fig. 14).
When using MMD-CD as the selection criterion, we obtain models of similar quality and at similar stopping epochs (appendix, Table 6); the optimal number of
Gaussians in this case was 40.
[Table] [Table]
Quantitative Evaluation.
Upon selection of the models, we compare them with respect to their capacity to generate synthetic samples.
In two different sets of experiments, we measure how well the distribution of the generated samples resembles both the train and test splits of the ground truth distribution, by using our models to generate a set of synthetic point clouds and using the metrics from Section 2 to compare the train or set respectively.
are reported in Table 2.
We also average classiﬁcation probability for those samples being recognized as a chair using the classiﬁer (Qi
2016a), is of the for is
Additionally, the achieved ﬁdelity and coverage are very close to the reconstruction baseline, namely, the lower bounds for the JSD and MMD achieved by the AE on which the generative models operate (appendix, Table 5).
For example, the AE-EMD achieved an MMD-EMD of 0.05 with respect to the ground truth training data , which is comparable with the MMD-EMD value of 0.06 achieved by the GMMs with respect to the test data.
Finally, by comparing Table 2 and Table 3 we can again establish the generalization ability of our models, since their performance for the training vs. testing splits is comparable.
This is highlighted in more detail in Fig.
13 in the appendix.
Note: The number of synthetic point clouds we generate for the train split experiment is equal to the size of the train dataset.
For the test split experiment, as well as for the validation split comparisons done for model selection, we generate synthetic datasets that are three times bigger than the ground truth dataset (the test resp.
validation set); this is possible due to the relatively small size of the test resp.
validation sets, and helps reduce sampling bias.
This is only necessary when measuring MMD or Coverage statistics.
Fidelity of metrics.
In Table 3 we note that the MMD-CD distance to the test set appears to be relatively small for the r-GANs.
This seeming advantage of the r-GANs is counter to what a qualitative inspection of the results yields.
We attribute this effect to the inadequacy of the chamfer distance to distinguish pathological cases such as the ones demonstrated in Fig.
In the ﬁgure, we show two triplets of images.
In each triplet, an r-GAN and an l-GAN is used to generate a synthetic set of point clouds; the left triplet shows an l-GAN on the AE-CD and the right an l-GAN on the
AE-EMD.
For a given ground truth point cloud from the test set (leftmost image of each triplet), we ﬁnd its nearest neighbor in each synthetic set under the chamfer distance - the middle image in each triplet shows the nearest neighbor in the synthetic results of the r-GAN and the right most image the nearest neighbor in the l-GAN set.
We report the distances between these nearest neighbors and the ground truth using both CD and EMD (in-image numbers).
Note that the CD values miss the fact that the r-GAN results are visibly of lesser quality.
The underlying reason appears to be that r-GANs tend to generate clouds with many points concentrated in the areas that are most likely to be occupied in the underlying shape class (e.g. the seat of chairs in the ﬁgure).
This implies that one of the two terms in the CD –namely, the one going from the synthetic point cloud to the ground truth– is likely to be very small for r-GAN results.
The “blindness” of the CD metric to only partial matches between shapes has the additional interesting side-effect that the CD-based coverage metric is consistently bigger than that reported by EMD, as noted in Table 3.
Instead, the EMD distance promotes a one-to-one mapping and thus correlates more strongly to visual quality; this means that it heavily penalizes the r-GAN result both in terms of MMD and coverage.
Training trends.
We performed extensive measurements during training of our models, to understand their behavior during training, as shown in Fig. 4.
On the left, we plot the JSD distance between the ground truth test set and good coverage of the test set no matter the metric used; which alludes to the well-established fact that end-to-end GANs are generally difﬁcult to train.
The l-GAN (AE-CD) performs better in terms of ﬁdelity with much fewer epochs as measured by JSD/MMD-EMD, but its coverage remains low.
We attribute this to the CD promoting unnatural topologies – cf.
Fig.
3 that visually shows this phenomenon.
Switching to an EMD-based AE for the representation and otherwise using the same latent GAN architecture (l-GAN, AE-EMD), yields a dramatic improvement in coverage and ﬁdelity.
Both l-GANs though suffer from the known issue of mode collapse: Half-way through training, ﬁrst coverage starts dropping with ﬁdelity still at good levels, which implies that they are overﬁtting a small subset of the data.
Later on, this is followed by a more comprehensive collapse, with coverage dropping as low as 0.5%.
Switching to a latent WGAN largely eliminates this collapse, as expected.
Comparisons to voxel-based methods.
To the best of our knowledge we are the ﬁrst to propose
GANs on point-cloud data.
To ﬁnd out against other 3D generative methods, in Table 2 and Fig.
4 we compare to a recent voxel-grid based approach (Wu et al., 2016) in terms of the JSD on the training set of the chair category - other shape categories can be found in the appendix (Table 7).
We convert their voxel grid output into a point-set with 2048 points by performing farthest-point-sampling on the isosurface of the grid values.
Per the authors’ suggestion, we used an isovalue parameter of 0 . 1 and isolated the largest connected component from the isosurface.
Since
Wu et al.
(2016) do not use any train/test split, we perform 5 rounds of sampling 1k synthetic results from their models, and report the best values of the computed evaluation metrics in Table 2.
The r-GAN mildly outperforms Wu et al.
in terms of its diversity (as measured by JSD/MMD), while also creating realistic-looking results, as shown by the classiﬁcation score.
The l-GANs perform even better, both in terms of classiﬁcation and diversity, with less training epochs.
Note also that the training time for one epoch of the l-GAN is more than an order of magnitude smaller than for the r-GAN, due to its much smaller architecture and dimensionality.
For fairness, we acknowledge that since Wu et al.
(2016) operates on voxel grids, it is not necessarily on equal standing when it comes to generating point clouds.
Qualitative evaluation.
In Fig.
5, we show some synthetic results produced by our l-GANs (top row) and the 32-component GMM, both trained on the AE-EMD latent space.
We notice high quality results from either model - this highlights the strength of our learned representation, which makes it possible for the simple GMM model to perform well.
The shapes (after decoding) corresponding to the 32 means of the Gaussian components can be found in the appendix (Fig. 15), as well as results using the r-GAN (Fig. 11).
The l-GAN produces crisper and less noisy results than the r-GAN, demonstrating an advantage of using a good structural loss on the decoupled, pre-trained AE.
Extensions to multiple classes We have performed preliminary experiments with an AE-EMD trained on a mixed set containing point clouds from 5 categories ( chair, airplane, car, table, sofa ).
The training and testing datasets for this AE were constructed by randomly picking and adding models from each class; 2K models per class for the training set, 200 models for testing and 100 for validation.
The multi-class AE has the same bottleneck size of 128 and was trained for 1000 epochs.
We compare against the class-speciﬁc AEs with the 85-5-10 train-val-test-split, which we trained for 500 epochs.
The precise AE model in all cases was selected based on the minimal reconstruction loss on the the respective validation set.
On top of all six AEs, we train six l-WGANs for 2K epochs, and evaluate their ﬁdelity/coverage using the MMD-CD between the respective testing sets and a synthesized dataset of 3x the size, as above.
It turns out that the l-WGANs based on the multi-class AE perform similarly to the dedicated class-speciﬁcally trained ones (Table 4).
A qualitative comparison (Fig. 6) also reveals that by using a multi-class AE-EMD we do not sacriﬁce much in terms of visual quality compared to the dedicated AEs.
[Table]
Limitations.
Fig.
7 shows some failure cases of our models.
Chairs with rare geometries (left two images) are sometimes not faithfully decoded.
Additionally, the AEs may miss high-frequency geometric details, e.g. a hole in the back of a chair (middle), thus altering the style of the input shape.
Finally, the r-GAN often struggles to create realistic-looking shapes (right) for some shape classes – while the r-GAN chairs that are easily visually recognizable, it has a harder time on cars.
Designing more robust raw-GANs for point clouds remain an interesting avenue for future work.
R ELATED WORK
A number of recent works (Wu et al.
(2016), Wang et al.
(2016), Girdhar et al.
(2016b), Brock et al.
(2016), Maimaitimin et al.
(2017), Zhu et al.
(2016)) have explored generative and discriminative representations for geometry.
They operate on different modalities, typically voxel grids or view-based image projections.
To the best of our knowledge, our work is the ﬁrst to study such representations for point clouds.
Training Gaussian mixture models (GMM) in the latent space of an autoencoder is closely related to VAEs (Kingma & Welling, 2013).
One documented issue with VAEs is over-regularization: the regularization term associated with the prior, is often so strong that reconstruction quality suffers (Bowman et al., 2015;
Sønderby et al., 2016; Kingma et al., 2016; Dilokthanakul et al., 2016)
The literature contains methods that start only with a reconstruction penalty and slowly increase the weight of the regularizer.
In our case, we ﬁnd that ﬁxing the AE before we train our generative models yields good results.
C ONCLUSION
We presented a novel set of architectures for 3D point-cloud representation learning and generation.
Our results show good generalization to unseen data and our representations encode meaningful semantics.
In particular our generative models are able to produce faithful samples and cover most of the ground truth distribution without memorizing a few examples.
Interestingly, we see that the best-performing generative model in our experiments is a GMM trained in the ﬁxed latent space of an AE.
While, this might not be a universal result, it suggests that simple classic tools should not be dismissed.
A thorough investigation on the conditions under which simple latent GMMs are as powerful as adversarially trained models would be of signiﬁcant interest
A AE D ETAILS
With the exception of the larger AE used for the cross-class classiﬁcation experiment of Section 4.1, the ﬁrst 4 MLP layers of our AE were implemented as 1D-convolutions with ReLUs and stride of 1, i.e. treating each point independently.
Their ﬁlter sizes were 64, 128, 128, 256 and k respectively, with k being the bottle-neck size.
The decoder had 3 FC-ReLU layers with 256, 256, 2048 × neurons each.
For training we used Adam with initial learning rate of 0 . 0005 , beta-1 of 0 .
9 and a batch size of 50 point-clouds.
Different AE setups (denoising/regularised) brought no noticeable advantage over our “vanilla” architecture.
Adding drop-out layers resulted in worse reconstructions and using batch-norm on the encoder only, sped up training and gave us slightly better generalization error when the AE was trained with single-class data.
B R -GAN D ETAILS
The discriminator’s ﬁrst 5 layers are MLPs with ﬁlter sizes of { 64 , 128 , 256 , 256 , 512 } each and leaky-ReLU between them.
They are followed by a feature-wise max-pool.
The last 2 FC-leakyReLU layers have { 128 , 64 } , neurons each leading to single sigmoid neuron.
We use a leak of 0 .units.
The generator consists of 5 FC-ReLU layers with { 64 , 128 , 512 , 1024 , 2048 × 3 } neurons each.
We trained r-GAN with ADAM with initial learning rate of 0 . 0001 , and beta-1 of 0 . 5 in batches of size 50 .
The noise vector was drawn by a spherical Gaussian of 128 dimensions with zero mean and 0 .units of standard deviation.
C L -GAN D ETAILS
The discriminator consists of 2 FC-ReLU layers with { 256 , 512 } neurons each and a ﬁnal FC layer with a single sigmoid neuron.
The generator consists of 2 FC-ReLUs with { 128 , 128 } neurons each.
When used the l-Wasserstein-GAN, we used a gradient penalty regularizer λ
= 10 .
The training parameters (learning rate, batch size) and the generator’s noise distribution were the same used for the r-GAN.
D AE RECONSTRUCTION QUALITY
Table 5 shows the reconstruction quality of the two AEs (CD- and EMD-based), in terms of the JSD of the reconstructed datasets (train or test, as noted in the table) with respect to their ground truth counterparts.
Note that, in general, the quality of reconstruction is comparable between the training and test datasets, indicating that the AEs are indeed able to generalize.
[Table]
E A PPLICATIONS OF THE LATENT SPACE REPRESENTATION
For shape editing applications, we use the embedding we learned with the AE-EMD trained across all 55 object classes, not separately per-category.
This showcases its ability to encode features for different shapes, and enables interesting applications involving different kinds of shapes.
Editing shape parts.
We use the shape annotations of Yi et al.
Yi et al.
(2016a)
as guidance to modify shapes.
As an example, assume that a given object category (e.g. chairs) can be further subdivided into two sub-categories A and B : every object A ∈ A possesses a certain structural property (e.g. has armrests, is four-legged, etc.) and objects B ∈ B do not.
Using our latent representation we can model this structural difference between the two sub-categories by the difference between their average latent representations x B − x A , where x A = (cid:80)
A ∈A x A , x B = (cid:80) B ∈B x B .
Then, given an object A ∈ A , we can change its property by transforming its latent representation: x A (cid:48)
=
x A + x B − x A , and decode x A (cid:48) to obtain A (cid:48) ∈ B .
This process is shown in Figure 8.
Note that the volume of chairs with armrests is on average 10% larger than the chairs without, which is reﬂected in the output of this process.
Interpolating shapes.
By linearly interpolating between the latent representations of two shapes and decoding the result we obtain intermediate variants between the two shapes.
This produces a “morph-like” sequence with the two shapes at its end points (Fig. 2, 9).
Our latent representation is powerful enough to support removing and merging shape parts, which enables morphing between shapes of signiﬁcantly different appearance.
Our cross-category latent representation enables morphing between shapes of different types, cfg.
the second row for an interpolation between a bench and a sofa.
More results are available in the additional material.
Shape Analogies.
Another demonstration of the euclidean nature of the latent space is demonstrated by ﬁnding “analogous” shapes by a combination of linear manipulations and euclidean nearestneighbor searching.
Concretely, we ﬁnd the difference vector between A and A (cid:48) , we add it to shape
B and search in the latent space for the nearest-neighbor of that result, which yields shape B (cid:48) .
We demonstrate the ﬁnding in Fig.
10 with images taken from the meshes used to derive the underlying point-clouds to help the visualization.
Finding shape analogies has been of interest recently in the geometry processing community Rustamov et al.
F F URTHER E VALUATION AND R ESULTS
A
A’
B
B’
[Figure]
R ec
[Figure]
t r u c ti
[Figure]
ss
[Figure] [Figure]
[Table]
[Figure]
[Table]