U NSUPERVISED G EOMETRIC D ISENTANGLEMENT FOR S URFACES VIA CFAN-VAE
A BSTRACT
Geometric disentanglement, the separation of latent codes for intrinsic (i e. identity) and extrinsic (i e. pose) geometry, is a prominent task for generative models of non-Euclidean data such as 3D deformable models.
It provides greater interpretability of the latent space and leads to more control in generation.
This work introduces a mesh feature, the conformal factor and normal feature (CFAN) for use in mesh convolutional autoencoders.
We further propose CFAN-VAE, a novel architecture that disentangles identity and pose using the CFAN feature.
Requiring no label information on the identity or pose during training, CFAN-VAE achieves geometric disentanglement in an unsupervised way.
Our comprehensive experiments, including reconstruction, interpolation, generation and identity/pose transfer, demonstrate CFAN-VAE achieves state-of-the-art performance on unsupervised geometric disentanglement
I NTRODUCTION
With the recent popularity of geometric deep learning (Bronstein et al., 2017), mesh based convolutional autoencoders (MeshVAEs) are now a popular tool for surface generation in the surface processing community (Cheng et al., 2019; Litany et al., 2018;
Ma et al., 2019; Ranjan et al., 2018).
With these VAEs achieving state-of-the-art performance on tasks such as reconstruction, more attention is being given towards tasks such as latent space interpretability.
Geometric disentanglement, where the latent variables controlling intrinsic (properties independent of surface embedding) and extrinsic (properties dependent on surface embedding) geometry are separated (AumentadoArmstrong et al., 2019), is an important open problem related to such interpretability.
A typical application of a disentangled latent space is the separation of identity and pose in the case of human body generation (Jiang et al., 2019a; Tan et al., 2018).
This is useful in ﬁelds such as graphics where we may be interested in the procedural generation of certain assets.
We are interested in creating an architecture that explicitly leads to geometric disentanglement in an unsupervised setting.
Namely, we do not require labels for mesh identity and pose, allowing the architecture to be applied more broadly.
In this work, we:
1. introduce a feature, the conformal factor and normal feature (CFAN), that decouples intrinsic and extrinsic geometry for use in mesh convolutional autoencoders.
propose a novel architecture, CFAN-VAE, for unsupervised geometric disentanglement.
For a given mesh, we compute the CFAN feature, and encode its components separately into latent vectors representing intrinsic and extrinsic geometry.
Related Work Geometric disentanglement is an important feature leading to a more interpretable latent space and thus more control in generation.
For instance, (Jiang et al., 2019a;b; Tan et al., 2018) learn a latent space separating identity and pose, advocating for the use of the as-consistentas-possible deformation representation (ACAPDR).
These works require a reference pose for all identities during training.
Recently, several works proposed networks using non-Euclidean convolution for this task (Cosmo et al., 2020; Levinson et al., 2019; Zhou et al., 2020).
These networks either utilize supervision or loss functions that require identiﬁcation of isometric pairs in training data.
In practice, the latter needs supervised identity to scale on large datasets.
Training datasets lacking isometries may be desireable in applications such as differential privacy (Abadi et al., 2016).
There is limited work to achieve unsupervised geometric disentanglement that completely eliminates reliance on both identity and pose labeling.
To our knowledge, the only previous work concerning unsupervised disentanglement for 3D data is (Aumentado-Armstrong et al., 2019), in which the authors introduce GDVAE, a VAE built on a PointNet (Charles et al., 2017).
This network requires the computation of Laplace-Beltrami eigenvalues on each surface of the training set.
We compare our results to this work as it is the state-of-the-art in unsupervised geometric disentanglement.
Later, we will ﬁnd advantages of our approach include the use of a signal, unlike Laplace-Beltrami eigenvalues, that is not globally sensitive to local deformation.
Additionally, as our model is not trained on
PointNet embeddings, we are able to use signiﬁcantly less parameters (on the order of millions), for our architecture.
CFAN-VAE
Our novel method for geometric disentanglement is motivated by the Fundamental Theorem of Surfaces (Do Carmo, 2016).
We describe surfaces using conformal factors and surface normal vectors.
This CFAN feature is easily leveraged by our introduced CFAN-VAE for geometric disentanglement.
CFAN Feature
The Fundamental Theorem of Surfaces states that a surface can be uniquely reconstructed up to rigid motion given feasible metric tensor and surface normals (Do Carmo, 2016).
Then we can use metric tensors and normal vector ﬁelds to characterize surfaces.
In this work, we consider genus zero surfaces.
It is well known that all genus zero surfaces are conformally equivalent (Jost & Jost, 2008).
Namely, given two genus-zero surfaces, ( M 1 , g 1 ) and ( M 2 , g 2 ) , there exists a diffeomorphism, φ : ( M 1 , g 1 ) →
( M 2 , g 2 ) where φ ∗ ( g 2 ) = exp( λ ) g 1 .
Here the function, λ , is known as the conformal factor and deﬁnes a conformal deformation from
M 1 to M 2 .
Then any pair of conformally equivalent surfaces can be deformed into one another (up to isomorphism) by choosing the correct conformal factor.
The surface embedding is deﬁned up to translation by the normal ﬁeld.
These two components deﬁne the CFAN feature, with the conformal factor and normal ﬁeld representing intrinsic and extrinsic geometry respectively.
Given a triangle mesh ( P , T ) , where P ∈ R n × 3 is the set of vertices and T ∈ R k × 3 is the corresponding set of faces, we deﬁne the discretized CFAN feature.
Each face τ in T has a corresponding exterior face normal n τ .
We compute the weighted average of these face normals around the ﬁrst ring structure of each vertex to deﬁne a pointwise normal.
In the continuous setting, the local area on the surface is given by √ det g 2 , so it follows that the logarithm of local area is an afﬁne function of the conformal factor, λ + 12 log det g 1 . We have equivalence if g 1 is taken to be the trivial metric. With this in mind, we approximate a discrete conformal factor by taking the logarithm of a third of the area of the ﬁrst ring structure about each point.
Formally we deﬁne the CFAN feature, ( c i , n i ) as:
[Equation]
Network Architecture Based on the CFAN feature, we propose a simple architecture, CFANVAE, as shown in Figure 2 to achieve unsupervised geometric disentanglement.
The intuition is to encode the conformal factor and the normal features separately.
The 3D coordinates p of an input mesh under a ﬁxed triangulation are ﬁrst converted to the CFAN feature by transformations φ c and φ n provided by equation 1, yielding c = φ c ( p ) and n = φ n ( p ) .
Then, the feature components are separately encoded by E c and E n to create two different latent variables, the conformal latent variable z c and the normal latent variable z n in the disentangled latent space Z c,n =
Z c × Z n .
That is, z c = E c ( c ) , z n = E n ( n ) , and ˆ p =
D ( z c , z n ) .
As a result, z c corresponds to intrinsic geometry, controlling surface identity, and z n corresponds to extrinsic geometry, controlling surface pose.
After that, the CFAN latent variable z c,n = [ z c , z n ] ∈
Z c,n is decoded by D to obtain the reconstructed 3D coordinates ˆ p ,
Our model is built on geometric convolutions given by PTC layers (Schonsheck et al., 2018).
The mesh signal is decimated during encoding and reﬁned during decoding by a factor of four.
Sampling is precomputed by the geodesic Farthest Point Sampling (FPS) method (Moenning & Dodgson, 2003).
After the ﬁnal mesh convolution in the encoder, dense layers map the signal to the variational statistics, µ :=
[ µ c , µ n ] and σ :=
[ σ c , σ n ] .
A dense layer also maps z c,n to the ﬁrst mesh signal in the decoder.
Details on the loss function used for training can be found in Section A.1.
E XPERIMENTS AND R ESULTS
We consider three datasets in our experiments.
This includes DFAUST, a real-world motion-capture dataset of 10 people performing several poses (Bogo et al., 2017).
We randomly split the dataset into a training/validation/test set of 35,720/500/5,000 meshes respectively.
In addition, we generated a synthetic dataset of animals referred to as SMAL following (Zufﬁ et al., 2017), based on the SMPL model introduced in (Loper et al., 2015).
Similary, we formed a synthetic dataset of humans referred to as SURREAL from (Varol et al., 2017).
These datasets are created following (Aumentado-Armstrong et al., 2019).
For SURREAL, we add small normal noise with σ = 0 .
2 to the intrinsic shape parameters β to eliminate any isometry in the dataset.
These two datasets provide challenging environments where identity-supervised methods fail.
Details on the network and training hyperparameters are provided in Section A.2.
For comparisons to GDVAE, we use the available dataset-speciﬁc pretrained models available in (AumentadoArmstrong et al., 2019).
We verify that CFAN-VAE is able to accurately reconstruct surfaces.
Figure 3 displays high quality reconstructions for each dataset using CFAN-VAEs for DFAUST,
SURREAL, and SMAL.
Surface Interpolation Figure 1 displays examples of disentangled interpolations for each dataset using CFAN-VAE.
In this ﬁgure, the vertical and horizontal axis represents conformal and normal interpolations respectively.
The color map denotes the pointwise normalized conformal factor of the reconstructions.
These results verify that CFAN-VAE produces meaningful geometric disentanglement between intrinsic and extrinsic information.
Thus, it enjoys ﬂexibility to generate high quality meshes preserving either identity or pose.
Notice, the colormaps and position are almost identical for ﬁxed z c and z n respectively.
We include additional examples in Figure 5 in the appendix.
We compare the performance of CFAN-VAE to GDVAE on transfer tasks.
For SURREAL and
SMAL, we can create ground-truth mesh solutions to the transfer task by exchanging the appropriate [Table]
SMPL parameters of the source meshes.
Then we compare the Chamfer distance of the network generated mesh, with appropriate latent code exchanged, to the SMPL generated ground truth.
These results are summarized in Table 1.
It is clear that CFAN-VAE outperforms GDVAE, particularly on the SMAL dataset.
This nice property of the CFAN-VAE latent space enables ﬂexible control of the geometry of generated surfaces.
We stress that these two datasets lack isometric pairs, so other methods (Cosmo et al., 2020; Zhou et al., 2020) are not of use for comparison.
Further analysis of the learned latent space is available in Section A.3.
Surface Generation Figure 4 plots random generations where latent codes are sampled from a multivariate Gaussian.
This Gaussian, N ( µ , Σ ) , is a product of marginals ﬁt to the embedding of the test set for each latent variable.
We sample from N ( µ , 0 . 8 Σ ) .
Here the normal latent code and conformal code is ﬁxed along the vertical axis and horizontal axis respectively.
Clearly, pose and identity are successfully ﬁxed.
The generated meshes are of comparable visual quality to the reconstructions from the test set.
Then the latent space of CFAE-VAE allows for high-quality generation while providing interpretability of geometric features.
C ONCLUSION
We propose a novel architecture, CFAN-VAE, for unsupervised geometric disentanglement in mesh convolutional autoencoders.
This is accomplished by utilizing the conformal factor and normal (CFAN) feature to separate intrinsic and extrinsic geometry.
There is clearly strong geometric disentanglement in CFAN-VAEs.
We achieve state-of-the-art performance on transfer tasks and disentanglement compared to prior work.
The continued integration of geometric theory into neural network architectures can only lead to more interesting results in the future A CKNOWLEDGEMENTS
J. Tatro and R. Lai’s work is supported in part by NSF CAREER Award (DMS—1752934).
J. Tatro’s work is also supported in part by IBM-RPI AIRC program.
Part of this research was performed while the authors were visiting the Institute for Pure and Applied Mathematics (IPAM), which is supported by the NSF DMS-1440415, during the Geometry and Learning from Data in 3D and Beyond long program.
A A PPENDIX A. L OSS F UNCTION
The training loss for our model is given by:
[Equation]
The ﬁrst term is simply the L 1 error of the reconstruction which promotes robustness to vertex outliers.
L D and L M are disentanglement and metric penalties that we deﬁne shortly.
The fourth term of the loss function is the KL-divergence of the latent representation from the unit normal distribution, given by the Bayesian prior assumption (Kingma & Welling, 2013).
In CFAN-VAE, geometric disentanglement is a result of separately encoding intrinsic and extrinsic geometric information.
We stress that structural conditions prevent guaranteeing complete independence of the separate latent vectors.
This is intuitive, considering motion can be restricted by intrinsic properties such as body mass distribution.
To promote independence, we consider a pair of encoded meshes.
We create a latent variable with the conformal latent code of the ﬁrst mesh, while the normal code is a random convex combination of the two normal codes.
We decode and re-encode this new latent variable.
We then have an Lpenalty on the resulting change in the conformal code upon re-encoding as it should not be affected by this change in the normal code.
We penalize a change in the normal code analogously.
This disentanglement penalty, L D , is formalized below,
[Equation]
Inspired by the metric regularization introduced in (Cosmo et al., 2020), we add a metric penalty,
L M , to the loss function,
[Equation]
where l α is the α convex combination of its entries.
This promotes smoothness in the metric of reconstructions.
A. N ETWORK H YPERPARAMETERS
Each VAE contains 5 layer encoders and decoder.
For the PTC kernels, we use a 13 point stencil with the support of the kernel about a point being its 9 nearest geodesic neighbors.
The size of the conformal/normal latent vectors are (32, 32) for DFAUST and (8, 32) for SMAL.
For visualization, we use models with a latent dimension of (32, 32).
For corresponding xyz-VAEs, the latent dimension is the sum of the two latent dimensions.
The latent dimension of the GDVAE models are (5 , 15) and (5 , 11) for SURREAL and SMAL respectively.
When comparing to CFAN-VAE models for SURREAL, we choose a corresponding latent dimension of (5 , 15) .
We use batch normalization (BN) layers in the encoder.
When training with the disentanglement penalty L D , we freeze the BN layers during reencoding.
The activation functions are ReLU and
ELU for the encoders and decoder.
We use the AdamW optimizer (Loshchilov & Hutter, 2017) with a learning rate of 1E-3 and a weight decay of 5E-5.
The value of λ KL is set at 1E-4, while λ D and λ M are 5E-2.
Each model is trained for 300 epochs with a batch size of 32.
We train two instances of each network using different random seeds.
A. L ATENT S PACE A NALYSIS
To better assess the geometric disentanglement in our networks, we analyze the latent spaces of the
CFAN-VAEs.
We quantitatively measure the disentanglement in latent space in the manner of (Zhou et al., 2020).
First, we embed the test sets for SURREAL and SMAL in the latent space.
Given a mesh embedding, we ﬁnd its nearest neighbor in either the intrinsic or extrinsic latent spaces, Z c or Z n .
As these meshes were created via SMPL shape and pose parameters, β and θ , we measure the distance between these parameters of the embedded mesh and its neighbor.
These distances are summarized in Table 2.
If the network latent spaces truly possess disentanglement, we expect the z c neighbor to be closer in β distance and the z n neighbor to be closer in θ distance.