Neural Volumetric Mesh Generator
Abstract
Deep generative models have shown success in generating 3D shapes with different representations.
In this work, we propose Neural Volumetric Mesh Generator (NVMG), which can generate novel and high-quality volumetric meshes.
Unlike the previous 3D generative model for point cloud, voxel, and implicit surface, volumetric mesh is a ready-to-use representation in industry with details on both the surface and interior.
Generating this kind of highly-structured data thus brings a great challenge.
To tackle this problem, we first propose to use a diffusion-based generative model to generate voxelized shapes with realistic shape and topology information.
With the voxelized shape, we can simply obtain a tetrahedral mesh as a template.
Further, we use a voxel-conditional neural network to predict the surface conditioned on the voxels, and progressively project the tetrahedral mesh to the predicted surface under regularization.
As shown in the experiments, without any post-processing, our pipeline can generate high-quality artifact-free volumetric and surface meshes.
Introduction
Automatic 3D contents creation is a key problem with the development of AR/VR.
Although generative models have revealed their power on audio and image synthesis [8, 15, 10, 8, 4, 11, 24, 12, 25], their performance is still unsatisfying in 3D generation.
In this work, we consider the problem of directly generating ready-to-use volumetric meshes.
Volumetric mesh is one of the most important representations of 3D shapes, which is widely adopted in computer graphics and engineering [23, 9, 13].
Without carefully handling geometric constraints [3, 19, 17, 18, 22], the generated meshes will have various defects, including flipping, self-intersection, large distortion, etc.
Thus, we present a novel pipeline, termed Neural Volumetric Mesh Generator (NVMG), for learning generative models of volumetric meshes.
We first generate a rough voxelized shape by a diffusion model adapted to voxels.
Splitting voxels into tetrahedra, we can easily get a standardized tetrahedral mesh that then serves as a template, which allows sufficient variety in topological changes [6] compared with pre-defined templates like ellipsoid.
Next, NVMG learns a neural network to estimate the continuous surface conditioned on the generated voxelized shape.
We progressively deform the outmost vertices on the tetrahedral mesh by projecting them to the predicted surface under a series of regularization terms.
The projection objective is carefully designed to promote smoothness of the mesh, preserve the topology of the tetrahedra and suppress the magnitude of distortion.
Like squeezing the clay, the deformation of the outmost vertex on the surface is propagated into the interior region implicitly by our regularization terms.
After the deformation, NVMG generates a ready-to-use volumetric mesh without defects like flipping or high distortion.
We empirically evaluate NVMG on unconditional volumetric mesh generation.
We show our generated volumetric has a competitive generation quality compared with state-of-the art point cloud generator [27].
When we compare our mesh deformation module with the state-of-the-art voxel-tomesh method, Neural Dual Contouring, we obtain a robust performance on converting generated voxel to mesh.
Neural Volumetric Mesh Generator 2.1 Voxel Generation with Diffusion Models
Voxel Generation with DDPM
The first step of our framework is voxel generation.
We exploit denoising diffusion probabilistic model (DDPM)
[12] for this task because of its simplicity and effectiveness.
In our scenario, shapes are normalized to a unit cube with r × r × r voxels.
A voxel representation of a shape is encoded by a binary tensor V ∈ { 0 , 1 } r × r × r , where V ( x, y, z ) = {+
 /means+} that the ( x, y, z ) -th voxel is unfilled/filled.
The training set for voxel generation is created by discretizing the original meshes in the training set to their voxel representations with specific resolution.
We adopt a similar 3D convolutional neural network as in [27] to parameterize the noise predictor ϵ ϕ , with several modifications to accommodate it to the voxel generation task.
Tetrahedral Mesh from Voxel Division
We then divide each voxel into 6 tetrahedra as in
Figure 1(Division) to achieve the face-to-face consistency across the tetrahedral mesh.
Viewing each voxel as a cube mesh, we can extract a hexahedral mesh H from the voxel representation H =
( V , F quad ) =
( ∪ c { v ic } i =
i
=1 , ∪ c { f jc } j =6 j =1 ) , where V is vertex set, F quad is qudrilateral face set, v ic is the i -th vertex of the c -th cube and f jc is the j -th quadrilateral face of the c -th cube.
By splitting each cube mesh into 6 tetrahedra, we can further extract a tetrahedral mesh
T =
( V , F tri ) =
( ∪ c 6 ∪ k =1 { v ic,k } i =4 i =1 , ∪ c 6 ∪ k =1 { f jc,k } j =4 j =1 ) , where V is the vertex set same as hexahedral mesh, F tri is triangle face set, v ic,k is the i -th vertex of the k -th tetrahedron in the c -th cube and f jc,k is the j -th triangular face of the k -th tetrahedron in the c -th cube.
For volumetric meshes, as there are interior meshes, we use V s to denote the set of the outmost vertices on the surface.
2.2 A Voxel-Conditional Neural Closest Point Predictor
The voxelized shape and its corresponding tetrahedral mesh is a coarse representation the shape of interest.
In this section, we use a neural network to predict the fine-grained surface given the voxelized shape.
We represent the surface by the closest point function.
Given a surface S , the closest point function maps any point x ∈ R 3 to its closest point on S , CP( x ) =
arg min p ∈ S
∥ p − x
∥ 2 .
However, in generation tasks, S is not known beforehand.
Therefore, we use a neural network h θ ( V, x ) : { 0 , 1 } r 3 × R 3 → R {+
+} to learn the closest point function conditioned on the voxelized shape V .
Given V , the surface S is implicitly defined by S
= { x ∈ R 3 | h θ ( V, x ) − x = 0 } .
To prepare training samples, we calculate the closest point p s on the ground truth object surface to each vertex v s ∈ V s on triangular mesh surface.
2.3 Physically Robust Mesh Deformation with Regularized Projection
Now we introduce how to get the resulting mesh from the voxelized shape and the trained voxelconditional neural surface predictor h θ .
A straightforward idea is to directly project the vertices to its closest point inferred by h θ .
Nonetheless, this naive projection does not work well in practice.
Therefore, we resort to a regularized projection to optimize the vertices collectively.
The regularization terms are designed to encourage the set of vertices to satisfy the necessary conditions for successfully constructing a volumetric mesh.
Our optimization problem is formulated as follows,
[Equation]
˜ L proj is the projection term and R smooth and R orientation are the regulatization terms.
Note that our regularization term R ( V ) only changes the distribution of the vertices in the space, but not modify the target surface S .
We define R ( V ) to be differentiable so that L ( V ) can be easily optimized with any gradient-based optimizer.
1) Smooth term for uniform structure
A satisfying volumetric mesh should not only have a surface mesh T s that fits the target surface closely, but should also have a uniform inner volumetric tetrahedral mesh.
For this purpose, we add R smooth :
[Equation]
We keep the mesh connectivity F tri and F stri invariant during optimization.
R edge penalizes the length of edges to prevent extremely long edges; R laplacian is a graph-based term smoothing the mesh and R normal enforces normal consistency among adjacent faces on the surface.
2) Orientation term to prevent defects Defects on mesh connectivity like flipping breaks local injectivity[1] and causes invalid tetrahedral mesh.
It will make the mesh unqualified for certain downstream applications without applying non-trivial mesh repair.
The orientation term, which is widely used in many simulation tasks [17, 18, 22], ensures a minimal positive volume v 0 of tetrahedron, which helps to avoid unexpected artifacts on the generated mesh:
[Equation]
where M c,k ∈ R 4 × 4 is the matrix stacked by the homogeneous coordinates of four vertices of the k -th tetrahedron in the c-th cube.
3) Robust Projection for distortion suppression We empirically observe that, if all the vertices approaches the surface, they will collide with each other, causing excessive gradients, and finally gives meshes with large distortion.
To avoid this, we replace L proj with a robust version,
[Equation]
5) where n ∼ N (0 , 1) is a random Gaussian noise, k is a coefficient that gradually decays to 0 as optimization proceeds.
By adding this noise term, the ‘lucky’ vertices arrive at the surface earlier, while the ‘unlucky’ vertices arrive later.
Thus the collisions and the excessive gradients are avoided, and the distortion on the resulting mesh is significantly reduced.
After obtaining the voxelized shape and the tetrahedral mesh from Sec.
2.1, we optimize Eq.
(1) to yield the final volumetric mesh.
Experiment
In this section [Table] we empirically show that NVMG can generate high-fidelity shapes with fine details on the mesh surface by comparing with other unconditional generative model on 3D shape generation task.
For the evaluation metric, as discussed in [26, 27], 1-Nearest Neighbor(1-NN) is a more robust metric comparing with other metrics, so we choose it in our experiment.
For data processing, we use
Chair and Airplane classes and voxelized the raw mesh to 32 × 32 × 32 into voxel with the scale fitting the bounding box, which means making full use of the voxel resolution.
After generating the mesh, we uniformly sample 2048 points on the surface to calculate the 1-NN score.
Result.
We see from Table [Table] NVMG gets a comparable and even better performance compared with the state-of-the-art point cloud generation baseline.
Figure 3 shows the visualization of our generated samples comparing with point cloud generative model PVD [27] by sampling the point on the mesh
[Table]
Lower score means better generate quality and diversity.
[Table] [Figure]
[Figure]
Robustness of the mesh deformer.
We compare our mesh deformation part with the state-of-the- method Neural Dual Contouring (NDC) [7] by taking our generated voxel samples as input to show our designed mesh deformer is more robust and suit for our proposed pipeline.
From Table 1 last two lines, we observed a higher performance of our method compared with NDC.
Further in Figure 2, we visualize some cases and see our result is more robust with less artifact in the hard cases.
The above result indicates our design of the deformation-based method and robustness regularization loss works better in our proposed pipeline comparing with the off-the-shell marching cube based voxel to mesh method.