3DILG: Irregular Latent Grids for 3D Generative Modeling
Abstract
We propose a new representation for encoding 3D shapes as neural ﬁelds.
The representation is designed to be compatible with the transformer architecture and to beneﬁt both shape reconstruction and shape generation.
Existing works on neural ﬁelds are grid-based representations with latents deﬁned on a regular grid.
In contrast, we deﬁne latents on irregular grids, enabling our representation to be sparse and adaptive.
In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy.
For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models.
We show different applications that improve over the current state of the art.
First, we show results for probabilistic shape reconstruction from a single higher resolution image.
Second, we train a probabilistic model conditioned on very low resolution images.
Third, we apply our model to category-conditioned generation.
All probabilistic experiments conﬁrm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.
Introduction
[Figure]
Neural ﬁelds for 3D shapes are popular in machine learning because they are generally easier to process with neural networks than other alternative representations, e g
, triangle meshes or spline surfaces.
Earlier works represent a shape with a single global latent [29, 13, 7, 36, 66].
This already gives promising results in shape autoencoding and shape reconstruction.
However, shape details are often missing and hard to recover from a global latent.
Later, local latent grids for shapes were proposed [39, 23, 3, 48].
A local latent mainly inﬂuences the shape (surface) in a local 3D neighborhood, thus perceiving shape details. However, in contrast to global latents, local latents
[Figure]
require positional information about their location, e.g ., as implicitly deﬁned by a regular grid.
Furthermore, multiscale latent pyramid grids [9, 6] provide some performance improvement over basic grids.
We illustrate the three current ways of modeling neural ﬁelds in Fig. 2.
In this paper, we set out to study irregular grids as 3D shape representation for neural ﬁelds.
This extends previous grid-based representations, but allows each latent to have an arbitrary position in space, rather than a position that is pre-determined on a regular grid.
We do not want to give up the advantages of a ﬁxed length representation, and therefore propose to encode shapes as a ﬁxed length sequence of tuples ( x i , z i ) i ∈M , where x i are the 3D positions and z i are the latents.
The overall advantages of this representation are that it is ﬁxed length, works well with transformer-based architectures using established building blocks, and that it can adapt to the 3D shape it encodes by placing latents at the most important positions.
The representation also scales better than grid-based (and especially pyramid-based)
representations to a larger number of latents.
The number of latents in a representation is a factor that greatly inﬂuences the training time of transformer architectures, as the computation time is quadratic in the number of latents.
While our proposed representation brings some improvement for 3D shape reconstruction from point clouds, the improvement is very signiﬁcant for 3D generative models that have been less explored.
We believe that the most promising avenue for 3D shape generation is to follow recent image generative methods based on vector quantization and auto-regressive models using transformer, e.g . VQGAN [16].
These models operate on discrete latent vectors represented by indices [53].
During training, an autoregressive probabilistic model is trained to predict discrete indices.
When doing inference (sampling), discrete indices are sampled from the autoregressive model and are decoded to images with a learned decoder.
These models work well in the image domain, because a single image can comfortably be represented by medium size, e.g .
× 16 latent grids oflatents [16].
This does not directly scale to 3D shapes, as a 16 × 16 × 16 grid is too large to be comfortably trained on 4-8 GPUs due to the quadratic complexity of the transformer architecture [55].
As a result, generative models based on low-resolution regular grids lead to artifacts in the details, whereas our representation yields a nice and clean surface ( e.g . Fig. 9).
We show the following applications of our proposed representations (see Fig. 1).
For 3D shape reconstruction, we show results for 3D reconstruction from a point cloud.
For generative modeling, we show results for image-conditioned 3D shape generation, object category-conditioned 3D shape generation, and point-cloud-conditioned 3D shape generation.
The generative model is probabilistic and can generate multiple different 3D shapes for the same conditioning information.
We summarize the contributions of our work as follows: 1) We propose irregular latent grids as 3D shape representation for neural ﬁelds.
Our shape representation thereby extends existing works using a global latent, a local latent grid or a multi-scale grid.
We improve upon grid-based SOTA methods for 3D shape reconstruction for point-clouds.
We improve state-of-the art generative models for 3D shape generation, including image-conditioned generation, object category-conditioned generation and point cloud-conditioned generation.
Related Work 2.1 Neural shape representations
Shape analysis with neural networks processes shapes in different representations.
Common representations include voxels [27, 10, 12, 44] and point clouds [40, 41, 57, 58, 51].
More recently, researchers study shapes represented with neural ﬁelds [63], e.g ., signed distance functions (SDFs) or occupancy (indicator) functions of shapes modeled by neural networks.
Subsequently, meshes can be extracted by contouring methods such as marching cubes [28].
The methods have been called neural implicit representations [29, 30, 36, 18, 13, 7, 66] or coordinate-based representations [47].
We decided to use the term neural ﬁelds in this paper [63].
Point cloud processing.
Earlier works for point cloud processing rely on multilayer perceptrons (MLPs), e.g ., PointNet [40].
PointNet++ [41] and DGCNN [57] extended the idea by employing a hierarchical structure to capture local information.
Inspired by ViT [15], which treats images as a set of patches, PCT [20] and PT [67] propose a transformer backbone for point cloud processing.
Both works introduce extra modules which are not standard transformers [55] anymore.
Since our main goal is not to develop a general backbone for point clouds, we simply work with a standard transformer for shape autoencoding (ﬁrst-stage training).
Similarly, both PointBERT[65] and PointMAE [34] use standard transformers for point cloud self-supervised learning.
Neural ﬁelds for 3D shapes.
One possible approach is to represent a shape with a single global latent, e.g ., OccNet [29], CvxNet [13] and Zhang et al .
While this method is very simple, it is not suitable to encode surface details or shapes with complex topology.
Later works studied the use of local latents.
Compared to global latents, these representations represent shapes with multiple local latents.
Each latent is responsible for reconstruction in a local region.
ConvONet [39], as a follow-up work of OccNet, learns latents on a grid (2d or 3d).
IF-Net [9], trains a model which outputs 3d latent grids of several different resolutions.
The grids are then concatenated into a ﬁnal representation.
Some more recent works [17, 26, 49] learn to put latents on 3D positions.
2.2 Generative models
Generative models include generative adversarial networks (GANs) [19], variational autoencoders (VAEs) [24], energy-based models [25, 59, 61, 62, 60], normalizing ﬂows (NFs) [43, 14] and auto-regressive models (ARs) [54].
Our shape representation is designed to be compatible with auto-regressive generative models.
Thus we will mainly discuss related work in this area.
In the image domain, earlier AR works generate pixels one-by-one, e.g ., PixelRNN [54] and its follow-up works [52, 45].
Combining this idea with autoregressive transformers, similar approaches are applied to 3D data, e.g ., PointGrow [46] for point cloud generation and PolyGen [33] for polygonal mesh generation.
Other use cases include ﬂoor plan generation [35] and indoor scene generation [56, 38].
An important ingredient of many auto-regressive models is vector quantization (VQ) that was originally integrated into VAEs.
VQVAE [53] adopts a two-stage training strategy.
The ﬁrst stage is an autoencoder with vector quantization, where bottleneck latents are quantized to discrete indices.
In the second stage, an autoregressive model is trained to predict the discrete indices.
This idea is further developed in VQGAN [16] which builds the autoregressive model with a transformer [55].
Vector quantization, has been shown to be an efﬁcient way for generating high resolution images.
We study how VQ can be used in the 3D domain, speciﬁcally, neural ﬁelds for 3D shapes.
There are some works adopting VQVAE to the 3D domain.
Canonical Mapping [8] proposes point cloud generation models with VQVAE.
In concurrent work, AutoSDF [32] generalizes VQVAE to 3D voxels.
However, the bottleneck latent grid resolution is low and it is difﬁcult to capture details as in ConvONet and IF-Net.
Increasing the resolution is the key for detailed 3D shape reconstruction.
On the other hand, higher resolution leads to more difﬁcult training of autoregressive transformers.
Another concurrent work, ShapeFormer [64], shows a solution by sparsifying latent grids.
However,
ShapeFormer still only accepts voxel grids as input (point clouds need to be voxelized).
Additionally, the sequence length varies for different shapes, requiring the method to go through a whole dataset to ﬁnd a maximum sequence length.
This relies on dataset-dependent maximum sequence lengths.
In contrast to existing works, our method processes point clouds directly, learns ﬁxed-size latents and outputs neural ﬁelds.
A comparison summary can be found in Table 1.
Shape Reconstruction
Here we build a shape autoencoding method.
Given a shape surface, we want to output the same shape surface.
We preprocess a shape surface via uniform sampling to a point cloud { x i : x i ∈ R 3 } i ∈N (also in matrix form X ∈ R N × 3 ), our goal is to predict the indicator function O ( · ) : R 3 → [0 , 1] corresponding to the volume represented by the point cloud.
Our representation is being generated in
[Table]
3DILG
[Table]
R N ×
R M ×
FPS
R M × K ×
KNN
R M × C
PointNet
R M × C
Transformer
R 3 → [0 , 1]
Reconstruction three steps (See Fig. 3): patch construction (Sec. 3.1), patch information processing (Sec. 3.2), and reconstruction (Sec. 3.3).
3.1 Patch Construction
We sub-sample the input point cloud via Farthest Point Sampling (FPS),
[Equation]
where M ⊂ N and |M| =
M .
Next, for each point in the sub-sampled point set { x i } i ∈M , we apply the K-nearest neighbor (KNN) algorithm to ﬁnd K − 1 points to form a point patch of size K ,
[Equation]
where N i is the neighbor index set for point x i and |N
i | =
K − 1 .
Thus we have a collection of point patches, { ( x i , { x j } j ∈N i ) :
|N i | =
K − 1 } i ∈M =
X M ,K ∈ R M
×
K × 3 .
We project each patch with a mini-PointNet-like [40] module to an embedding vector
[Equation]
where C is the embedding dimension for patches.
3.2 Transformer
Furthermore, we build a transformer to learn local latents { z i } i ∈M .
The point coordinates { x i } i ∈M are converted to positional embeddings (PEs) [31], p i =
PE( x i ) ∈ R C .
Our transformer takes as input the sequence of patch-PE pairs,
Transformer
[Equation]
[Equation]
The transformer is composed of L blocks.
We denote the intermediate outputs of all blocks as z (0) i , z (1) i , · · · , z ( l ) i , · · · , z ( L ) i , where z (0) i is the input e i and z ( L ) i is the output z i .
C ooo...
T r a n s f o r m e r oooo...
C zzz... x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,... T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s zzzz...
Steps (cid:45)
Vector Quantization.
This model can be used with vector quantization [53].
We simply replace the intermediate output at block l with its closest vector in a dictionary D ,
[Equation]
The dictionary D contains |D| = D vectors.
As in VQVAE [53], the gradient with respect to z ( l ) i is approximated with the straight-through estimator [1].
3.3 Reconstruction
We expect that each latent z i is responsible for shape reconstruction near the point x i .
However, our goal is to estimate O ( · ) for an arbitrary x ∈ R 3 .
We interpolate latent z x for point x with the
Nadaraya-Watson estimator, z x = (cid:80) i ∈M exp( − β [Equation] x − x i (cid:107)
[Equation]
z i (cid:80) i ∈M exp( − β (cid:107) x − x i (cid:107) 2 ) , where β controls the smoothness of interpolation and can be ﬁxed or learned.
The ﬁnal indicator is an MLP with a sigmoid activation, ˆ
O ( x ) =
Sigmoid(MLP( x , z x )) .
Loss Function.
We optimize the estimated ˆ O ( · ) by comparing to ground-truth O ( · ) via a binary cross entropy loss (BCE) and an additional commitment loss term [53],
[Equation]
where sg( · ) is the stop-gradient operation.
When λ = 0 , it means we train the model without vector quantization.
Autoregressive Generative Modeling
With Vector Quantization (Eq. 5), we compress the bit size of intermediate latents { ˆz ( l ) i } i ∈M to log 2 D where D is the size of the dictionary D .
We denote the compressed index as z i ∈ { 0 , 1 , . . .
, D − 1 } .
We also quantize point coordinates x i to ( x i, 1 , x i, 2 , x i, 3 ) where each entry is a 8-bit integer { 0 , 1 , . . .
, 255 } , As a result, we obtain a discrete representation of the 3D shape { ( x i, 1 , x i, 2 , x i, 3 , z i ) } i ∈M .
4.1 Unidirectional Transformer
Autoregressive generation with unidirectional transformer is a more classical approach.
We often need to sequentialize an unordered set if the order is not deﬁned yet.
Speciﬁcally, we re-order the representations in ascending order by the ﬁrst component x i, 1 , then by the second component x i, 2 , and ﬁnally by the third component x i, 3 ,
[Equation]
Our goal to predict sequence elements one-by-one.
A common way is to ﬂatten the sequence S by concatenating the quadruplets, for examples, PointGrow [46] and PolyGen [33].
However, here we consider an approach generating quadruplet-by-quadruplet .
We write o i =
( x i, 1 , x i, 2 , x i, 3 , z i ) .
The likelihood of generating o i autoregressively is
[Equation]
where C is conditioned context.
Similar to ATISS [38], we predict components of o i =
( x i, 1 , x i, 2 , x i, 3 , z i ) autoregressively:
[Equation]
where the context C is omitted to avoid clutter.
The model is shown in Fig.
Different from
ATISS [38] which uses MLPs to decode different components, instead we continue to apply transformer blocks for component decoding.
4.2 Bidirectional Transformer
Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [5].
Here we show our model can also be combined with bidirectional transformer.
However, here we consider a different task.
We generate { z i } i ∈M conditioned on { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M .
In training phase, we sample a subset { z i } i ∈V⊂M of { z i } i ∈M as the input of the bidirectional transformer.
We aim to predict { z i } i ∈M\V .
The coordinates { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M are converted to positional embeddings (either learned or ﬁxed) as condition.
The likelihood of generating M \ V is as follows,
[Equation]
In practice, the bidirectional transformer takes as input all tokens except that { z i } i ∈M\V are replaced by a special masked token [ m ] .
When decoding (inference), we iteratively predict multiple tokens at the same time.
Tokens are sampled based on their probabilities (transformer output).
See [5] for a detailed explanation.
We show visualization of decoding steps in Fig.
Reconstruction Experiments
We set the size of the input point cloud to N = 2048 .
The number and the size of point patches are M = 512 and K = 32 , respectively.
In the case of Vector Quantization, there are D =vectors in the dictionary D .
Other details of the implementation can be found in the Appendix.
We use the datset ShapeNet-v2 [4] for shape reconstruction.
We split samples into train/val/test ( 48597 / 1283 / 2592 ) set.
Following the evaluation protocol of [13, 66], we include three metrics, volumetric IoU, the Chamfer-L1 distance, and F-Score [50].
We also show reconstruction results on another object-level dataset ABO [11], a real-world dataset D-FAUST [2], and a synthetic scene-level dataset in the appendix.
Results We compare our method with three existing works, OccNet [29], ConvONet [39] and
IF-Net [9].
The results can be found in Table 2.
We select 7 categories among 55 categories with largest training set ( table, car, chair, airplane, sofa, riﬂe, lamp ).
Detailed results can be found in the
Appendix.
We show different choices of M .
As we can see in this table, increasing M from 64 togives a performance boost.
Even with the simplest model M = 64 , our results outperform ConvONet.
The best results are achieved when setting M = 512 .
In this case, our results lead in most categories when comparing to IF-Net.
The results of IF-Net are better than ours in terms of metric F-Score in some categories.
However, we argue that in this case the metric F-score is saturated (values are close to 1 ), which making it hard to compare.
We also show results after introducing vector quantization to our model in the same table.
We can see that vector quantization harms the performance slightly.
[Table]
Qualitative Comparison Qualitative results are shown in Fig.
From the visualization, it can be seen that the reconstruction quality increases as M increases, particularly for shapes with complex topology.
OccNet, as a global latent method, often fails to recover complex structures.
ConvONet can recover better structures than OccNet, due to its localized latents.
By learning multiscale latents,
IF-Net improves further upon ConvONet.
Our method with M = 64 outperforms ConvONet, and with M = 128 , the results are comparable with IF-Net.
Generative Experiments
We introduce three experiments to show how our model can be combined with auto-regressive transformers for generative modeling.
In Sec. 6.1, we show image-conditioned generation as probabilistic shape reconstruction from single image.
In Sec. 6.2, we show how we generate samples given a category label (using the 55 ShapeNet categories).
In Sec.
6.3, we further show generation conditioned on downsampled point clouds.
In contrast to the ﬁrst two tasks, the point-cloud-conditioned generation is based on a bidirectional transformer described in Sec 4.2.
More generative results on additional datasets ABO and D-FAUST can be found in the appendix.
[Figure]
6.1 Probabilistic Shape Reconstruction from a Single Image
We train a uni-directional transformer for this task (see Sec. 4.1).
The context C is an image.
To train this model, we render 40 images ( 224 × 224 ) of different views for each shape in ShapeNet.
The implementation of the uni-directional transformer is based on GPT [42].
It contains 24 blocks, where each block has an attention layer with 16 heads and 1024 embedding dimension.
When sampling, nucleus sampling [22] with top- p (0.85) and top- k (100) are applied to predicted token probabilities.
High resolution images.
We show some generated samples when C are high resolution images in Fig.
We compare our results with a deterministic method, OccNet [29].
As we can see in the results, the deterministic method (OccNet) tends to create blurred meshes.
However, our probabilistic reconstruction is able to output detailed meshes.
Low resolution images.
We also consider another more challenging task.
The input images are downsampled to low resolution ( 16 × 16 ).
In this case, the generative model has more freedom to ﬁnd multiple plausible interpretations, including variations with different topology (see Fig.
6.2 Category-Conditioned Generation
To generate shapes given a category label, we use the context C to encode the category label.
We employ the uni-directional transformer based on GPT as in Sec. 6.1.
We compare to our proposed [Figure] [Figure] baseline model that encodes shapes as a latent grid of resolution 8 3 .
To make a fair comparison, we also extend ViT [15] to the 3D voxel domain in the ﬁrst stage of training.
In the second stage, we use the same uni-directional transformer for sequence prediction.
Here the sequence length is 8 3 = 512 , which is the same as in our proposed model.
The baseline model is named Grid- 8 3 .
The comparison to the baseline model is in Fig.
We can see that the baseline is unable to generate high quality shapes.
We argue that this is because the representation is not expressive enough to capture surface details.
Furthermore, we show more generated samples of our model in Fig.
The three selected categories are bookshelf, bench and chair .
For both the baseline and our model, we render 10 images of predicted shapes, and calculate the Fréchet Inception Distance (FID) [21, 37] between predictions and test sets.
The metrics can be found in Table 3.
A perceptual study on the quality of generated samples can be found in the appendix.
6.3 Point Cloud Conditioned Generation
We train a bidirectional transformer described in Sec 4.2.
The model takes as input { x i } i ∈M .
The results can be found in Fig.
Conclusion
We have studied neural ﬁelds for shapes and presented a new representation.
In contrast to common approaches which deﬁne latents a on a regular grid or multiple regular grids, we position latents on an irregular grid.
Comparing to existing works, the representation better scales to larger models, because [Figure] [Table] the irregular grid is sparse and adapts to the underlying 3D shape of the object.
In our results, we demonstrated an improvement in 3D shape reconstruction from point clouds and generative modeling conditioned on images, object category, or point-clouds over alternative grid-based methods.
In future work, we suggest to explore other applications of our proposed representation, e.g ., shape completion, and extensions to textured 3D shapes and 3D scenes. Acknowledgements
We would like to acknowledge support from the SDAIA-KAUST Center of Excellence in Data
Science and Artiﬁcial Intelligence.
[Figure]
(cid:45)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] We specify the random seed.
The results are reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] See Appendix.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes]