3DILG: Irregular Latent Grids for 3D Generative Modeling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a new representation for encoding 3D shapes as neural ﬁelds.
Therepresentation is designed to be compatible with the transformer architecture andto beneﬁt both shape reconstruction and shape generation.
Existing works onneural ﬁelds are grid-based representations with latents deﬁned on a regular grid.
In contrast, we deﬁne latents on irregular grids, enabling our representation to besparse and adaptive.
In the context of shape reconstruction from point clouds, ourshape representation built on irregular grids improves upon grid-based methods interms of reconstruction accuracy.
For shape generation, our representation promoteshigh-quality shape generation using auto-regressive probabilistic models.
We showdifferent applications that improve over the current state of the art.
First, we showresults for probabilistic shape reconstruction from a single higher resolution image.
Second, we train a probabilistic model conditioned on very low resolution images.
Third, we apply our model to category-conditioned generation.
All probabilisticexperiments conﬁrm that we are able to generate detailed and high quality shapesto yield the new state of the art in generative 3D shape modeling.
Introduction
[Figure]
Neural ﬁelds for 3D shapes are popular in machine learning because they are generally easier to process with neural networks than other alternative representations
, triangle meshes or spline surfaces
Earlier works represent a shape with a single global latent [26, 11, 6, 33, 57].
This alreadygives promising results in shape autoencoding and shape reconstruction.
However, shape detailsare often missing and hard to recover from a global latent.
Later, local latent grids for shapes wereproposed [36, 21, 2].
A local latent mainly inﬂuences the shape (surface) in a local 3D neighborhood,
[Figure]
thus perceiving shape details.
However, in contrast to global latents, local latents require positionalinformation about their location, e.g ., as implicitly deﬁned by a regular grid.
Furthermore, multiscalelatent pyramid grids [8, 5] provide some performance improvement over basic grids.
We illustratethe three current ways of modeling neural ﬁelds in Fig.
In this paper, we set out to study irregular grids as 3D shape representation for neural ﬁelds.
Thisextends previous grid-based representations, but allows each latent to have an arbitrary position inspace, rather than a position that is pre-determined on a regular grid.
We do not want to give upthe advantages of a ﬁxed length representation, and therefore propose to encode shapes as a ﬁxedlength sequence of tuples ( x i , z i ) i ∈M , where x i are the 3D positions and z i are the latents.
Theoverall advantages of this representation are that it is ﬁxed length, works well with transformer-basedarchitectures using established building blocks, and that it can adapt to the 3D shape it encodes byplacing latents at the most important positions.
The representation also scales better than grid-based(and especially pyramid-based)
representations to a larger number of latents.
The number of latentsin a representation is a factor that greatly inﬂuences the training time of transformer architectures, asthe computation time is quadratic in the number of latents.
While our proposed representation brings some improvement for 3D shape reconstruction frompoint clouds, the improvement is very signiﬁcant for 3D generative models that have been lessexplored.
We believe that the most promising avenue for 3D shape generation is to follow recentimage generative methods based on vector quantization and auto-regressive models using transformer,e.g . VQGAN [14].
These models operate on discrete latent vectors represented by indices [48].
During training, an autoregressive probabilistic model is trained to predict discrete indices.
Whendoing inference (sampling), discrete indices are sampled from the autoregressive model and aredecoded to images with a learned decoder.
These models work well in the image domain, becausea single image can comfortably be represented by medium size, e.g .
× 16 latent grids of 256latents [14].
This does not directly scale to 3D shapes, as a 16 × 16 × 16 grid is too large to becomfortably trained on 4-8 GPUs due to the quadratic complexity of the transformer architecture [50].
As a result, generative models based on low-resolution regular grids lead to artifacts in the details,whereas our representation yields a nice and clean surface ( e.g . Fig. 9).
We show the following applications of our proposed representations (see Fig. 1).
For 3D shapereconstruction, we show results for 3D reconstruction from a point cloud.
For generative modeling,we show results for image-conditioned 3D shape generation, object category-conditioned 3D shapegeneration, and point-cloud-conditioned 3D shape generation.
The generative model is probabilisticand can generate multiple different 3D shapes for the same conditioning information.
We summarize the contributions of our work as follows: 1) We propose irregular latent grids as 3Dshape representation for neural ﬁelds.
Our shape representation thereby extends existing works usinga global latent, a local latent grid or a multi-scale grid.
We improve upon grid-based SOTA methodsfor 3D shape reconstruction for point-clouds.
We improve state-of-the art generative models for 3Dshape generation, including image-conditioned generation, object category-conditioned generationand point cloud-conditioned generation.
Related Work
2.1 Neural shape representations
Shape analysis with neural networks processes shapes in different representations.
Common representations include voxels [24, 9, 10, 41] and point clouds [37, 38, 52, 53, 46].
More recently, researchersstudy shapes represented with neural ﬁelds [54], e g ., signed distance functions (SDFs) or occupancy(indicator) functions of shapes modeled by neural networks.
Subsequently, meshes can be extracted by contouring methods such as marching cubes [25].
The methods have been called neural implicitrepresentations [26, 27, 33, 16, 11, 6, 57] or coordinate-based representations [44].
We decided touse the term neural ﬁelds in this paper [54].
Point cloud processing.
Earlier works for point cloud processing rely on multilayer perceptrons(MLPs), e.g ., PointNet [37].
PointNet++ [38] and DGCNN [52] extended the idea by employing ahierarchical structure to capture local information.
Inspired by ViT [13], which treats images as a setof patches, PCT [18] and PT [58] propose a transformer backbone for point cloud processing.
Bothworks introduce extra modules which are not standard transformers [50] anymore.
Since our main goalis not to develop a general backbone for point clouds, we simply work with a standard transformerfor shape autoencoding (ﬁrst-stage training).
Similarly, both PointBERT[56] and PointMAE [31] usestandard transformers for point cloud self-supervised learning.
Neural ﬁelds for 3D shapes.
One possible approach is to represent a shape with a single globallatent, e.g ., OccNet [26], CvxNet [11] and Zhang et al .
While this method is very simple, itis not suitable to encode surface details or shapes with complex topology.
Later works studied theuse of local latents.
Compared to global latents, these representations represent shapes with multiplelocal latents.
Each latent is responsible for reconstruction in a local region.
ConvONet [36], as afollow-up work of OccNet, learns latents on a grid (2d or 3d).
IF-Net [8], trains a model whichoutputs 3d latent grids of several different resolutions.
The grids are then concatenated into a ﬁnalrepresentation.
Some more recent works [15, 23] learn to put latents on some 3D positions.
2.2 Generative models
Generative models include generative adversarial networks (GANs) [17], variational autoencoders(VAEs)
[22], normalizing ﬂows (NFs)
[40, 12] and auto-regressive models (ARs) [49].
Our shaperepresentation is designed to be compatible with auto-regressive generative models.
Thus we willmainly discuss related work in this area.
In the image domain, earlier AR works generate pixels one-by-one, e.g ., PixelRNN [49] and itsfollow-up works [47, 42].
Combining this idea with autoregressive transformers, similar approachesare applied to 3D data, e.g ., PointGrow [43] for point cloud generation and PolyGen [30] forpolygonal mesh generation.
Other use cases include ﬂoor plan generation [32] and indoor scenegeneration [51, 35].
An important ingredient of many auto-regressive models is vector quantization (VQ) that wasoriginally integrated into VAEs.
VQVAE [48] adopts a two-stage training strategy.
The ﬁrst stage isan autoencoder with vector quantization, where bottleneck latents are quantized to discrete indices.
In the second stage, an autoregressive model is trained to predict the discrete indices.
This idea isfurther developed in VQGAN [14] which builds the autoregressive model with a transformer [50].
Vector quantization, has been shown to be an efﬁcient way for generating high resolution images.
Westudy how VQ can be used in the 3D domain, speciﬁcally, neural ﬁelds for 3D shapes.
There are some works adopting VQVAE to the 3D domain.
Canonical Mapping [7] proposes pointcloud generation models with VQVAE.
In concurrent work, AutoSDF [29] generalizes VQVAE to3D voxels.
However, the bottleneck latent grid resolution is low and it is difﬁcult to capture details asin ConvONet and IF-Net.
Increasing the resolution is the key for detailed 3D shape reconstruction.
On the other hand, higher resolution leads to more difﬁcult training of autoregressive transformers.
Another concurrent work, ShapeFormer [55], shows a solution by sparsifying latent grids.
However,
ShapeFormer still only accepts voxel grids as input (point clouds need to be voxelized).
Additionally,the sequence length varies for different shapes, requiring the method to go through a whole dataset toﬁnd a maximum sequence length.
This relies on dataset-dependent maximum sequence lengths.
Incontrast to existing works, our method processes point clouds directly, learns ﬁxed-size latents andoutputs neural ﬁelds.
A comparison summary can be found in Table 1.
Shape Reconstruction
Here we build a shape autoencoding method. Given a shape surface, we want to output the same shapesurface. We preprocess a shape surface via uniform sampling to a point cloud { x i : x i ∈ R 3 } i ∈N(also in matrix form X ∈ R N × 3 ), our goal is to predict the indicator function O ( · ) : R 3 → [0 , 1]
[Table]
[Table] ConvONet [36] [Table] IF-Net [8]
[Table]
ShapeFormer [Table] [Table] 3DILG (Ours)
[Table]
Blocks
R N ×
R M ×
FPS
R M × K ×
KNN
R M × C
PointNet
R M × C
Transformer
R 3 → [0 , 1]
Reconstruction corresponding to the volume represented by the point cloud.
Our representation is being generated inthree steps (See Fig. 3): patch construction (Sec. 3.1), patch information processing (Sec. 3.2), andreconstruction (Sec. 3.3).
3.1 Patch Construction
We sub-sample the input point cloud via Farthest Point Sampling (FPS),
[Equation]
where M ⊂ N and |M| =
M .
Next, for each point in the sub-sampled point set { x i } i ∈M , we applythe K-nearest neighbor (KNN) algorithm to ﬁnd K − 1 points to form a point patch of size K ,
[Equation]
where N i is the neighbor index set for point x i and |N
i | =
K − 1 .
Thus we have a collection ofpoint patches, { ( x i , { x j } j ∈N i ) :
|N i | =
K − 1 } i ∈M =
X M ,K ∈ R M
×
K × 3 .
We project each patchwith a mini-PointNet-like [37] module to an embedding vector
[Equation]
where C is the embedding dimension for patches.
3.2 Transformer
Furthermore, we build a transformer to learn local latents { z i } i ∈M .
The point coordinates { x i } i ∈Mare converted to positional embeddings (PEs) [28], p i =
PE( x i ) ∈ R C .
Our transformer takes asinput the sequence of patch-PE pairs,
Transformer ( { ( e i , p i ) } i ∈M )
= { z i : z i ∈ R C } i ∈M [Equation]
The transformer is composed of L blocks.
We denote the intermediate outputs of all blocks asz (0) i , z (1) i , · · · , z ( l ) i , · · · , z ( L ) i , where z (0) i is the input e i and z ( L ) i is the output z i .
C ooo...
T r a n s f o r m e r oooo...
C zzz... x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,... T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s zzzz...
Steps (cid:45)
Vector Quantization.
This model can be used with vector quantization [48].
We simply replacethe intermediate output at block l with its closest vector in a dictionary D ,
[Equation]
The dictionary D contains |D| = D vectors.
As in VQVAE [48], the gradient with respect to z ( l ) i isapproximated with the straight-through estimator [1].
3.3 Reconstruction
We expect that each latent z i is responsible for shape reconstruction near the point x i .
However,our goal is to estimate O ( · ) for an arbitrary x ∈ R 3 .
We interpolate latent z x for point x with the
Nadaraya-Watson estimator, z x = (cid:80) i ∈M exp( − β (cid:107) x − x i (cid:107) 2 )
z i (cid:80) i ∈M exp( − β (cid:107) x − x i (cid:107) 2 ) , where β controls the smoothness ofinterpolation and can be ﬁxed or learned.
The ﬁnal indicator is an MLP with a sigmoid activation,ˆ O
( x ) =
Sigmoid(MLP( x , z x )) .
Loss Function.
We optimize the estimated ˆ O ( · ) by comparing to ground-truth O ( · ) via a binarycross entropy loss (BCE) and an additional commitment loss term [48],
[Equation]
where sg( · ) is the stop-gradient operation.
When λ = 0 , it means we train the model without vectorquantization.
Autoregressive Generative Modeling
With Vector Quantization (Eq. 5), we compress the bit size of intermediate latents { ˆz ( l ) i } i ∈Mto log 2 D where D is the size of the dictionary D .
We denote the compressed index as z i ∈{ 0 , 1 , . . .
, D − 1 } .
We also quantize point coordinates x i to ( x i, 1 , x i, 2 , x i, 3 ) where each entry isa 8-bit integer { 0 , 1 , . . .
, 255 } , As a result, we obtain a discrete representation of the 3D shape{ ( x i, 1 , x i, 2 , x i, 3 , z i ) } i ∈M .
4.1 Unidirectional Transformer
Autoregressive generation with unidirectional transformer is a more classical approach.
We oftenneed to sequentialize an unordered set if the order is not deﬁned yet.
Speciﬁcally, we re-order therepresentations in ascending order by the ﬁrst component x i, 1 , then by the second component x i, 2 ,and ﬁnally by the third component x i, 3 ,
[Equation]
Our goal to predict sequence elements one-by-one.
A common way is to ﬂatten the sequence S byconcatenating the quadruplets, for examples, PointGrow [43] and PolyGen [30].
However, here weconsider an approach generating quadruplet-by-quadruplet .
We write o i =
( x i, 1 , x i, 2 , x i, 3 , z i ) .
Thelikelihood of generating o i autoregressively is
[Equation]
where C is conditioned context.
Similar to ATISS [35], we predict components of o i =( x
i, 1 , x i, 2 , x i, 3 , z i ) autoregressively:
[Equation]
[Equation]
where the context C is omitted to avoid clutter.
The model is shown in Fig.
Different from
ATISS [35] which uses MLPs to decode different components, instead we continue to apply transformer blocks for component decoding.
4.2 Bidirectional Transformer
Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [4].
Herewe show our model can also be combined with bidirectional transformer.
However, here we considera different task.
We generate { z i } i ∈M conditioned on { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M .
In training phase, wesample a subset { z i } i ∈V⊂M of { z i } i ∈M as the input of the bidirectional transformer.
We aim topredict { z i } i ∈M\V .
The coordinates { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M are converted to positional embeddings(either learned or ﬁxed) as condition.
The likelihood of generating M \ V is as follows,
[Equation]
In practice, the bidirectional transformer takes as input all tokens except that { z i } i ∈M\V are replacedby a special masked token [ m ] .
When decoding (inference), we iteratively predict multiple tokens atthe same time.
Tokens are sampled based on their probabilities (transformer output).
See [4] for adetailed explanation.
We show visualization of decoding steps in Fig.
Reconstruction Experiments
We set the size of the input point cloud to N = 2048 .
The number and the size of point patchesare M = 512 and K = 32 , respectively.
In the case of Vector Quantization, there are D = 1024vectors in the dictionary D .
Other details of the implementation can be found in the Appendix.
We use the datset ShapeNet-v2 [3] for shape reconstruction.
We split samples into train/val/test( 48597 / 1283 / 2592 ) set.
Following the evaluation protocol of [11, 57], we include three metrics, volumetric IoU, the Chamfer-L1 distance, and F-Score [45].
An evaluation on scene level reconstructioncan be found in the appendix.
Results We compare our method with three existing works, OccNet [26], ConvONet [36] and
IF-Net [8].
The results can be found in Table 2.
We select 7 categories among 55 categories withlargest training set ( table, car, chair, airplane, sofa, riﬂe, lamp ).
Detailed results can be found in the Appendix.
We show different choices of M .
As we can see in this table, increasing M from 64 to 512gives a performance boost.
Even with the simplest model M = 64 , our results outperform ConvONet.
The best results are achieved when setting M = 512 .
In this case, our results lead in most categorieswhen comparing to IF-Net.
The results of IF-Net are better than ours in terms of metric F-Score insome categories.
However, we argue that in this case the metric F-score is saturated (values are closeto 1 ), which making it hard to compare.
We also show results after introducing vector quantization toour model in the same table.
We can see that vector quantization harms the performance slightly.
[Table]
[Figure]
Qualitative Comparison Qualitative results are shown in Fig.
From the visualization, it can beseen that the reconstruction quality increases as M increases, particularly for shapes with complextopology.
OccNet, as a global latent method, often fails to recover complex structures.
ConvONetcan recover better structures than OccNet, due to its localized latents.
By learning multiscale latents, IF-Net improves further upon ConvONet.
Our method with M = 64 outperforms ConvONet, andwith M = 128 , the are comparable with IF-Net.
Generative Experiments
[Figure] We introduce three experiments to show how our model can be combined with auto-regressive transformers for generative modeling.
In Sec. 6.1, we show image-conditioned generation as probabilisticshape reconstruction from single image.
In Sec. 6.2, we show how we generate samples given acategory label (using the 55 ShapeNet categories).
In Sec.
6.3, we further show generation conditioned on downsampled point clouds.
In contrast to the ﬁrst two tasks, the point-cloud-conditionedgeneration is based on a bidirectional transformer described in Sec 4.2.
6.1 Probabilistic Shape Reconstruction from a Single Image
We train a uni-directional transformer for this task (see Sec. 4.1).
The context C is an image.
To trainthis model, we render 40 images ( 224 × 224 ) of different views for each shape in ShapeNet.
Theimplementation of the uni-directional transformer is based on GPT [39].
It contains 24 blocks, whereeach block has an attention layer with 16 heads and 1024 embedding dimension.
When sampling,nucleus sampling [20] with top- p (0.85) and top- k (100) are applied to predicted token probabilities.
High resolution images.
We show some generated samples when C are high resolution imagesin Fig.
We compare our results with a deterministic method, OccNet [26].
As we can see in theresults, the deterministic method (OccNet) tends to create blurred meshes.
However, our probabilisticreconstruction is able to output detailed meshes.
[Figure] [Figure]
Low resolution images.
We also consider another more challenging task.
The input images aredownsampled to low resolution ( 16 × 16 ).
In this case, the generative model has more freedom toﬁnd multiple plausible interpretations, including variations with different topology (see Fig.
6.2 Category-Conditioned Generation
To generate shapes given a category label, we use the context C to encode the category label.
Weemploy the uni-directional transformer based on GPT as in Sec. 6.1.
We compare to our proposedbaseline model that encodes shapes as a latent grid of resolution 8 3 .
To make a fair comparison, wealso extend ViT [13] to the 3D voxel domain in the ﬁrst stage of training.
In the second stage, we usethe same uni-directional transformer for sequence prediction.
Here the sequence length is 8 3 = 512 ,which is the same as in our proposed model.
The baseline model is named Grid- 8 3 .
The comparisonto the baseline model is in Fig.
We can see that the baseline is unable to generate high qualityshapes.
We argue that this is because the representation is not expressive enough to capture surfacedetails.
Furthermore, we show more generated samples of our model in Fig.
The three selectedcategories are bookshelf, bench and chair .
For both the baseline and our model, we render 10 imagesof predicted shapes, and calculate the Fréchet Inception Distance (FID) [19, 34] between predictionsand test sets.
The metrics can be found in Table 3.
A perceptual study on the quality of generatedsamples can be found in the appendix.
[Figure]
[Table]
6.3 Point Cloud Conditioned Generation
We train a bidirectional transformer described in Sec 4.2.
The model takes as input { x i } i ∈M .
Theresults can be found in Fig.
[Figure]
(cid:45)
Conclusion
We have studied neural ﬁelds for shapes and presented a new representation.
In contrast to commonapproaches which deﬁne latents a on a regular grid or multiple regular grids, we position latents on anirregular grid.
Comparing to existing works, the representation better scales to larger models, becausethe irregular grid is sparse and adapts to the underlying 3D shape of the object.
In our results, wedemonstrated an improvement in 3D shape reconstruction from point clouds and generative modelingconditioned on images, object category, or point-clouds over alternative grid-based methods.
In futurework, we suggest to explore other applications of our proposed representation, e.g ., shape completion,and extensions to textured 3D shapes and 3D scenes. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix.(b)
Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] We specify the random seed.
The results are reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster,
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Appendix.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes]