3DILG: Irregular Latent Grids for 3D Generative Modeling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a new representation for encoding 3D shapes as neural ﬁelds.
Therepresentation is designed to be compatible with the transformer architectureand to beneﬁt both shape reconstruction and shape generation.
Existing workson neural ﬁelds are grid-based representations with latents being deﬁned on aregular grid.
In contrast, we deﬁne latents on irregular grids which facilitates ourrepresentation to be sparse and adaptive.
In the context of shape reconstructionfrom point clouds, our shape representation built on irregular grids improves upongrid-based methods in terms of reconstruction accuracy.
For shape generation,our representation promotes high-quality shape generation using auto-regressiveprobabilistic models.
We show different applications that improve over the currentstate of the art.
First, we show results of probabilistic shape reconstruction from asingle higher resolution image.
Second, we train a probabilistic model conditionedon very low resolution images.
Third, we apply our model to category-conditionedgeneration.
All probabilistic experiments conﬁrm that we are able to generatedetailed and high quality shapes to yield the new state of the art in generative 3Dshape modeling.
Introduction
[Figure]
This alreadygives promising results in shape autoencoding and shape reconstruction.
However, shape detailsare often missing and hard to recover from a global latent.
Later, local latent grids for shapes were
[Figure]
proposed [33, 20, 2]. A local latent mainly inﬂuences the shape (surface) in a local 3D neighborhood,thus perceiving shape details.
However, in contrast to global latents, local latents require positionalinformation about their location, e.g ., as implicitly deﬁned by a regular grid.
Furthermore, multiscalelatent pyramid grids [8, 5] provide some performance improvement over basic grids.
We illustratethe three current ways of modeling neural ﬁelds in Fig.
In this paper, we set out to study irregular grids as 3D shape representation for neural ﬁelds.
Thisextends previous grid-based representations, but allowing each latent to have an arbitrary positionin space, rather than a position that is pre-determined on a regular grid.
We do not want to give upthe advantages of a ﬁxed length representation, and therefore propose to encode shapes as a ﬁxedlength sequence of tuples ( x i , z i ) i ∈M , where x i are the 3D positions and z i are the latents.
Theoverall advantages of this representation are that it is ﬁxed length, works well with transformer-basedarchitectures using established building blocks, and that it can adapt to the 3D shape it encodes byplacing latents at the most important positions.
The representation also scales better than grid-based(and especially pyramid-based)
representations to a larger number of latents.
The number of latentsin a representation is a factor that greatly inﬂuences the training time of transformer architectures, asthe computation time is quadratic in the number of latents.
While our proposed representation brings some improvement for 3D shape reconstruction frompoint clouds, the improvement is very signiﬁcant for 3D generative models that have been lessexplored.
We believe that the most promising avenue for 3D shape generation is to follow recentimage generative methods based on vector quantization and auto-regressive models using transformer,e.g . VQGAN [14].
These models operate on discrete latent vectors represented by indices [44].
During training, an autoregressive probabilistic model is trained to predict discrete indices.
Whendoing inference (sampling), discrete indices are sampled from the autoregressive model and aredecoded to images with a learned decoder.
These models work well in the image domain, becausea single image can comfortably be represented by medium size, e.g .
× 16 latent grids of 256latents [14].
This does not directly scale to 3D shapes, as a 16 × 16 × 16 grid is too large to becomfortably trained on 4-8 GPUs due to the quadratic complexity of the transformer architecture [46].
As a result, generative models based on low-resolution regular grids lead to artifacts in the details,whereas our representation yields a nice and clean surface ( e.g . Fig. 9).
We show the following applications of our proposed representations (see Fig. 1).
For 3D shapereconstruction, we show results for 3D reconstruction from a point cloud.
For generative modeling,we show results for image-conditioned 3D shape generation, object category-conditioned 3D shapegeneration, and point-cloud-conditioned 3D shape generation.
The generative model is probabilisticand can generate multiple different 3D shapes for the same conditioning information.
We summarize the contributions of our work as follows: 1) We propose irregular latent grids as 3Dshape representation for neural ﬁelds.
Our shape representation thereby extends existing works usinga global latent, a local latent grid or a multi-scale grid.
We improve upon grid-based SOTA methodsfor 3D shape reconstruction for point-clouds.
We improve state-of-the art generative models for 3Dshape generation, including image-conditioned generation, object category-conditioned generationand point cloud-conditioned generation.
Related Work2.1 Neural shape representations
Shape analysis with neural networks processes shapes in different representations.
Common representations include voxels [9, 10, 37] and point clouds [34, 35, 48, 49, 42].
More recently, researchersstudy shapes represented with neural ﬁelds [50], e.g ., signed distance functions (SDFs) or occupancy (indicator) functions of shapes modeled by neural networks.
Subsequently, meshes can be extractedby contouring methods such as marching cubes [22].
The methods have been called neural implicitrepresentations [23, 24, 30, 15, 11, 6, 53] or coordinate-based representations [40].
We decided touse the term neural ﬁelds in this paper [50].
Point cloud processing.
Earlier works for point cloud processing rely on multilayer perceptrons(MLPs), e.g ., PointNet [34].
PointNet++ [35] and DGCNN [48] extended the idea by employing ahierarchical structure to capture local information.
Inspired by ViT [13], which treats images as a setof patches, PCT [17] and PT [54] propose a transformer backbone for point cloud processing.
Bothworks introduce extra modules which are not standard transformers [46] anymore.
Since our main goalis not to develop a general backbone for point clouds, we simply work with a standard transformerfor shape autoencoding (ﬁrst-stage training).
Similarly, both PointBERT[52] and PointMAE [28] usestandard transformers for point cloud self-supervised learning.
Neural ﬁelds for 3D shapes.
One possible approach is to represent a shape with a single globallatent, e.g ., OccNet [23], CvxNet [11] and Zhang et al .
While this method is very simple, it isnot suitable to encode surface details or shapes with complex topology.
Later works studied the use oflocal latents.
Compared to global latents, these representations represent shapes with multiple locallatents.
Each latent is responsible for reconstruction in a local region.
ConvONet [33], as a follow-upwork of OccNet, learns latents on a grid (2d or 3d).
IF-Net [8], trains a model which outputs 3d latentgrids of several different resolutions.
The grids are then concatenated into a ﬁnal representation.
2.2 Generative models
Generative models include generative adversarial networks (GANs) [16], variational autoencoders(VAEs)
[21], normalizing ﬂows (NFs)
[36, 12] and auto-regressive models (ARs) [45].
Our shaperepresentation is designed to be compatible with auto-regressive generative models.
Thus we willmainly discuss related work in this area.
In the image domain, earlier AR works generate pixels one-by-one, e.g ., PixelRNN [45] and itsfollow-up works [43, 38].
Combining this idea with autoregressive transformers, similar approachesare applied to 3D data, e.g ., PointGrow [39] for point cloud generation and PolyGen [27] forpolygonal mesh generation.
Other use cases include ﬂoor plan generation [29] and indoor scenegeneration [47, 32].
An important ingredient of many auto-regressive models is vector quantization (VQ) that wasoriginally integrated into VAEs.
VQVAE [44] adopts a two-stage training strategy.
The ﬁrst stage isan autoencoder with vector quantization, where bottleneck latents are quantized to discrete indices.
In the second stage, an autoregressive model is trained to predict the discrete indices.
This idea isfurther developed in VQGAN [14] which builds the autoregressive model with a transformer [46].
Vector quantization, has been shown to be an efﬁcient way for generating high resolution images.
Westudy how VQ can be used in the 3D domain, speciﬁcally, neural ﬁelds for 3D shapes.
There are some works adopting VQVAE to the 3D domain.
Canonical Mapping [7] proposes pointcloud generation models with VQVAE.
In concurrent work, AutoSDF [26] generalizes VQVAE to3D voxels.
However, the bottleneck latent grid resolution is low and it is difﬁcult to capture details asin ConvONet and IF-Net.
Increasing the resolution is the key for detailed 3D shapes reconstruction.
On the other hand, higher resolution, means more difﬁcult training of autoregressive transformers.
Another concurrent work, ShapeFormer [51], shows a solution by sparsifying latent grids.
However,
ShapeFormer still only accepts voxel grids as input (point clouds need to be voxelized).
Also, thesequence length varies for different shapes requiring the method to go through a whole dataset to ﬁnda maximum sequence length.
Again, the maximum sequence length is dataset-dependent.
Comparingto existing works, our method processes point clouds directly, learns ﬁxed-size latents and outputsneural ﬁelds.
A comparison summary can be found in Table 1.
Shape Reconstruction
Given an input point cloud { x i : x i ∈ R 3 } i ∈N (also in matrix form X ∈ R N × 3 ), our goal is topredict the indicator function O ( · ) : R 3 → [0 , 1] corresponding to the volume represented by the
[Table]
3DILG
[Table]
Ours
[Table]
R N ×
R M ×
FPS
R M × K ×
KNN
R M × C
PointNet
R M × C
Transformer
R 3 → [0 , 1]
Reconstruction point cloud.
Our representation is being generated in three steps (See Fig. 3): patch construction(Sec. 3.1), patch information processing (Sec
3.1 Patch Construction
We sub-sample the input point cloud via Farthest Point Sampling (FPS),
[Equation]
where M ⊂ N and |M| =
M .
Next, for each point in the sub-sampled point set { x i } i ∈M , we applythe K-nearest neighbor (KNN) algorithm to ﬁnd K − 1 points to form a point patch of size K ,
[Equation]
where N i is the neighbor index set for point x i and |N
i | =
K − 1 .
Thus we have a collection ofpoint patches, { ( x i , { x j } j ∈N i ) :
|N i | =
K − 1 } i ∈M =
X M ,K ∈ R M
×
K × 3 .
We project each patchwith a mini-PointNet-like [34] module to an embedding vector
[Equation]
where C is the embedding dimension for patches.
3.2 Transformer
Furthermore, we build a transformer to learn local latents { z i } i ∈M .
The point coordinates { x i } i ∈Mare converted to positional embeddings (PEs) [25], p i =
PE( x i ) ∈ R C .
Our transformer takes asinput the sequence of patch-PE pairs,
Transformer ( { ( e i , p i ) } i ∈M )
= { z i : z i ∈ R C } i ∈M .
The transformer is composed of L blocks.
We denote the intermediate outputs of all blocks asz (0) i , z (1) i , · · · , z ( l ) i , · · · , z ( L ) i , where z (0) i is the input e i and z ( L ) i is the output z i .
C ooo...
T r a n s f o r m e r oooo...
C zzz... x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,x 0 ,x 1 ,x 2 ,
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,... T r a n s f o r m e r B l o c k s x 0 ,x 1 ,x 2 ,x 3 ,...
T r a n s f o r m e r B l o c k s zzzz...
Steps (cid:45)
Vector Quantization.
This model can be used with vector quantization [44].
We simply replacethe intermediate output at block l with its closest vector in a dictionary D ,
[Equation]
The dictionary D contains |D| = D vectors.
As in VQVAE [44], the gradient with respect to z ( l ) i isapproximated with the straight-through estimator [1].
3.3 Reconstruction
We expect that each latent z i is responsible for shape reconstruction near the point x i .
However,our goal is to estimate O ( · ) for an arbitrary x ∈ R 3 .
We interpolate latent z x for point x with the
Nadaraya-Watson estimator, z x = (cid:80) i ∈M exp( − β (cid:107) x − x i (cid:107) 2 )
z i (cid:80) i ∈M exp( − β (cid:107) x − x i (cid:107) 2 ) , where β controls the smoothness ofinterpolation and can be ﬁxed or learned.
The ﬁnal indicator is an MLP with a sigmoid activation,ˆ O
( x ) =
Sigmoid(MLP( x , z x )) .
Loss Function.
We optimize the estimated ˆ O ( · ) by comparing to ground-truth O ( · ) via a binarycross entropy loss (BCE) and an additional commitment loss term [44],
[Equation]
where sg( · ) is the stop-gradient operation.
When λ = 0 , it means we train the model without vectorquantization.
Autoregressive Generative Modeling
With Vector Quantization (Eq. 5), we compress the bit size of intermediate latents { ˆz ( l ) i } i ∈Mto log 2 D where D is the size of the dictionary D .
We denote the compressed index as z i ∈{ 0 , 1 , . . .
, D − 1 } .
We also quantize point coordinates x i to ( x i, 1 , x i, 2 , x i, 3 ) where each entry isa 8-bit integer { 0 , 1 , . . .
, 255 } , As a result, we obtain a discrete representation of the 3D shape{ ( x 2 , x i, 3 , z i ) } i ∈M .
4.1 Unidirectional Transformer
Autoregressive Generation with unidirectional transformer is a more classical approach.
We oftenneed to sequentialize an unordered set if the order is not deﬁned yet.
Speciﬁcally, we re-order therepresentations ascendingly by ﬁrst component x i, 1 , then by second component x i, 2 , then by thirdcomponent x i, 3 ,
[Equation]
Our goal to predict sequence elements one-by-one.
A common way is to ﬂatten the sequence S byconcatenating the quadruplets, for examples, PointGrow [39] and PolyGen [27].
However, here weconsider an approach generating quadruplet-by-quadruplet .
We write o i =
( x i, 1 , x i, 2 , x i, 3 , z i ) .
Thelikelihood of generating o i autoregressively is
[Equation]
[Equation]
where C is conditioned context.
Similar to ATISS [32], we predict components of o i =( x
i, 1 , x i, 2 , x i, 3 , z i ) autoregressively:
[Equation]
9) where the context C is omitted to avoid clutter.
The model is shown in Fig.
Different from
ATISS [32] which uses MLPs to decode different components, instead we continue to apply transformer blocks for component decoding.
4.2 Bidirectional Transformer
Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [4].
Herewe show our model can also be combined with bidirectional transformer.
However, here we considera different task.
We generate { z i } i ∈M conditioned on { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M .
In training phase, wesample a subset { z i } i ∈V⊂M of { z i } i ∈M as the input of the bidirectional transformer.
We aim topredict { z i } i ∈M\V .
The coordinates { ( x i, 1 , x i, 2 , x i, 3 ) } i ∈M are converted to positional embeddings(either learned or ﬁxed) as condition.
The likelihood of generating M \ V is as follows,
[Equation]
In practice, the bidirectional transformer takes as input all tokens except that { z i } i ∈M\V are replacedby a special masked token [ m ] .
When decoding (inference), we iteratively predict multiple tokens atthe same time.
Tokens are sampled based on their probabilities (transformer output).
See [4] for adetailed explanation.
We show visualization of decoding steps in Fig.
Reconstruction Experiments
We set the size of the input point cloud to N = 2048 .
The number and the size of point patchesare M = 512 and K = 32 , respectively.
In the case of Vector Quantization, there are D = 1024vectors in the dictionary D .
Other details of the implementation can be found in the Appendix.
We use the datset ShapeNet-v2 [3] for shape reconstruction.
We split samples into train/val/test( 48597 / 1283 / 2592 ) set.
Following the evaluation protocol of [11, 53], we include three metrics, volumetric IoU, the Chamfer-L1 distance, and F-Score [41].
An evaluation on scene level reconstructioncan be found in the appendix.
Results We compare our method with three existing works, OccNet [23], ConvONet [33] and
IF-Net [8].
The results can be found in Table 2.
We select 7 categories among 55 categories withlargest training set ( table, car, chair, airplane, sofa, riﬂe, lamp ).
Detailed results can be found inthe Appendix.
We show different choices of M .
As we can see in this table, increasing M from 64to 512 gives performance boost.
Even with the simplest model M = 64 , our results are better than ConvONet’s.
The best results are achieved when setting M = 512 .
In this case, our results lead inmost categories when comparing to IF-Net.
The results of IF-Net are better than ours in terms ofmetric F-Score in some categories.
However, we argue that in this case the metric F-score is saturated(values are close to 1 ), which making it hard to compare.
We also show results after introducingvector quantization to our model in the same table.
We can see that, vector quantization harms theperformance slightly.
[Table]
[Figure]
Qualitative Comparison The qualitative results are shown in Fig.
From the visualization, it canbe seen that the reconstruction quality increases as M increases, especially for shapes with complextopology.
OccNet, as a global latent method, often fails to recover complex structures.
The recoveringability of ConvONet are much better than OccNet because of localized latents.
By learning multiscalelatents, IF-Net improves further upon ConvONet.
Regarding our method, results of M = 64 arebetter than ConvONet. When M = 128 , the results are comparable with IF-Net.
Generative Experiments
We introduce three experiments to show how our model can be combined with auto-regressivetransformers for generative modeling.
In Sec. 6.1, we show image-conditioned generation, a.k.a,probabilistic shape reconstruction from single image.
In Sec. 6.2, we show how we generate samplesgiven a category label (using the 55 ShapeNet categories).
In Sec.
6.3, we further show generationconditioned on downsampled point clouds.
Different from the ﬁrst two tasks, the point-cloudconditioned generation is based on a bidirectional transformer described in Sec 4.2.
6.1 Probabilistic Shape Reconstruction from a Single Image
We train a uni-directional transformer for this task (see Sec. 4.1).
The context C is an image.
To trainthis model, we render 40 images ( 224 × 224 ) of different views for each shape in ShapeNet.
Theimplementation of the uni-directional transformer is based on GPT.
It contains 24 blocks, where eachblock has an attention layer with 16 heads and 1024 embedding dimension.
When sampling, nucleussampling [19] with top- p (0.85) and top- k (100) are applied to predicted token probabilities.
High resolution images.
We show some generated samples when C are high resolution imagesin Fig.
We compare our results with a deterministic method, OccNet [23].
As we can see in theresults, the deterministic method (OccNet) tends to create blurred meshes.
However, probabilisticreconstruction (ours) is able to output detailed meshes.
[Figure] [Figure]
Low resolution images.
We also consider another more challenging task.
The input images aredown-sampled to low resolution ones ( 16 × 16 ).
In this case, the generative model has more freedomto ﬁnd multiple plausible interpretations, including variations with different topology (see Fig.
6.2 Category-Conditioned Generation
To generate shapes given a category label, we use the context C to encode the category label.
Weemploy the uni-directional transformer based on GPT as in Sec.
6.1.We compare to our proposedbaseline model that encodes shapes as a latent grid of resolution 8 3 .
To make a fair comparison, wealso extend ViT [13] to the 3D voxel domain in the ﬁrst stage of training.
In the second stage, we usethe same uni-directional transformer for sequence prediction.
Here the sequence length is 8 3 = 512 ,which is the same as in our proposed model.
The baseline model is named Grid- 8 3 .
The comparisonto the baseline model is in Fig.
We can see that the baseline is unable to generate high qualityshapes.
We argue that this is because the representation is not expressive enough to capture surfacedetails.
Furthermore, we show more generated samples of our model in Fig.
The three selectedcategories are bookshelf, bench and chair .
For both the baseline and our model, we render 10 imagesof predicted shapes, and calculate the Fréchet Inception Distance (FID) [18, 31] between predictionsand test sets.
The metrics can be found in Table 3.
A perceptual study on the quality of generatedsamples can be found in the appendix.
[Figure]
[Table]
6.3 Point Cloud Conditioned Generation
We train a bidirectional transformer described in Sec 4.2.
The model takes as input { x i } i ∈M .
Theresults can be found in Fig.
[Figure]
(cid:45)
Conclusion
We studied neural ﬁelds for shapes and presented a new representation.
Different from commonapproaches which deﬁne latents a on a regular grid or multiple regular grids, we position latents on anirregular grid.
Comparing to existing works, the representation better scales to larger models, becausethe irregular grid is sparse and adapts to the underlying 3D shape of the object.
In our results, wedemonstrated an improvement in 3D shape reconstruction from point clouds and generative modelingconditioned on images, object category, or point-clouds over alternative grid-based methods.
In futurework, we suggest to explore other applications of our proposed representation, e.g ., shape completion,and extensions to textured 3D shapes and 3D scenes.