GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Anonymous Author(s)
AffiliationAddress email
Abstract
As several industries are moving towards modeling massive 3D virtual worlds,the need for content creation tools that can scale in terms of the quantity, quality,and diversity of 3D content is becoming evident.
In our work, we aim to trainperformant 3D generative models that synthesize textured meshes which can bedirectly consumed by 3D rendering engines, thus immediately usable in downstream applications.
Prior works on 3D generative modeling either lack geometricdetails, are limited in the mesh topology they can produce, typically do not supporttextures, or utilize neural renderers in the synthesis process, which makes theiruse in common 3D software non-trivial.
In this work, we introduce GET3D, a
G enerative model that directly generates E xplicit T extured 3D meshes with complex topology, rich geometric details, and high fidelity textures.
We bridge recentsuccess in the differentiable surface modeling, differentiable rendering as well as2D Generative Adversarial Networks to train our model from 2D image collections.
GET3D is able to generate high-quality 3D textured meshes, ranging fromcars, chairs, animals, motorbikes and human characters to buildings, achievingsignificant improvements over previous methods.
Introduction
Diverse, high-quality 3D content is becoming increasingly important for several industries, includinggaming, robotics, architecture and social platforms.
However, manual creation of 3D assets is verytime-consuming and requires specific technical knowledge as well as artistic modeling skills.
Oneof the main challenges is thus scale – while one can find 3D models on 3D marketplaces such as
Turbosquid [4] or Sketchfab [3], creating many 3D models to, say, populate a game or a movie with acrowd of characters that all look different still takes a significant amount of artist time.
To facilitate the content creation process and make it accessible to a variety of (novice) users,generative 3D networks that can produce high-quality and diverse 3D assets have recently become anactive area of research [5, 13, 36, 39, 46, 54, 58, 51].
However, to be practically useful for currentreal-world applications, 3D generative models should ideally fulfill the following requirements: ( capacity generate shapes detailed geometry topology output should be a textured mesh, which is a primary representation used by standard graphics software packages such as Blender [14] and Maya [1], and (c) We should be able to leverage 2D images for supervision, as they are more widely available than explicit 3D shapes
Prior work on 3D generative modeling has focused on subsets of the above requirements but nomethod to date fulfills all of them (Tab. 1).
For example, methods that generate 3D point clouds [5, typically do not produce textures and have to be converted to a mesh in post-processing
[Table]
Methods generating voxels often lack geometric details and do not produce texture [53, 18, 23, 33].
Generative models based on neural fields [36, 13] focus on extracting geometry but disregard texture.
Most of these also require explicit 3D supervision.
Finally, methods that directly output textured3D meshes [47, 46] typically require pre-defined shape templates and cannot generate shapes withcomplex topology and variable genus.
Recently, rapid progress in neural volume rendering [38] and 2D Generative Adversarial Networks(GANs) [29, 30, 28, 25, 45] has led to the rise of 3D-aware image synthesis [7, 50, 8, 42, 44, 21].
However, this line of work aims to synthesize multi-view consistent images using neural rendering inthe synthesis process and does not guarantee that meaningful 3D shapes can be generated.
While amesh can potentially be obtained from the underlying neural field representation using the marchingcube algorithm [32], extracting the corresponding texture is non-trivial.
In this work, we introduce a novel approach that aims to tackle all the requirements of a practicallyuseful 3D generative model.
Specifically, we propose GET3D, a G enerative model for 3D shapesthat directly outputs E xplicit T extured 3D meshes with high geometric and texture detail and arbitrarymesh topology.
In the heart of our approach is a generative process that utilizes a differentiableexplicit surface extraction method [51] and a differentiable rendering technique [40, 31].
The formerenables us to directly optimize and output textured 3D meshes with arbitrary topology, while the latterallows us to train our model with 2D images, thus leveraging powerful and mature discriminatorsdeveloped for 2D image synthesis.
Since our model directly generates meshes and uses a highlyefficient (differentiable) graphics renderer, we can easily scale up our model to train with imageresolution as high as 1024 × 1024 , allowing us to learn high-quality geometric and texture details.
We demonstrate state-of-the-art performance for unconditional 3D shape generation on multiplecategories with complex geometry from ShapeNet [9], Turbosquid [4] and Renderpeople [2], such aschairs, motorbikes, cars, human characters, and buildings.
With explicit mesh as output representation,
GET3D is also very flexible and can easily be adapted to other tasks, including: (a) learning togenerate decomposed material and view-dependent lighting effects using advanced differentiablerendering [12], without supervision, (b) text-guided 3D shape generation using CLIP [49] embedding.
Related Work
We review recent advances in 3D generative models for geometry and appearance, as well as 3D-awaregenerative image synthesis.3D Generative Models
In recent years, 2D generative models have achieved photorealistic qualityin high-resolution image synthesis [29, 30, 28, 45, 25, 17, 15].
This progress has also inspiredresearch in 3D content generation.
Early approaches aimed to directly extend the 2D CNN generatorsto 3D voxel grids [53, 18, 23, 33], but the high memory footprint and computational complexity of3D convolutions hinder the generation process at high resolution.
As an alternative, other works haveexplored point cloud [5, 54, 58, 39], implicit [36, 13], or octree [26] representations.
However, theseworks focus mainly on generating geometry and disregard appearance.
Their output representationsalso need to be post-processed to make them compatible with standard graphics engines.
More similar to our work, Textured3DGAN [47, 46] and DIBR [11] generate textured 3D meshes,but they formulate the generation as a deformation of a template mesh, which prevents them fromgenerating complex topology or shapes with varying genus, which our method can do.
PolyGen [41]and SurfGen [34] can produce meshes with arbitrary topology, but do not synthesize textures.
3D-Aware Generative Image Synthesis Inspired by the success of neural volume rendering [38]and implicit representations [36, 13], recent work started tackling the problem of 3D-aware imagesynthesis [7, 50, 42, 22, 21, 59, 8, 44].
However, neural volume rendering networks are typicallyslow to query, leading to long training times [7, 50], and generate images of limited resolution.
GIRAFFE [42] and StyleNerf [21] improve the training and rendering efficiency by performingneural rendering at a lower resolution and then upsampling the results with a 2D CNN.
However,the performance gain comes at the cost of a reduced multi-view consistency.
By utilizing a dualdiscriminator, EG3D [8] can partially mitigate this problem.
Nevertheless, extracting a texturedsurface from methods that are based on neural rendering is a non-trivial endeavor.
In contrast,
GET3D directly outputs textured 3D meshes that can be readily used in standard graphics engines.
Method
We now present our GET3D framework for synthesizing textured 3D shapes.
Our generation processis split into two parts: a geometry branch, which differentiably outputs a surface mesh of arbitrarytopology, and a texture branch that produces a texture field that can be queried at the surface points toproduce colors or can be extended to other surface properties such as material (Sec. 4.3.1).
Duringtraining, an efficient differentiable rasterizer is utilized to render the resulting textured mesh into2D high-resolution images.
The entire process is differentiable, allowing for adversarial trainingfrom images (with masks indicating an object of interest) by propagating the gradients from the 2Ddiscriminator to both generator branches.
Our model is illustrated in Fig.
In the following, we firstintroduce our 3D generator in Sec 3.1, before proceeding to the differentiable rendering and lossfunctions in Sec 3.2.
3.1 Generative Model of 3D Textured Meshes
We aim to learn a 3D generator M, E = G ( z ) to map a sample from a Gaussian distributionz ∈ N (0 , I ) to a mesh M with texture E .
Since the same geometry can have different textures, and the same texture can be applied to differentgeometries, we sample two random input vectors z 1 ∈ R 512 and z 2 ∈ R 512 .
Following StyleGAN [29,30, 28], we then use non-linear mapping networks f geo and f tex to map z 1 and z 2 to intermediatelatent vectors w 1 = f geo ( z 1 ) and w 2 = f tex ( z 2 ) which are further used to produce styles that controlthe generation of 3D shapes and texture, respectively.
We formally introduce the generator forgeometry in Sec.
3.1.1 and the texture generator in Sec. 3.1.2.
3.1.1 Geometry Generator
We design our geometry generator to incorporate DMTet [51], a recently proposed differentiablesurface representation.
DMTet represents geometry as a signed distance field (SDF) defined on adeformable tetrahedral grid [20], from which the surface can be differentiably recovered throughmarching tetrahedra [16].
Deforming the grid by moving its vertices results in a better utilization of its resolution.
By adopting DMTet for surface extraction, we can produce explicit meshes witharbitrary topology and genus.
We next provide a brief summary of DMTet and refer the reader to theoriginal paper for further details.
Let ( V T , T ) denote the full 3D space that the object lies in, where V T are the vertices in the tetrahedralgrid T .
Each tetrahedron T k ∈ T is defined using four vertices { v a k , v b k , v c k , v d k } , with k ∈{ 1 , . .
, K } , where K is the total number of tetrahedra, and
v i k ∈ V T , v i k ∈ R 3 .
In addition to its3D coordinates, each vertex v i contains the SDF value s i ∈ R and the deformation ∆ v i ∈ R 3 of thevertex from its initial canonical coordinate.
This representation allows recovering the explicit meshthrough differentiable marching tetrahedra [51], where SDF values in continuous space are computedby a barycentric interpolation of their value s i on the deformed vertices v ′
i = v i + ∆ v i .
Network Architecture We map w 1 ∈ R 512 to SDF values and deformations at each vertex v ithrough a series of conditional 3D convolutional and fully connected layers.
Specifically, we first use3D convolutional layers to generate a feature volume conditioned on w 1 .
We then query the featureat each vertex v i ∈ V T using trilinear interpolation and feed it into MLPs that outputs the SDF values i and the deformation ∆ v i .
In cases where modeling at a high-resolution is required (e.g. motorbikewith thin structures in the wheels), we further use volume subdivision following [51].
Differentiable Mesh Extraction After obtaining s i and ∆ v i for all the vertices, we use thedifferentiable marching tetrahedra algorithm to extract the explicit mesh.
Marching tetrahedradetermines the surface topology within each tetrahedron based on the signs of s i .
In particular,a mesh face is extracted when sign ( s i ) ̸ = sign ( s j ) , where i, j denotes the indices of vertices inthe edge of tetrahedron, and the vertices m i,j of that face are determined by a linear interpolationas m i,j = v ′
i s j − v ′
j s i s j − s i .
Note that the above equation is only evaluated when s
i ̸
= s j , thus it isdifferentiable, and the gradient from m i,j can be back-propagated into the SDF values s i anddeformations ∆ v i .
With this representation, the shapes with arbitrary topology can easily begenerated by predicting different signs of s i .
3.1.2 Texture Generator
Directly generating a texture map consistent with the output mesh is not trivial, as the generated shapecan have an arbitrary genus and topology.
We thus parameterize the texture as a texture field [43].
Specifically, we model the texture field with a function f t that maps the 3D location of a surface pointp ∈ R 3 , conditioned on the w 2 , to the RGB color c ∈ R 3 at that location.
Since the texture fielddepends on geometry, we additionally condition this mapping on the geometry latent code w 1 , suchthat c = f t ( p , w 1 ⊕ w 2 ) , where ⊕ denotes concatenation.
Network Architecture We represent our texture field using a tri-plane representation, which isefficient and expressive in reconstructing 3D objects [48] and generating 3D-aware images [8] .
Specifically, we follow [8, 30] and use a conditional 2D convolutional neural network to map thelatent code w 1 ⊕ w 2 to three axis-aligned orthogonal feature planes of size N × N ×
( C × 3) , where
N = 256 denotes the spatial resolution and C = 32 the number of channels.
Given the feature planes,the feature vector f t ∈ R 32 of a surface point p can be recovered as f t = (cid:80) e ρ ( π e ( p )) , where π e ( p ) is the projection of the point p to the feature plane e and ρ ( · ) denotes bilinear interpolation of thefeatures.
An additional fully connected layer is then used to map the aggregated feature vector f t tothe RGB color c .
Note that, different from other works on 3D-aware image synthesis [8, 21, 7, 50]that also use a neural field representation, we only need to sample the texture field at the locations ofthe surface points (as opposed to dense samples along a ray).
This greatly reduces the computationalcomplexity for rendering high-resolution images and guarantees to generate multi-view consistentimages by construction.
3.2 Differentiable Rendering and Training
In order to supervise our model during training, we draw inspiration from Nvdiffrec [40] that performsmulti-view 3D object reconstruction by utilizing a differentiable renderer.
Specifically, we renderthe extracted 3D mesh and the texture field into 2D images using a differentiable renderer [31], andsupervise our network with a 2D discriminator, which tries to distinguish the image from a real objector rendered from the generated object.
Differentiable Rendering We assume that the camera distribution C that was used to acquire theimages in the dataset is known.
To render the generated shapes, we randomly sample a camera c from
C , and utilize a highly-optimized differentiable rasterizer Nvdiffrast [31] to render the 3D mesh into a2D silhouette as well as an image where each pixel contains the coordinates of the corresponding 3Dpoint on the mesh surface.
These coordinates are further used to query the texture field to obtain the
RGB values.
Since we operate directly on the extracted mesh, we can render high-resolution imageswith high efficiency, allowing our model to be trained with image resolution as high as 1024 × 1024.
Discriminator & Objective We train our model using an adversarial objective.
We adopt thediscriminator architecture from StyleGAN [29], and use the same non-saturating GAN objective with
R1 regularization [35].
We empirically find that using two separate discriminators, one for RGBimages and another one for silhouettes, yields better results than a single discriminator operating onboth.
Let D x denote the discriminator, where x can either be an RGB image or a silhouette.
Theadversarial objective is then be defined as follows:
[Equation]
where g ( u ) is defined as g ( u ) =
− log(1+exp( − u )) , p x is the distribution of real images, R denotesrendering, and λ is a hyperparameter.
Since R is differentiable, the gradients can be backpropagatedfrom 2D images to our 3D generators.
Regularization To remove internal floating faces that are not visible in any of the views, we furtherregularize the geometry generator with a cross-entropy loss defined between the SDF values of theneighboring vertices [40]:
[Equation]
where H denotes binary cross-entropy loss and σ denotes the sigmoid function.
The sum in Eq. 2 isdefined over the set of unique edges S e in the tetrahedral grid, for which sign ( s i ) ̸
= sign ( s j ) .
The overall loss function is then defined as:
[Equation]
where µ is a hyperparameter that controls the level of regularization.
Experiments
We conduct extensive experiments to evaluate our model.
We first compare the quality of the3D textured meshes generated by GET3D to the existing methods using the ShapeNet [9] and
Turbosquid [4] datasets.
Next, we ablate our design choices in Sec. 4.2.
Finally, we demonstrate theflexibility of GET3D by adapting it to downstream applications in Sec. 4.3.
Additional experimentalresults and implementation
[Table]
4.1 Experiments on Synthetic Datasets
Datasets For evaluation on ShapeNet [9], we use three categories with complex geometry – Car ,
Chair , and Motorbike , which contain 7497, 6778, and 337 shapes, respectively.
We randomly spliteach category into training (70%), validation (10 %), and test (20 %), and further remove from thetest set shapes that have duplicates in the training set.
To render the training data, we randomlysample camera poses from the upper hemisphere of each shape.
For the Car and Chair categories,we use 24 random views, while for Motorbike we use 100 views due to less number of shapes.
Asmodels in ShapeNet only have simple textures, we also evaluate GET3D on an Animal dataset (442shapes) collected from TurboSquid [4], where textures are more detailed and we split it into training,validation and test as defined above.
Finally, to demonstrate the versatility of GET3D, we alsoprovide qualitative results on the House dataset collected from Turbosquid (563 shapes), and Human
Body dataset from Renderpeople [2] (500 shapes).
We train a separate model on each category.
Baselines We compare GET3D to two groups of works: 1) 3D generative models that rely on3D supervision: PointFlow [54] and OccNet [36].
Note that these methods only generate geometrywithout texture.
2) 3D-aware image generation methods: GRAF [50], PiGAN [7], and EG3D [8].
Metrics To evaluate the quality of our synthesis, we consider both the geometry and texture ofthe generated shapes.
For geometry, we adopt metrics from [5] and use both Chamfer Distance(CD) and Light Field Distance [10] (LFD) to compute the Coverage score and Minimum Matching
Distance.
For OccNet [36], GRAF [50], PiGAN [7] and EG3D [8], we use marching cubes to extractthe underlying geometry.
For PointFlow [54], we use Poisson surface reconstruction to convert apoint cloud into a mesh when evaluating LFD.
To evaluate texture quality, we adopt the FID [24]metric commonly used to evaluate image synthesis.
In particular, for each category, we render the testshapes into 2D images, and also render the generated 3D shapes from each model into 50k imagesusing the same camera distribution.
We then compute FID on the two image sets.
As the baselinesfrom 3D-aware image synthesis [50, 7, 8] do not directly output textured meshes, we compute FIDscore in two ways: ( i ) we use their neural volume rendering to obtain 2D images, which we refer toas FID-Ori, and ( ii ) we extract the mesh from their neural field representation using marching cubes,render it, and then use the 3D location of each pixel to query the network to obtain the RGB values.
We refer to this score, that is more aware of the actual 3D shape, as FID-3D. Further details on theevaluation metrics are available in the Appendix.
Experimental Results We provide quantitative results in Table.
2 and qualitative examples in Fig.
3and Fig.
Additional results are available in the supplementary video.
Compared to OccNet [36] thatuses 3D supervision during training, GET3D achieves better performance in terms of both diversity(COV) and quality (MMD), and our generated shapes have more geometric details.
PointFlow [54]outperforms GET3D in terms of MMD on CD, while GET3D is better in MMD on LFG.
Wehypothesize that this is because PointFlow directly optimizes on point locations, which favours
CD.
GET3D also performs favourably when compared to 3D-aware image synthesis methods, weachieve significant improvements over PiGAN [7] and GRAF [50] in terms of all metrics on alldatasets.
Our generated shapes also contain more detailed geometry and texture.
Compared withrecent work EG3D [8].
We achieve comparable performance on generating 2D images (FID-ori),while we significantly improve on 3D shape synthesis in terms of FID-3D, which demonstrates theeffectiveness of our model on learning actual 3D geometry and texture.
Since we synthesize textured meshes, we can export our shapes into Blender 1 .
We show renderingresults in Fig.
1 and 5.
GET3D is able to generate shapes with diverse and high quality geometry andtopology, very thin structures (motorbikes), as well as complex textures on cars, animals, and houses.
Shape Interpolation GET3D also enables shape interpolation, which can be useful for editingpurposes.
We explore the latent space of GET3D in Fig.
6, where we interpolate the latent codes togenerate each shape from left to right.
GET3D is able to faithfully generate a smooth and meaningfultransition from one shape to another.
We further explore the local latent space by slightly perturbing
[Figure]
the latent codes to a random direction.
GET3D produces novel and diverse shapes when applyinglocal editing in the latent space (Fig. 7).
4.2 Ablations
We ablate our model in two ways: 1) w/and w/o volume subdivision, 2) training using different image resolutions.
Further ablations are provided in the Appendix.
[Table]
Ablation of Volume Subdivision As shown in
Tbl. 2, volume subdivision significantly improvesthe performance on classes with thin structures (e.g.,motorbikes), while not getting gains on other classes.
We hypothesize that the initial tetrahedral resolutionis already sufficient to capture the detailed geometryon Chairs and Cars, and hence the subdivision cannot provide further improvements.
Ablating Different Image Resolutions We ablate the effect of the training image resolution in
Tbl. 3.
As expected, increased image resolution improves the performance in terms of FID and shapequality, as the network can see more details, which are often not available in the low-resolutionimages.
This corroborates the importance of training with higher image resolution, which are oftenhard to make use of for implicit-based methods.
4.3 Applications
4.3.1 Material Generation for View-dependent Lighting Effects
GET3D can easily be extended to also generate surface materials that are directly usable in moderngraphics engines.
In particular, we follow the widely used Disney BRDF [6, 27] and describe thematerials in terms of the base color ( R 3 ), metallic ( R ), and roughness ( R ) properties.
As a result, werepurpouse our texture generator to now output a 5-channel reflectance field (instead of only RGB).
To accommodate differentiable rendering of materials, we adopt an efficient spherical Gaussian(SG)
based deferred rendering pipeline [12].
Specifically, we rasterize the reflectance field into a
G-buffer, and randomly sample an HDR image from a set of real-world outdoor HDR panoramas
S light =
{ L SG } K , where L SG ∈ R 32 × 7 is obtained by fitting 32 SG lobes to each panorama.
The SGrenderer [12] then uses the camera c to render an RGB image with view-dependent lighting effects,which we feed into the discriminator during training.
Note that GET3D does not require materialsupervision during training and learns to generate decomposed materials in an unsupervised manner.
[Figure]
We provide qualitative results of generated surface materials in Fig. 8.
Despite unsupervised, GET3Ddiscovers interesting material decomposition, e.g., the windows are correctly predicted with a smallerroughness value to be more glossy than the car’s body, and the car’s body is discovered as moredielectric while the window is more metallic.
Generated materials enable us to produce realisticrelighting results, which can account for complex specular effects under different lighting conditions.
4.3.2 Text-Guided 3D Synthesis
Similar to image GANs, GET3D also supports text-guided 3D content synthesis by fine-tuninga pre-trained model under the guidance of CLIP [49].
Note that our final synthesis result is atextured 3D mesh.
To this end, we follow the dual-generator design from styleGAN-NADA [19],where a trainable copy G t and a frozen copy G f of the pre-trained generator are adopted.
Duringoptimization G t and G f both render images from 16 random camera views.
Given a text query, wesample 500 pairs of noise vectors z 1 and z 2 .
For each sample, we optimize the parameters of G t tominimize the directional CLIP loss [19] (the source text labels are “car”, “animal” and “house” forthe corresponding categories), and select the samples with minimal loss.
To accelerate this process,we first run a small number of optimization steps for the 500 samples, then choose the top 50 sampleswith the lowest losses, and run the optimization for 300 steps.
The results and comparison against a
SOTA text-driven mesh stylization method, Text2Mesh [37], are provided in Fig.
Conclusion
We introduced GET3D, a novel 3D generative model that is able to synthesize high-quality 3Dtextured meshes with arbitrary topology.
GET3D is trained using only 2D images as supervision.
We experimentally demonstrated significant improvements on generating 3D shapes over previousstate-of-the-art methods on multiple categories.
We hope that this work brings us one step closer todemocratizing 3D content creation using A.I..
Limitations While GET3D makes a significant step towards a practically useful 3D generativemodel of 3D textured shapes, it still has some limitations.
In particular, we still rely on 2D silhouettesas well as the knowledge of camera distribution during training.
As a consequence, GET3D wascurrently only evaluated on synthetic data.
A promising extension could use the advances in instancesegmentation and camera pose estimation to mitigate this issue and extend GET3D to real-worlddata.
GET3D is also trained per-category; extending it to multiple categories in the future, couldhelp us better represent the inter-category diversity.
Broader Impact We proposed a novel 3D generative model that generates 3D textured meshes, 304 which can be readily imported into current graphics engines.
Our model is able to generate shapes witharbitrary topology, high quality textures and rich geometric details, paving the path for democratizing 306 A.I. tool for 3D content creation.
As all machine learning models, GET3D is also prone to biasesintroduced in the training data.
Therefore, an abundance of caution should be applied when dealing 308 with sensitive applications, such as generating 3D human bodies, as GET3D is not tailored for theseapplications.
We do not recommend using GET3D if privacy or erroneous recognition could lead to 310 potential misuse or any other harmful applications.
Instead, we do encourage practitioners to carefullyinspect and de-bias the datasets before training our model to depict a fair and wide distribution of 312 possible skin tones, races or gender identities.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We provide extensive experiments in Sec. 4.
(b) Did you describe the limitations of your work?
[Yes] We provide the discussion onlimitations in Sec. 5, and failure cases in Appendix.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We providethe discussion in the Broader Impact paragraph with further discussions in the main
paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[No] We will release the code and instructions to reproduce all experiments upon acceptance of the manuscript.
The internal guidelines of our institution prevent us from releasing code at this stage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide training details in both Sec. 4 in the main paper andAppendix.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[No] Training existing 3D models, including ours, onlarge-scale 3D datasets is too computation costly to repeat multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We provide in the Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used ShapeNet [9]
core dataset, Turbosquid [4], Renderpeople [2] in Sec. 4.1.
We also used officialcode to reproduce baselines with citations.
In particular, OccNet [36], PiGAN [7],Pointflow [54], Eg3D [8].
(b) Did you mention the license of the assets?
[Yes] We provided the license of ShapeNet,
Turbosquid and RenderPeople in the Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We did not include new assets in Supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We provide discussion on this in the Appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] We provide discussion on this in Appendix.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Additional Experiments
In this section, we provide three additional experiments to demonstrate the robustness and effectiveness of GET3D as per reviewers’ suggestions.
We first provide additional results of GET3Dwhen trained noisy cameras and predicted 2D silhouettes from 2D segmentation networks in Sec.
6.1and 6.2.
We further provide addition experiments on StyleGAN generated realistic dataset from
GANverse3D [56] in Sec.
Finally, we provide additional comparison with EG3D [8] on humancharacter in Sec. 6.4.
[Table]
6.1 Robustness to Noisy Cameras
To demonstrate the robustness of GET3D to noisy cameras, we add Gaussian noises to the cameraposes when training GET3D, and provide experimental results in this section.
Experimental Settings When rendering the generated 3D shapes during training, we add a randomperturbation to the sampled camera pose.
In particular, for the rotation angle, we add a noise sampledfrom a Gaussian distribution with zero mean, and 10 degrees variance.
For the elevation angle, wealso add a noise sampled from a Gaussian distribution with zero mean, and 2 degrees variance.
Wetrain GET3D on ShapeNet Car dataset [9] in this experiment.
Experimental Results We provide quantitative results in Table 4, with qualitative examples in
Figure 10.
Adding camera noise only hurts the quantitative performance a little bit in terms of FID,while the qualitative results are almost the same as shown in Figure 5, demonstrating the robustnessof our method to a moderate level of noise in the camera poses.
One possible reason for the drop inthe FID is because the testing dataset that we used to calculate the FID score is rendered with actualcamera pose distribution, and has a mismatch with the one we used for training, making our modelhave an inferior score.
6.2 Robustness to Imperfect 2D Silhouettes
To demonstrate the robustness of GET3D when trained with imperfect 2D silhouettes, we replaceground truth 2D masks with the ones obtained from Detectron2 2 using pretrained PointRendcheckpoint, mimicking how people in the real world would obtain the 2D segmentation masks.
Experimental Settings Since our training images are rendered with the black background, we usetwo approaches to obtain the 2D silhouettes, the first is directly feeding the original training imageinto Detectron2 to obtain the predicted segmentation mask (we refer this as Mask-Black), and thesecond is adding a background image, which is randomly sampled from PASCAL-VOC 2012 dataset(we refer this as Mask-Random).
{+For Mask-Black version, the Detectron2 model achieved 97.
+}
Intersection-over-Union (IoU) and achieved 95.8 IoU in the Mask-Random version.
For simplicity,we also train the model on the Shapenet Car dataset [9] in this experiment.
Experimental Results Quantitative results are provided in Table 4, with qualitative examples in
Figure 11 and 12.
Although we see drops in the FID scores, the qualitative results are still similarto the original GET3D in Figure 5.
Our model is robust to a moderate level of imperfect masksobtained from a pre-trained 2D segmentation network.
Note that, in this scenario, the training datafor GET3D is different from the testing data that is used to compute the FID score, which couldbecome one reason for worse performance.
6.3 Experiments on "Real" Image
Since many real-world datasets lack camera poses, we follow GANverse3D [56] and utilize pretrained2D StyleGAN to generate a realistic car dataset.
We train GET3D on this dataset to demonstrate thepotential applications to real-world data.
Experimental Setting
Following GANverse3D [56], we manipulate the latent codes of 2D StyleGAN and generate multi-view car images.
To obtain the 2D segmentation of each image, we use
DatasetGAN [57] to predict the 2D silhouette.
We then use SfM [52] to obtain the camera initialization for each generated image.
We refer the readers to the original GANverse3D paper for moredetails.
Note that, in this dataset, both cameras and 2D silhouettes are imperfect, while our modelstill performs robustly to it.
Experimental Results We provide qualitative examples in Fig. 13.
GET3D performs decently wellon this dataset, generating reasonable 3D textured meshes, with variation in geometry and texture.
[Table]
[Figure]
6.4 Comparison with EG3D [8] on Human Body
As per the reviewer’s suggestion, we train the EG3D model on the Human Body dataset renderedfrom Renderpeople [2].
Details for the dataset are provided in the main paper and appendix.
We provide quantitative results in Table 5 and qualitative comparisons in Figure 14.
Comparedwith EG3D [8], we achieve comparable performance in generating 2D images (FID-ori), while wesignificantly improve on 3D shape synthesis in terms of FID-3D, which demonstrates the effectivenessof our model in learning actual 3D geometry and texture.