Rethinking training of 3D GANs
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We are witnessing a surge of works on building and improving 3D-aware generators.
To induce a 3D-aware bias, such models rely on volumetric rendering, whichis expensive to employ at high resolutions.
The dominant strategy to addressthe scaling issue is to train a separate 2D decoder to upsample a low-resolutionvolumetrically rendered representation.
But this solution comes at a cost.
Not onlydoes it break multi-view consistency, e.g. shape and texture change when a cameramoves, but it also learns the geometry in a low ﬁdelity.
In this work, we take adifferent route to 3D synthesis and develop a non-upsampler-based generator withstate-of-the-art image quality, high-resolution geometry and which trains 2 .
5 ⇥faster .
For this, we revisit and improve patch-based optimization in two ways.
First, we design a location- and scale-aware discriminator by modulating its ﬁlterswith a hypernetwork.
Second, we modify the patch sampling strategy based on anannealed beta distribution to stabilize training and accelerate the convergence.
Wetrain on four datasets (two introduced in this work) at 256 2 and 512 2 resolutions,directly, without the need of a 2D upsampler, and our model attains better orcomparable FID and has higher ﬁdelity geometry than the current SotA.
Code/data/visualizations: https://rethinking-3d-gans.github.io
Introduction
Generative models for image synthesis achieved remarkable success in recent years and enjoy a lot ofpractical applications [55, 24].
While initially they mainly focused on 2D images [21, 65, 25, 4, 28],recent research explored generative frameworks with partial 3D control over the underlying objectin terms of texture/structure decomposition, novel view synthesis or lighting manipulation (e g.,[58, 56, 7, 67, 6, 12, 49]).
These techniques are typically built on top of the recently emerged neuralradiance ﬁelds (NeRF) [38] to explicitly represent the object (or its latent features) in 3D space.
NeRF is a powerful framework, which made it possible to built expressive 3D-aware generatorsfrom challenging RGB datasets [7, 12, 6].
Under the hood, it trains a multi-layer perceptron (MLP)
F ( x ; d ) =
( c ,  ) to represent a scene by encoding a density  2 R + for each coordinate positionx 2 R 3 and a color value c 2 R 3 from x and view direction d 2 S 2 [38].
To synthesize an image,one renders each pixel independently by casting a ray r ( q )
=
o
+ q d (for q 2 R + ) from origino 2 R 3 into the direction d 2 S 2 and aggregating many color values along it with their correspondingdensities.
Such a representation is very expressive, but comes at a cost: rendering a single pixel iscomputationally expensive and makes it intractable to produce a lot of pixels in one forward pass.
Itis not fatal for reconstruction tasks where the loss can be robustly computed on a subset of pixels,but it creates signiﬁcant scaling problems for generative NeRFs: they are typically formulated in a
GAN-based framework [14] with 2D convolutional discriminators requiring a full image as input.
People address these scaling issues of NeRF-based GANs in different ways, but the dominatingapproach is to train a separate 2D decoder to produce a high-resolution image from a low-resolution image or feature grid rendered from a NeRF backbone [43].
During the past six months, thereappeared more than a dozen of methods which follow this paradigm (e.g., [6, 15, 70, 47, 78, 35,74, 23, 71, 77, 63]).
While using the upsampler allows to scale the model to high resolution, itcomes with two severe limitations: 1) it breaks multi-view consistency of a generated object, i.e. itstexture and shape change when the camera moves; and 2) the geometry gets only represented in alow resolution ( ⇡ 64 3 ).
In our work, we show that by dropping the upsampler and using a simplepatch-wise optimization scheme, one can build a 3D generator with better image quality, fastertraining speed and without the above limitations.
Patch-wise training of NeRF-based GANs was originally proposed by GRAF [56] and got largelyneglected by the community since then.
The idea is simple: instead of training the generative model onfull-size images, one does this on small random crops.
Since the model is coordinate-based [59, 64],it does not face any issues to synthesize only a subset of pixels.
This serves as a good way to savecomputation for both the generator and the discriminator, since it makes both of them operate onpatches of small spatial resolution.
To make the generator learn both the texture and the structure,crops are sampled to be of variable scales (but having the same number of pixels): in some sense, thiscan be seen as optimizing the model on low-resolution images + high-resolution patches.
In our work, we improve patch-wise training in two crucial ways.
First, we redesign the discriminatorby making it better suited to operating on image patches of variable scales and locations.
Convolutional ﬁlters of a neural network learn to capture different patterns in their inputs depending on theirsemantic receptive ﬁelds [30, 46].
That’s why it is detrimental to reuse the same discriminator tojudge both high-resolution local and low-resolution global patches, inducing additional burden on it tomix ﬁlters’ responses of different scales.
To mitigate this, we propose to modulate the discriminator’sﬁlters with a hypernetwork [16], that predicts which ﬁlters to suppress or reinforce from a given patchscale and location.
Second, we change the random scale sampling strategy from an annealed uniform to an annealed betadistribution.
Typically, patch scales are sampled from a uniform distribution s ⇠ U [ s ( t ) , 1] [56, 36, 5],where the minimum scale s ( t ) is gradually decreased (i.e. annealed) till some iteration T froms (0) = 0 .
9 to a smaller value s ( T ) (in the interval [0 .
125  0 .
5] ) during training.
This samplingstrategy prevents learning high-frequency details early on in training and puts too little attention on thestructure after s ( t ) reached its ﬁnal value s ( T ) .
This makes the overall convergence of the generatorslower and less stable that’s why we propose to sample patch scales using the beta distribution
Beta (1 ,  ( t )) instead, where  ( t ) is gradually annealed from  (0) ⇡ 0 to some maximum value ( T ) .
In this way, the model starts learning high-frequency details immediately after the trainingstarts and has more focus on the structure after the growth is done.
This simple change stabilizes thetraining and allows to converge faster than the typically used uniform distribution [56, 5, 36].
We use those two ideas to develop a novel state-of-the-art 3D GAN: Patch-wise Generative Radiance Fields (PGRF).
We employ it for high-resolution 3D-aware image synthesis on four datasets:
F
M S
SynthBlock
Linear
Linear normalize w
Linear
Linear const
SynthBlock 
 SynthBlock
RGB  volumetric rendering z ⇠ N ( 0 , I )
P
D
H
DiscrBlock DiscrBlock
DiscrBlock
DiscrEpilogue real/fake
ModConv2d
ModConv2d
Conv2d downsample downsample
A  x  y s  x y
PosEnc
Linear
Linear s,  x ,  t ⇠ p t +
A
[Figure]
FFHQ [25], Cats [76], Megascans Plants and Megascans Food.
The last two benchmarks are introduced in our work and contain 360  renderings of photo-realistic scans of different plants and foodobjects (described in §4).
They are much more difﬁcult in terms of geometry and are well suited forassessing the structural limitations of modern 3D-aware generators.
Our model uses a pure NeRF-based backbone, that’s why it represents geometry in high resolutionand does not suffer from multi-view synthesis artifacts, as opposed to upsampler-based generators.
Moreover, it has higher or comparable image quality (as measured by FID [20]) and 2 .
5 ⇥ lowertraining cost.
Also, in contrast to upsampler-based 3D GANs, our generator can naturally incorporatethe techniques from the traditional NeRF literature.
To demonstrate this, we incorporate backgroundseparation into our framework by simply copy-pasting the corresponding code from NeRF++ [75].
Related work
Neural Radiance Fields .
Neural Radiance Fields (NeRF) is an emerging area [38], which combinesneural networks with volumetric rendering techniques to perform novel-view synthesis [38, 75, 2],image-to-scene generation [73], surface reconstruction [45, 68, 44] and other tasks [9, 17, 50].
In ourwork, we employ them in the context of 3D-aware generation from a dataset of RGB images [56, 7].3D generative models .
A popular way to learn a 3D generative model is to train it on 3D data or inan autoencoder’s latent space (e.g., [10, 69, 1, 34, 31, 39, 29]).
This requires explicit 3D supervisionand there appeared methods which train from RGB datasets with segmentation masks, keypointsor multiple object views [13, 32, 54].
Recently, there appeared works which train from single-view
RGB only, including mesh-generation methods [19, 72, 53] and methods that extract 3D structurefrom pretrained 2D GANs [58, 48].
And recent neural rendering advancements allowed to train
NeRF-based generators [56, 7, 42] from purely RGB data from scratch, which became the dominatingdirection since then and which are typically formulated in the GAN-based framework [14].
NeRF-based GANs .
HoloGAN [41] generates a 3D feature voxel grid which is projected on aplane and then upsampled.
GRAF [56] trains a noise-conditioned NeRF in an adversarial manner.
⇡ -GAN [7] builds upon it and uses progressive growing and hypernetwork-based [16] conditioningin the generator.
GRAM [12] builds on top of ⇡ -GAN and samples ray points on a set of learnableiso-surfaces.
GNeRF [36] adapts GRAF for learning a scene representation from RGB imageswithout known camera parameters.
GIRAFFE [43] uses a composite scene representation for bettercontrollability.
CAMPARI [42] learns a camera distribution and a background separation networkwith inverse sphere parametrization [75].
To mitigate the scaling issue of volumetric rendering, manyrecent works train a 2D decoder under different multi-view consistency regularizations to upsample alow-resolution volumetrically rendered feature grid [6, 15, 70, 47, 78, 71, 77].
However, none of suchregularizations can currently provide the multi-view consistency of pure-NeRF-based generators.
Patch-wise generative models .
Patch-wise training had been routinely utilized to learn the texturalcomponent of image distribution when the global structure is provided from segmentation masks,sketches, latents or other sources (e.g., [22, 57, 11, 66, 52, 51, 33, 61]).
Recently, there appearedworks which sample patches at variable scales, in which way a patch can carry global informationabout the whole image.
Recent works use it to train a generative NeRF [56], ﬁt a neural representationin an adversarial manner [36] or to train a 2D GAN on a dataset of variable resolution [5].
Model
We build upon StyleGAN2 [26], replacing its generator with the tri-plane-based NeRF model [6]and using its discriminator as the backbone.
We train the model on r ⇥ r patches (we use r = 64everywhere) of random scales instead of the full images of resolution R ⇥ R .
Scales s 2 [ rR , 1] arerandomly sampled from a time-varying distribution s ⇠ p t ( s ) .
3.1 3D generator
Compared to upsampler-based 3D GANs [15, 43, 71, 78, 6, 77], we use a pure NeRF [38] as ourgenerator G and utilize the tri-plane representation [6, 8] as the backbone.
It consists of threecomponents: 1) mapping network M : z 7!
w which transforms a noise vector z ⇠ R 512 intothe latent vector w ⇠ R 512 ; 2) synthesis network S : w 7!
P which takes the latent vector wand synthesizes three 32-dimensional feature planes P =
( P xy , P yz , P xz ) of resolution R p ⇥ R p(i.e.
P ( ⇤ ) 2 R R p ⇥ R p ⇥ 32 ); 3) tri-plane decoder network F : ( x , P ) 7!
( c ,  ) 2 R 4 , which takesthe space coordinate x 2 R 3 and tri-planes P as input and produces the RGB color c 2 R 3 anddensity value  2 R + at that point by interpolating the tri-plane features in the given coordinate andprocessing them with a tiny MLP.
In contrast to classical NeRF [38], we do not utilize view directionconditioning since it worsens multi-view consistency [7] in GANs which are trained on RGB datasetswith a single view per instance.
To render a single pixel, we follow the classical volumetric renderingpipeline with hierarchical sampling [38, 7], using 48 ray steps in coarse and 48 ones in ﬁne samplingstages.
See the accompanying source code for more details.
3.2 2D scale/location-aware discriminator
Our discriminator D is built on top of StyleGAN2 [26].
Since we train the model in a patch-wisefashion, the original backbone is not well suited for this: convolutional ﬁlters are forced to adaptto signals of very different scales and extracted from different locations.
A natural way to resolvethis problem is to use separate discriminators depending on the scale, but that strategy has threelimitations: 1) each particular discriminator receives less overall training signal (since the batch sizeis limited); 2) from an engineering perspective, it is more expensive to evaluate a convolutional kernelwith different parameters on different inputs; 3) one can use only a small ﬁxed amount of possiblepatch scales.
This is why we develop a novel hypernetwork-modulated [16] discriminator architectureto operate on patches with continuously varying scale.
M V
C
GA
N
S t y l e
N e R
F
G R
A M
O u r s
FFHQ
Cats
M-Plants
M-Food
[Figure]
To modulate the convolutional kernels of D we deﬁne a hypernetwork H : ( s,  x ,  y ) (  1 , ...  L )as a 2-layer MLP with tanh non-linearity at the end which takes patch scale s and its cropping offsets as and produces modulations  ` 2 ( c ` out ( the is channels
Given a convolutional kernel k and input x 2 R c a straightforward strategy to apply the modulation is to multiply  on the weights (depicting the convolution operation by conv2d ( . ) and omitting its other parameters for simplicity):
[Equation]
(1) where we broadcast the remaining axes and y 2 R c out is the layer output (before the non-linearity).
However, using different kernel weights on top of different inputs is not too efﬁcient in moderndeep learning frameworks (even with the group-wise convolution trick [26]), that’s why we use anequivalent strategy of multiplying the weights on x instead:
[Equation]
This suppresses and reinforces different convolutional ﬁlters of the layer depending on the patchscale and location.
And to incorporate an even stronger conditioning, we also use the projectionstrategy [40] from StyleGAN2-ADA [24].
We depict our discriminator architecture in Fig 2.
As weshow in Tab 2, it allows to obtain ⇡ 15 % lower FID compared to the standard discriminator.
3.3 Patch-wise optimization with Beta-distributed scales
Training NeRF-based GANs is computationally expensive, because rendering each pixel via volumetric rendering requires many evaluations (e.g., in our case, 96) of the underlying MLP.
Forscene reconstruction tasks, it does not create issues since the typically used L 2 loss [38, 75, 68]can be robustly computed on a sparse subset of the pixels.
But for NeRF-based GANs, it becomesprohibitively expensive for high resolutions since convolutional discriminators operate on densefull-size images.
The currently dominating approach to mitigate this is to train a separate 2D decoderto upsample a low-resolution image representation, rendered from a NeRF-based MLP.
But thisbreaks multi-view consistency (i.e. object’s shape and texture change when the camera is moving)and learns the 3D geometry in a low resolution (from ⇡ 16 2 [71] to ⇡ 128 2 [6]).
This is why we buildupon the multi-scale patch-wise training scheme [56] and demonstrate that it can give state-of-the-artimage quality and training speed without the above limitations.
Patch-wise optimization works the following way.
On each iteration, instead of passing the full-size
R ⇥ R image to D , we instead input only a small patch with resolution r ⇥ r of random scales 2 [ r/R, 1] and extracted with a random offset (  x ,  y ) 2 [0 , 1  s ] 2 .
We illustrate this procedure in
Fig 2.
Patch parameters are sampled from distribution:
[Equation]
(3) where t is the current training iteration.
In this way, patch scales depend on the current trainingiteration t and offsets are sampled independently after we know s .
As we show next, the choice ofdistribution p t ( s ) has crucial inﬂuence on the learning speed and stability.
Typically, patch scales are sampled from the annealed uniform distribution [56, 36, 5] s :
[Equation]
where lerp is the linear interpolation function 1 , and the left interval bound s min ( t ) is graduallyannealed during the ﬁrst T iterations until it reaches the minimum possible value of r/R .
But thisstrategy does not let the model learn high-frequency details early on in training and puts little focus onthe structure when s min ( t ) is fully annealed to r/R (which is usually very small, e.g. r/R = 0 .
125for a typical 64 2 patch-wise training on 512 2 resolution).
As we show, the ﬁrst issue makes thegenerator converge slower, and the second one makes the overall optimization less stable.
To mitigate this, we propose a small change in the pipeline by simply replacing the uniform scalesampling distribution with:
[Equation]
where  ( t ) is gradually annealed from  (0) to some ﬁnal value  ( T ) .
Using beta distribution insteadof the uniform one gives a very convenient knob to shift the training focus between large patch scaless !
1 (carrying the global information about the whole image) and small patch scales r ! r/R(representing high-resolution local crops).
A natural way to do the annealing is to anneal from 0 to 1 : at the start, the model focuses entirely onthe structure, while at the end it transforms into the uniform distribution (See Fig 3).
We follow thisstrategy, but from the design perspective instead set  ( T ) to a value, which is slightly smaller than1 (we use  ( T ) = 0 .
8 everywhere) to keep more focus on the structure at the end of the annealingas well.
In our initial experiments, we observed that  ( T ) 2 [0 . 7 , 1] perform similarly.
The scalesdistributions comparison between beta and uniform sampling is provided in Fig 3 and the convergencecomparison in Fig 6.
3.4 Training details
We inherit the training procedure from StyleGAN2-ADA [24] with minimal changes.
The optimization is performed by Adam [27] with learning rate of 0.002 and betas of 0 and 0.99 for both G and
D .
We use  ( T ) = 0 .
8 for T = 10000 , z ⇠ N ( 0 , I ) and set R p = 512 .
D is trained with R1regularization [37] with  = 0 .
We train with the overall batch size of 64 for ⇡ 15 M imagesseen by D for 256 2 resolution and ⇡ 20 M for 512 2 .
Similar to previous works [6, 12], we use pose
In practice, those methods use a very slightly different distribution (see Appx B)
[Table] [Figure] Ours supervision for D for the FFHQ and Cats dataset to avoid geometry ambiguity.
For this, we take therotation and elevation angles, encode them with positional embeddings [59, 64] and feed them into a2-layer MLP.
After that, we multiply the obtained vector with the last hidden representation in thediscriminator, following the Projection GAN [40] strategy from StyleGAN2-ADA [24].
We train G in full precision and use mixed precision for D .
Further details can be found in the source code.
Experiments
4.1 Experimental setup
Benchmarks .
In our study, we consider four benchmarks: 1) FFHQ [25] in 256 2 and 512 2 resolutions,consisting of 70,000 (mostly front-view) human face images; 2) Cats 256 2 [76], consisting of 9,998(mostly front-view) cat face images; 3) Megascans Food (M-Food) 256 2 consisting of 199 modelsof different food items with 128 views per model (25472 images in total); and 4) Megascans Plants(M-Plants) 256 2 consisting of 1108 different plant models with 128 views per model (141824 images in total).
The last two datasets are introduced in our work to ﬁx two issues with the modern 3Dgeneration benchmarks.
First, existing benchmarks have low variability of global object geometry,focusing entirely on a single class of objects, like human/cat faces or cars, that do not vary much frominstance to instance.
Second, they all have limited camera pose distribution: for example, FFHQ [25]and Cats [76] are completely dominated by the frontal and near-frontal views (see Appx D).
That’swhy we obtain and render 1307 Megascans models from Quixel, which are photo-realistic (barelydistinguishable from real) scans of real-life objects with complex geometry.
Those benchmarks,together with the rendering code, will be made publicly available.
Metrics .
We use FID [20] to measure image quality and also estimate the training cost for eachmethod in terms of NVidia V100 GPU days needed for it to complete the training process.
Baselines .
For upsampler-based baselines, we compare to the following generators: StyleNeRF
StyleSDF [47], EG3D [6], VolumeGAN [70], MVCGAN [77] and GIRAFFE-HD [71].
Apart fromthat, we also compare to pi-GAN [7] and GRAM [12], which are non-upsampler-based GANs.
Tocompare on Megascans, we train StyleNeRF, MVCGAN, pi-GAN, and GRAM from scratch usingtheir ofﬁcial code repositories (obtained online or requested from the authors), using their FFHQ or
CARLA hyperparameters, except for the camera distribution and rendering settings.
We also train
StyleNeRF, MVCGAN and ⇡ -GAN on Cats 256 2 .
GRAM [12] restricts the sampling space to aset of learnable iso-surfaces which makes it not well-suited for datasets with varying geometry.
Tooptimize its hyperparameters for Megascans, we varied the number of surfaces (24 and 48), ﬁeld ofview (30 and 12) and batch size (16 and 32), but it was diverging for us in each case.
4.2 Results
PGRF achieves state-of-the-art image quality .
For Cats 256 2 , M-Plants 256 2 and M-Food 256 2 ,
PGRF outperforms all the baselines in terms of FID except for StyleNeRF, performing very similarto it on all the datasets despite the fact that it does not have a 2D upsampler.
For FFHQ, our modelattains very similar FID scores as the other methods, ranking 4/9 (including older ⇡ -GAN [7]),noticeably losing only to EG3D [6], which trains and evaluates on a different version of FFHQ anduses pose conditioning in the generator (which potentially improves FID at the cost of multi-viewconsistency).
We provide a visual comparison for different methods in Fig 4.
PGRF is much faster to train .
As reported in Tab 1, existing methods typically train for ⇡ 1 week on8 V100s, PGRF ﬁnishes training in just 2 days for 256 2 and 3 days for 512 2 resolutions, which is2  3 ⇥ faster.
Note that this high training efﬁciency is achieved without the use of an upsampler,which initially enabled high-resolution synthesis of 3D-aware GANs.
As to the non-upsamplermethods, we couldn’t train GRAM or ⇡ -GAN on 512 2 resolution due to the memory limitations ofthe setup with 8 NVidia V100 32GB GPUs (i.e., 256GB of GPU memory in total).
PGRF learns high-ﬁdelity geometry .
Using a pure NeRF-based backbone carries two crucial beneﬁts:it provides multi-view consistency and allows to learn the geometry in the full dataset resolution.
In
Fig 5, we visualize the learned shapes on M-Food and M-Plants for 1) ⇡
-GAN: a pure NeRF-basedgenerator without the geometry constraints; 2) MVC-GAN [77]: an upsampler-based generator withstrong multi-view consistency regularization; 3) our model.
We provide the details and analysis inthe caption of Fig 5.
PGRF easily capitalizes on techniques from the NeRF literature .
Since our generator is purely NeRFbased and renders images without a 2D upsampler, it is well coupled with the existing techniques fromthe NeRF scene reconstruction ﬁeld.
To demonstrate this, we adopted background separation from
NeRF++ [75] using the inverse sphere parametrization by simply copy-pasting the correspondingcode from their repo.
We depict the results in Fig 1 and provide the details in Appx B.
4.3 Ablations
We report the ablations for different discriminator architectures and patch sizes on FFHQ 512 2and M-Plants 256 2 in Tab 2.
We start with the traditional GRAF generator (in our tri-plane-basedbackbone and use T = 5000 as its annealing schedule since this value showed the best convergenceon Cats 256 2 .)
We then train it with our proposed beta scale sampling, which results in ⇡ 13% better
FID. Using several discriminators (via the group-wise convolution trick [26]) results in noticeablyslower training time and degrades the image quality a lot.
We hypothesize that the reason of it was the reduced overall training signal which each discriminator receives, which we tried to alleviate byincreasing the learning rate for them, but that did not improve the results.
A too small patch sizehampers the learning process and results in a ⇡ 80 % worse FID.
A too large one provides decent imagequality, but greatly reduces the training speed.
Using a single scale/position-aware discriminatorachieves the best performance, outperforming the standard one by ⇡ 15% on average.
To assess the convergence of our proposed patch sampling scheme, we compared against uniformsampling on Cats 256 2 for T 2 { 1000 , 5000 , 10000 } , representing different annealing speeds.
Weshow the results for it in Fig 6: our proposed beta scale sampling strategy with T = 10 k schedulerobustly converges to lower values than the uniform one with T = 5 k or T = 10 k
and does notﬂuctuate much compared to the T = 1 k uniform one (where the model reached its ﬁnal annealingstage in just 1k kilo-images seen by D ).
To analyze how hyper-modulation manipulates the convolutional ﬁlters of the discriminator, wevisualize the modulation weights  , predicted by H , in Fig 7 (see the caption for the details).
Thesevisualizations show that some of the ﬁlters are always switched on, regardless of the patch scale;while others are always switched off providing potential room for pruning [18].
And ⇡ 40 % of theﬁlters are getting switched on and off depending on the patch scale, which shows that H indeed learnsto perform meaningful modulation.
[Figure]
[Table]
Conclusion
In this work, we showed that it is possible to build a state-of-the-art 3D GAN framework without a2D upsampler, but using a pure NeRF-based generator trained in a multi-scale patch-wise fashion.
For this, we improved the traditional patch-wise training scheme in two important ways.
First,we proposed to use a scale/location-aware discriminator with convolutional ﬁlters modulated by ahypernetwork depending on the patch parameters.
Second, we developed a schedule for patch scalesampling based on the beta distribution, that leads to faster and more robust convergence.
We believethat the future of 3D GANs is a combination of efﬁcient volumetric representations, regularized 2Dupsamplers, and patch-wise training.
We propose this avenue of research for future work.
Our method also has several limitations.
Before switching to training 3D-aware generators, wespent a considerable amount of time exploring our ideas on top of StyleGAN2 for traditional 2Dgeneration.
This always resulted in increased FID scores (see Appx A).
Further, the discriminatorlooses information about global context.
We tried multiple ideas to incorporate global context, but itdid not lead to an improvement.
Finally, 3D GANs generating faces and humans may have negativesocietal impact as discussed in Appx G.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See §5 and Appx A.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We dothis in Appendix F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We discuss the potential ethical concerns of using our model in
Appendix F.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes] We provide
[Table] the code/data and additional visualizations on https://rethinking-3d-gans.github.io(as [Table] speciﬁed in the introduction).
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide the most important training details in §3.4.
The rest
[Table]
of the details are provided in Appx B and the provided source code.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
[Table]
iments multiple times)?
[No] .
That’s too computationally expensive and single-run
results are typically reliable in the GAN ﬁeld.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We report this numbers in Appx B.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all the
sources of the datasets which were used or mentioned in our submission.
(b) Did you mention the license of the assets?
[Yes] In this work, we release two new
datasets: Megascans Plants and Megascans Food.
We discuss their licensing in Appx D.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We provide our datasets on the project website: https://rethinking-3d-gans.github.io.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We specify the information on dataset collection in Appx D.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] As discussed in Appx D, the released data
does not contain personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]