VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Anonymous Author(s)
AffiliationAddress email
Abstract
State-of-the-art 3D-aware generative models rely on coordinate-based MLPs toparameterize 3D radiance fields.
While demonstrating impressive results, queryingan MLP for every sample along each ray leads to slow rendering.
Therefore,existing approaches often render low-resolution feature maps and process themwith an upsampling network to obtain the final image.
Albeit efficient, neuralrendering often entangles viewpoint and content such that changing the camerapose results in unwanted changes of geometry or appearance.
Motivated by recentresults in voxel-based novel view synthesis, we investigate the utility of sparsevoxel grid representations for fast and 3D-consistent generative modeling in thispaper.
Our results demonstrate that monolithic MLPs can indeed be replaced by 3Dconvolutions when combining sparse voxel grids with progressive growing, freespace pruning and appropriate regularization.
To obtain a compact representation ofthe scene and allow for scaling to higher voxel resolutions, our model disentanglesthe foreground object (modeled in 3D) from the background (modeled in 2D).
Incontrast to existing approaches, our method requires only a single forward passto generate a full 3D scene.
It hence allows for efficient rendering from arbitraryviewpoints while yielding 3D consistent results with high visual fidelity.
Introduction
Generating photorealistic renderings of scenes at high resolution is a long-standing goal in computervision and graphics.
The primary paradigm is to carefully design 3D models, which are thenrendered using realistic camera and illumination models.
In recent years, the computer visioncommunity has made significant headway towards reducing these design efforts by approachingcontent generation from a data-centric perspective.
Generative Adversarial Networks (GANs) [9]have emerged as a powerful class of generative models for photorealistic high-resolution imagesynthesis [2,16,17,19,20,35,36].
One benefit of these 2D models is that they can be trained withlarge collections of images which are readily available.
However, scaling GANs to 3D is nontrivial because 3D supervision is difficult to obtain.
Recently, 3D-aware GANs have emerged toaddress the gap between handcrafted 3D models and image synthesis with 2D GANs which lack 3Dconstraints [4,13,23,29,30,37].
3D-aware GANs combine 3D generators, differentiable renderingand adversarial training to synthesize novel images with explicit control over the camera pose and,potentially, other scene properties like object shape and appearance.
Early 3D-aware GANs explored voxel-based 3D representations [13,29].
To compensate for the cubicmemory growth of voxel grids, HoloGAN [29] generates features on a small 3D grid and uses a neuralnetwork to map 3D features to a 2D image.
While such a neural renderer allows scaling to higher image resolutions, it may also entangle viewpoint and generated content [37].
Consequently, earlyvoxel-based approaches were either limited in image resolution or lacking 3D consistency.
Aboutthe same time, Neural Radiance Fields (NeRF) [27] emerged in the context of view synthesis as apowerful alternative 3D representation.
In their seminal work, Mildenhall et al.
[27] represent a sceneas a function of color and density, parameterized by a coordinate-based MLP.
The predicted color anddensity values are then projected to an image with differentiable volume rendering.
GRAF [38] adapts
NeRF’s coordinate-based representation to Generative Radiance Fields (GRAF) and proposes a 3Daware GAN using a coordinate-based MLP and volume rendering.
This propelled 3D-aware imagesynthesis to higher image resolutions while better preserving 3D consistency due to the physicallybased and parameter-free rendering.
These benefits led to the establishment of coordinate-based
MLPs as new de facto standard for 3D-aware image synthesis [4, 31].
While recent 3D-aware
GANs [4, 6, 10] have started to attain image fidelity and resolution similar to 2D GANs, trainingand inference is computationally expensive as the MLP must be queried at multiple points alongeach ray for volume rendering.
However, querying 3D space densely is prohibitively costly.
Forexample, rendering an image at resolution 256 2 using 48 sample points along each ray requires toquery the neural network 256 2 · 48 ≈ 3 M times.
As a result, most recent works combine neural andvolume rendering to ease computational cost at high resolutions [30].
Consequently, viewpoint and3D content are often entangled, such that changing the camera pose might result in unwanted changesof the geometry or the appearance of the 3D scene.
Further, for many downstream applications,e.g. integrating assets into physics engines, it is desirable to generate 3D content at high resolutiondirectly.
These shortcomings identify the need for a 3D representation that can be efficiently renderedat high resolution with a model that is 3D-consistent by design .
Recently, there has been significant progress towards accelerated training of novel view synthesismodels for single scenes by removing the MLP from the representation.
In particular, DVGO [40]and Plenoxels [1] directly optimize a sparse voxel grid for novel-view synthesis, demonstrating thatthe visual fidelity attained by NeRF [27] is not primarily attributed to its MLP-based representationbut rather to volumetric rendering and gradient-based optimization.
In addition to impressive imagequality, [1,40] obtain large rendering speedups due to their fast density and color queries.
Takinginspiration from these works, we revisit voxel-based representations for 3D-aware GANs [13,29]in the context of volumetric rendering.
To circumvent the cubic memory growth that limits earlyvoxel-based approaches [13,29], we explore sparse voxel grids for the generative settings, see Fig.
We observe that sparsity is key to enable scaling the 3D representation to higher resolution and tocombine it with volume rendering.
Specifically, we propose a 3D-aware GAN with a sparse voxel grid generator at its core.
As a result, our approach inherits fast rendering and trilinear interpolationwhile being 3D-consistent by design, separating it from other recent 3D-aware GANs [3,6,10] whichrequire a forward pass for every point along each camera ray of every view.
Another difference toexisting 3D-aware GANs is that sparsity is a built-in feature of our representation, mitigating theneed for exploiting sophisticated strategies to sample points along camera rays as in [4,33,43].
Ourfinal model achieves image fidelity similar to recent 3D-aware GANs leveraging neural renderingwhile generating high-resolution geometry and improving 3D consistency.
During inference, ourmodel only requires a single forward pass which takes up most of the inference time.
Once the 3Dscene is generated, images can be rendered within milliseconds while existing approaches requireanother forward pass which is two orders of magnitude slower.
We refer to our model as VoxGRAF .
Related Work
2D GANs.
Rapid progress on Generative Adversarial Networks [9] now enables photorealisticsynthesis up to megapixel resolution [2, 16, 18–20, 26, 36].
While the disentangled style-spaceof StyleGANs [18–20] allows for control over the viewpoint of the generated images to someextent [12, 24, 39, 48], gaining precise 3D-consistent control is still non-trivial due to its lack ofphysical interpretation and operation in 2D.
In contrast, in this work we aim for explicit control overthe camera pose by incorporating a 3D representation into the generator.3D-Aware GANs.
The first 3D-aware GANs, i.e. GANs that incorporate a 3D representation intothe generator model, were voxel-based approaches.
Dense grid-based approaches [13,50] are limitedto a lower grid resolution due to their cubic memory growth.
Other works combine lower-resolutiongrids with neural rendering [23,29] which scale to higher resolutions, but the generated images lack3D consistency [38].
Recently, GRAF [37] and π -GAN combine volume rendering and coordinatebased representations [27] allowing to scale 3D-aware GANs with physically inspired rendering tohigh resolutions.
However, dense ray marching remains computationally expensive and limits imagefidelity.
GIRAFFE [31] therefore proposes a hybrid rendering approach.
They render a low-resolutionfeature map with ray marching and use a neural renderer to decode it into a high-resolution image.
Due to efficiency, this approach has been widely adopted in subsequent works [3,10,15,32,44,47,49].
While some approaches try to counteract introduced inconsistencies, e.g. via dual discrimination [3]or a reconstruction loss [10], we instead propose a model that is 3D-consistent by design and cangenerate the 3D object at high-resolution.
As an alternative to hybrid rendering, GOF [43], ShadeGAN [33] and GRAM [6] focus on reducingthe number of query points for volume rendering.
While aforementioned methods aim for reducingthe number of sample points as querying a large MLP is computationally expensive, we instead use asparse voxel grid as 3D representation.
This allows us to speed up rendering without reducing thesample size as feature querying via trilinear interpolation is fast and can be efficiently implementedvia custom CUDA kernels.
Sparse 3D Representations.
As NeRF requires an optimization time in the order of multiple daysper scene, a series of follow-up works [25,34,41,46] propose techniques to speed up this process.
Therecent works Plenoxels [1] and DVGO [40] demonstrate that sparse voxel grid representations canachieve even faster convergence and higher rendering speed.
In addition to efficient rendering, sparsevoxel grids enable fast trilinear interpolation when queried beyond their grid resolution.
Buildingon this representation, our approach inherits these benefits.
We remark that very recently InstantNGP [28] achieves even faster rendering by combining small MLPs with a multi-resolution hashtable.
Exploring this representation might be an interesting avenue for future extensions of our work.
Note that all of these works focus on novel-view-synthesis for single scenes and require multi-viewimage supervision.
Instead, we propose a generative model that trains with raw image collections andthat can generate multiple novel instances at inference.
G
G of resolution R G .
Given camera intrinsics K and a camera pose ξ ′ , a foreground image is obtained using differentiable volume rendering [27].
The values at the sampling points along the camera rays are computed by trilinearly interpolating the voxel grid [1].
The background is generated by a 2D GAN G bgθ b and combined with the foreground using alpha composition.
The discriminator D ϕ compares the generated image ˆI to the real image I real .
Method
We first provide the necessary background by summarizing the currently dominating paradigm fordesigning 3D-aware GANs which combines an MLP scene representation and volume rendering asintroduced in GRAF [38].
Next, we introduce our sparse voxel-based scene representation whichboosts rendering speed while retaining 3D-consistency by design.
We refer to our model as VoxGRAF.
3.1 3D-aware GANs with Coordinate-Based Scene Representations
Recent 3D-aware GANs typically consist of a learned coordinate-based MLP as 3D generator, adeterministic volume rendering step potentially combined with a learned neural renderer in the 2Dimage domain, and a learned 2D discriminator.
Let G MLPψ denote the 3D generator parameterized bya coordinate-based MLP with learnable parameters ψ .
The 3D generator predicts a radiance fielddefined by color c ∈ [0 , 1] and density values σ ∈ R + .
The radiance field is defined at any 3Dpoint x ∈ R 3 and for any viewing direction d ∈ S 2 .
To model different 3D scenes, the generator isadditionally conditioned on an M -dimensional latent variable z ∈ N ( 0 , 1 ) .
The inputs x and d areprojected to higher-dimensional features γ ( x ) ∈
L x and γ ( d )
∈ L d with a fixed positional encodingto overcome the MLP’s smoothness bias and model high-frequency content γ ( · ) [27,42].
Formally,
[Equation]
In this paper, we challenge this paradigm and investigate a sparse 3D CNN instead of a coordinatebased MLP as generator, as described in the next section.
The radiance field is rendered by approximating the intractable volumetric projection integral vianumerical integration.
First, the generator is queried at N sampling points along each camera ray ryielding colors and densities { ( c ir , σ ir ) } Ni =1 .
For each camera ray r , these points are projected to an
RGB color value c r and optionally an alpha mask a r using alpha composition
[Equation]
where T ir and α ir denote the transmittance and alpha value of sample point i along ray r andδ ir =
(cid:13)(cid:13)
x i +1 r − x ir (cid:13)(cid:13) 2 is the distance between neighboring sample points.
As volume rendering hasproven a powerful tool for high-fidelity reconstruction, VoxGRAF retains this rendering mechanismbut reduces its computational cost by leveraging sparse scene representations. [Figure]
3.2 VoxGRAF: Generating Radiance Fields on Sparse Voxel Grids
[Figure]
Our goal is to design a 3D-aware GAN based on a sparse scene representation that allows for efficientrendering.
Fig.
2 shows an overview over our approach.
In contrast to recent works [3,6,43], we donot use a coordinate-based MLP to parameterize the radiance field.
Instead, inspired by recent workon novel view synthesis [1,40], we generate values on a sparse voxel grid using a 3D convolutionalneural network.
Therefore, our generator requires only a single forward pass to generate a 3D scene.
To disentangle 3D content from the background we combine a 3D foreground generator G fgθ f with a2D background generator G bgθ b .
G fgθ f takes a camera matrix K , camera pose ξ and a latent code zas input and predicts colors c and density values σ on a sparse voxel grid.
For unstructured images,camera matrix K and pose ξ can be determined e.g. with an off-the-shelf pose detector as done in [4].
Volume rendering yields a foreground image and an alpha mask.
The background generator maps thelatent code z to a background image which is then combined with the foreground image using alphacomposition.
We train our model in an adversarial setting using a 2D discriminator on the full image.
Foreground Generator.
Our foreground generator builds on the popular StyleGAN2 architecture [19] which achieves high fidelity in the 2D image domain.
Conditioned on a camera pose ξ ∈ R P ,it maps a latent code z to color values c and density values σ on a sparse voxel grid
[Equation]
where θ f and R G denote the learnable parameters and the resolution of the voxel grid, respectively.
Note that in contrast to most existing coordinate-based 3D-aware GANs, see Eq.
(1), we do notcondition the 3D generator on the view direction (per-ray).
Instead, we follow [3] and condition it onthe pose ξ (per-image) to model directional dependencies.
For rendering, we compute ( c i r , σ i r ) viatrilinear interpolation of densities and colors stored at the nearest eight vertices [1].
We use the samerendering formulation outlined in 3.1 and leverage custom CUDA kernels 2 for efficiency.
To generate voxel grids instead of images, we replace all 2D operations of the StyleGAN2 generatorwith their 3D equivalent, e.g., 3D modulated convolutions and 3D upsampling.
At resolutions beyond32 3 , we use sparse convolutions 3 [11] instead of dense ones as they are more computationally efficient.
At smaller resolutions, we observe no additional [3]efit and keep the dense repres ξ ntation.
To sparsifythe representation, we combine progressive growing [16] with pruning [1] as illustrated in Fig. 3a.
Specifically, we start with training a dense model at resolution 32 3 .
After sufficient training, we addthe first layer of sparse convolutions and prune its inputs based on the rendered view.
Intuitively, thenext layer should only operate on voxels visible in the rendered view to yield a sparse representation.
Consequently, we prune voxels that are either occluded or have a low density.
Following Eq.
(2), therendered alpha value is a r = (cid:80) Ni =1 T ir α ir .
Accordingly, occluded voxels (low transmittance T i ) and empty voxels (low density σ i ) do not contribute to the final image.
The pruning operator ρ discardsall voxels along a camera ray r for which transmittance T i or density σ i is smaller than threshold τ Tor τ σ , respectively.
Let V r denote the set of voxels intersecting with ray r , then
[Equation]
where V pr is the set of the retained voxels.
After pruning, we upsample the kept voxels via sparsetransposed convolutions.
After the newly-added layer is sufficiently trained, we repeat this processfor the next added stage.
Fig.
3b shows images and voxel grids at different resolutions.
We implementthis on-the-fly pruning operation using efficient custom CUDA kernels.
In agreement with [3], we observe that it is crucial to account for view dependent effects or posecorrelated attributes in the training data, like eyes always looking into the camera.
We take two measures to increase the flexibility of our model:
Following [3], G fgθ f is conditioned on the pose ξ whichcorresponds to the rendering pose ξ ′ in 50% of the cases and is randomly chosen otherwise.
Formally,ξ ∼ p ξ | p ( ξ = ξ ′ )
=0 .
At inference, the pose-conditioning is fixed to retain 3D consistency.
However,we find that pose conditioning alone is not always sufficient to account for strong correlations in thedata.
Depending on the dataset, we optionally refine the rendered image with a shallow 2D CNNwith 2 hidden layers of dimension 16 and kernel size 3 .
While this refinement is powerful enough tomodel dataset biases, it is considerably less flexible than the neural rendering used in [4,10,30]: Our2D CNN operates on the rendered image instead of rendered features at smaller resolution.
Operatingon 3 channels at full resolution allows to keep its capacity at a minimum and, due to not using anyupsampling operation, results in a local receptive field.
Background Generator.
We consider datasets with a single object per image.
Modeling only theobject in 3D saves computation and is advantageous for potential downstream tasks, e.g., integratinggenerated assets into new environments Hence, we model the object in 3D and generate the background of the image with a 2D GAN.
Specifically, we use the StyleGAN2 generator [19] with reduced channel size as modeling the background requires less capacity than generating the full image: [Equation]
[Equation]
We choose the same latent code z for foreground and background to allow for modeling correlationslike lighting between the two.
Note that, unlike the foreground generator, the background generatoris not conditioned on the camera pose.
As the background remains fixed when changing the camerapose, the generator is encouraged to model pose-dependent content with the foreground generatorleading to disentanglement (see supp.
mat.
for qualitative disentanglement results).
The final image is obtained using alpha composition
[Equation]
The full generator is defined as
[Equation]
Regularization.
For fast rendering, it is crucial that most generated voxels are either fully opaque orempty such that early stopping and empty space skipping are effective.
By regularizing the varianceof the expected depth ˆ z r along each ray r , the foreground generator is encouraged to generate a single,sharp surface:
[Equation]
where τ is a hyperparameter that defines the thickness of the surface.
We find that thresholding the lossis important to avoid an empty foreground.
In addition, we find that adding the grid TV regularization
L TV from [1] and fore- and background coverage regularization L fgcvg and L bgcvg from [45] furtherstabilizes training (see sup.
mat.
for details).
The full regularization term of our generator is
[Equation]
Discriminator.
We use the StyleGAN2 discriminator and condition it on the camera pose asproposed in [3].
Similarly, we find that conditioning guides the generator to learn correct 3D priorsand a canonical representation.
Since rendering our sparse representation is fast, our discriminator isable to operate on the full image and does not need to consider image patches as done in GRAF [38].
3.3 Training
Given images I from the data distribution p D with known camera extrinsics ξ I and intrinsics K I andlatent codes z ∈ N ( 0 , 1 ) , we train our model using a GAN objective with R1-regularization [26]
[Equation]
where f ( t ) =
− log(1 + exp( − t )) and λ controls the strength of the R1-regularizer.
G θ and D ϕ aretrained with alternating gradient descent combining the GAN objective with the regularization terms:
[Equation]
In practice, we optimize the generator with a non-saturating variant of Eq.
We train ourapproach with Adam [21] using a batch size of 64 at grid resolution R G = 32 , 64 and 32 at R G = 128 .
We use a learning rate of 0 .
0025 for the generator and 0 .
002 for the discriminator.
For faster training,we first grow R I from 32 to 128 while keeping R G at 32 .
Then, we alternately increase the gridresolution and the image resolution until the dataset resolution and R G = 128 are reached.
Forsynthetic datasets, i.e. Carla [38], we do not add any refinement layers.
Depending on the dataset, wetrain our models for 3 to 7 days on 8 Tesla V100 GPUs.
Details on the network architectures can befound in the supplemental material.
Results
Datasets.
We validate our approach on standard benchmark datasets for 3D-aware image synthesis.
The synthetic Carla dataset [8, 37] contains 10 k images and camera poses of 18 car models withrandomly sampled colors.
FFHQ [19] comprises 70 k aligned face images.
AFHQv2 Cats [5] consistsof 4834 cat faces.
Following [4], we estimate camera poses for both datasets with off-the-shelf poseestimators [7,22] and augment all datasets with horizontal flips.
Due to the limited number of imagesin AFHQv2 and Carla, we use adaptive discriminator augmentation [17] for these datasets.
Evaluation Metrics.
We measure image fidelity by calculating the Fréchet Inception Distance( FID ) [14] between 20 k generated images and the full dataset.
For all runtime comparisons, we reporttimes on a single Tesla V100 GPU with a batch size of 1.
4.1 Ablation Study
[Table]
We investigate the sparsity of the generated voxelgrids and validate the importance of the depthvariance loss, see Eq.
Sparsity is evaluatedby fusing the pruned voxel grids from 16 equallyspaced camera views and reporting the numberof empty voxels divided by the total number ofvoxels R 3 G .
Table 1 shows results averaged over100 instances.
The depth variance loss increasessparsity from 74% to 95% , lowering the memoryconsumption.
With the sparser representation, the times needed for scene generation t G fg + bg andrendering t π are reduced significantly.
4.2 Baseline ComparisonBaselines.
We group the baselines into two categories: (i) methods that render low-resolutionfeatures and use 2D-upsampling, i.e. a neural renderer, to obtain the final image, e.g. StyleNeRF [10],
[Figure]
A M
S t y l e N e R
F
[Figure]
and (ii) methods that render the 3D representation directly at the final image resolution, e.g. πGAN [4].
VoxGRAF falls into the second group.
We use the official code release for all methods.
Further, we reference numbers reported by concurrent works GRAM [6] and EG3D [3].
Qualitative Results
Fig.
4 shows samples from multiple views for StyleNeRF, GRAM and
VoxGRAF on FFHQ at resolution 256 2 .
StyleNeRF can generate additional faces or strands of hairacross views, as indicated by the red frames in Fig.
These inconsistencies under viewpoint changesare introduced by StyleNeRF’s powerful neural renderer.
GRAM achieves more consistent results,but its plane representation creates stripe artifacts for large viewpoint ranges.
Due to the shallow 2D
CNN, VoxGRAF can model the dataset bias of eyes looking into the camera
but otherwise achieveshigh 3D-consistency even under large viewing angles.
Additional samples of our method and thecorresponding sparse voxel grids for all datasets are provided in Fig.
1 and Fig.
Quantitative Results.
Table 2 reports FID on all datasets.
As expected, methods that use neuralrendering with upsampling, i.e., StyleNeRF and EG3D, perform best in terms of image fidelity.
Thisis expected as a neural renderer can add flexibility. But, as shown in Fig. 4, for StyleNeRF it reduces3D-consistency.
Among methods without a neural renderer, VoxGRAF significantly improves overπ -GAN and GOF and surpasses the current state-of-the-art approach GRAM.
Runtime Comparison.
Lastly, we compare the rendering times at inference for methods with andmethods without a neural renderer.
In contrast to all baselines, VoxGRAF requires only a singleforward pass to generate the scene, which can then be rendered from different viewpoints efficiently.
Note that at inference, voxels are pruned solely based on their density to amortize the renderingcosts per scene.
We find that this does not visibly affect the rendered images.
We report the timesfor generating the scene and rendering one view separately in Table 3.
One of the earliest works, GIRAFFE, is the fastest among all approaches as it renders a low-resolution feature volume and usescomparably small neural networks.
StyleNeRF significantly increases the neural renderer’s size toimprove image fidelity, which comes at the cost of speed compared to GIRAFFE. Yet, StyleNeRF is [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
applications.
the fastest approach for generating a single image among the best-performing methods.
However, apotential application of 3D-aware GANs is generating novel views of a single instance in real-time.
In this setting, at resolution 256 2 , VoxGRAF generates novel views at 167 FPS, whereas StyleNeRFruns at 20 FPS as its rendering costs are not amortized per scene.
Limitations and Discussion
In this work, we investigate sparse voxel grids as representation for 3D-aware image synthesis.
Wefind that the key to generating sparse voxel grids is to combine progressive growing, pruning, andregularization to encourage a sharp surface that can be rendered efficiently.
Our approach outperformsall methods that do not employ a neural renderer.
Instead of discarding neural rendering entirely,we find it advantageous to utilize a shallow CNN for refinement.
This CNN can model dataset biasbut is significantly weaker than standard neural rendering approaches that upsample low resolutionfeature maps.
Our approach can reduce the gap to models that build heavily on neural rendering, yeta trade-off between 3D-consistency and image fidelity remains.
Whether a certain amount of neural rendering is inherently needed to reach best performance is an important direction for future research.
Lastly, the speed of our method depends on the sparsity of the modeled scene.
Therefore, renderingtimes will likely increase on more complex datasets than those commonly used in literature.