Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Anonymous Author(s)
AffiliationAddress email
Abstract 3D-aware image synthesis aims at learning a generative model that can renderphoto-realistic 2D images while capturing decent underlying 3D shapes.
A popularsolution is to adopt the generative adversarial network (GAN) and replace thegenerator with a 3D renderer, where volume rendering with neural radiance field(NeRF) is commonly used.
Despite the advancement of synthesis quality, existingmethods fail to obtain moderate 3D shapes.
We argue that, considering the twoplayer game in the formulation of GANs, only making the generator 3D-awareis not enough .
In other words, displacing the generative mechanism only offersthe capability, but not the guarantee, of producing 3D-aware images, because thesupervision of the generator primarily comes from the discriminator.
To addressthis issue, we propose GeoD through learning a geometry-aware discriminator toimprove 3D-aware GANs.
Concretely, besides differentiating real and fake samplesfrom the 2D image space, the discriminator is additionally asked to derive thegeometry information from the inputs, which is then applied as the guidance of thegenerator.
Such a simple yet effective design facilitates learning substantially moreaccurate 3D shapes.
Extensive experiments on various generator architectures andtraining datasets verify the superiority of GeoD over state-of-the-art alternatives.
Moreover, our approach is registered as a general framework such that a morecapable discriminator ( i.e. , with a third task of novel view synthesis beyond domainclassification and geometry extraction) can further assist the generator with a bettermulti-view consistency.
Code will be made publicly available.
Introduction
3D-aware image synthesis has received growing attention due to its potential in modeling the 3Dvisual world.
Compared to 2D generation [1, 13–16, 28], which primarily focuses on the image qualityand diversity, 3D-aware generation is expected to also learn an accurate 3D shape underlying thesynthesized image.
However, existing generative models, such as the popular generative adversarialnetwork (GAN) [8] fail to capture the 3D information from 2D images, as they produce an imagebased on 2D representations only.
To bridge this gap, recent studies [2, 3, 5, 9, 23–25, 29, 30, 34, 36]seek help from 3D rendering techniques, like neural radiance field (NeRF) [19], and propose toreplace the generator in a GAN with a 3D renderer.
Such a replacement enables explicit control ofviewpoints when generating an image, benefiting from the 3D awareness of the generator.
Many attempts have been made to improve 3D-aware GANs from the generator side, with either betterrepresentations [3, 5, 24, 25, 30, 34] or novel architectures [2, 9, 23, 29, 36].
Despite the advancementof 2D synthesis quality, they still suffer from unsatisfactory 3D shapes, which appear as flatten, noisy,or uneven surfaces as shown in Fig.
1a.
We argue that, given the generator-discriminator competition in training GANs, only making the generator 3D-aware is not enough .
Specifically, the generator
GeoD
R/F (b) G-NeRF + GeoD
𝑧 𝜉
G-NeRF
VR 𝑧 𝜉
G-NeRF
VR
D
R/F (a) G-NeRF
+ D
Geometry Geometry
[Figure]
Fake
[Figure]
Real image
Depth Normal ...
Mesh
Mesh obtains supervision ( i.e. , gradients) mostly from the discriminator, however, the discriminator remainsdifferentiating real and fake images from the 2D image space, as shown in Fig.
1a.
Under such a case,even though the generator is potentially capable of learning 3D shapes, it is still exposed to the risk ofshape-appearance ambiguity ( i.e. , well rendered 2D appearance does not necessarily imply a decent3D shape) because of the lack of adequate 3D guidance.
To tackle this problem, we propose a geometry-aware discriminator, termed as GeoD , which advances3D-aware GANs from the discriminator side.
As shown in Fig.
1b, besides the basic task of real/fakedomain classification, we assign the discriminator an additional task of geometry extraction.
Forone thing, the classification branch remains the same as that in conventional GANs, and helps thegenerator improve the realness of the syntheses from the 2D space.
For another, the geometry branchtargets deriving the shape-related information, such as depth and normal, from a given image.
Suchinformation is employed as an extra signal to supervise the generator for a fair 3D shape.
Here, thegeometry branch is unsupervisedly learned on real data only and jointly updated with the generator,forming a co-evolution manner.
We evaluate our approach on various 3D-aware GAN variants, including π -GAN [2], StyleNeRF
[9],and VolumeGAN [34], to verify its generalization ability.
Both qualitative and quantitative results ona range of datasets, including FFHQ (human faces) [14], AFHQ cat (animal faces) [4], and LSUNbedroom (indoor scenes) [35], validate that GeoD outperforms the baselines with a far more accurate3D shape without sacrificing the image quality.
Furthermore, we show that our framework is generalsuch that it can be applied for other 3D-related improvements.
In particular, when assigned with athird task of novel view synthesis, the discriminator can urge the generator to learn better multi-viewconsistency, demonstrating our motivation of making the discriminator 3D-aware .
Related work
3D-aware image synthesis.
3D-aware image synthesis has received lots of attention recently.
Thekey difference from 2D GANs is the use of 3D representation in the generator.
VON [37] and
HoloGAN [22] use the voxelized 3D representations to perform 3D-aware image synthesis, but thesemethods suffer from the 3D inconsistency due to the lack of underlying geometry.
[2, 5, 24, 29, 30]introduce NeRF [19] to synthesize 3D-aware images, and [3, 9, 23, 34, 36] propose to render lowresolution feature maps with NeRF and then leverage powerful 2D CNNs to generate high-fidelityimages.
The underlying shapes of these approaches, however, are noisy and imprecise.
To tacklethe inaccurate shape, ShadeGAN [25] proposes to introduce the albedo field instead of the radiancefield, and adds a lighting constraint by explicitly modeling the illumination process.
However, thediscriminators of these approaches still distinguish images from 2D space, which is risky to bringthe shape-appearance ambiguity due to the lack of 3D supervision.
Our main goal is to improve3D geometry from the discriminator’s perspective by introducing geometry-aware discrimination tosupervise the 3D-aware generator.
It is worth noting that our geometry-aware discriminator can beeasily applied to the approaches built
Geometry extraction.
Extracting geometries from images has been a long-standing problem.
As inmost cases only monocular data is available to train a generative model, we only discuss geometryextraction from monocular images here.
A traditional way is to train a network with multi-viewdata or labeled geometries in a supervised manner, such as depth estimation [7, 26, 27] and inverserendering [11, 17, 18, 32].
By introducing differentiable renderers, recent work [33] starts extractinggeometries from monocular images unsupervisedly.
Concretely, the network first decomposes animage into several geometry factors ( e.g. , albedo, depth, and lighting), and a differentiable rendererthen reconstructs the image following physical models.
However, such methods require assumptions,such as symmetry and shading, and thus they are only applicable on object images.
All these tasksand models can be integrated into the discriminator for geometry extraction.
Method3.1 Preliminaries
Generative neural radiance field.
The neural radiance field (NeRF) [19] adopts a continuousfunction, F ( x , d ) =
( c , σ ) , to learn the RGB color c ∈ R and volume density σ ∈ R given the3D coordinate x ∈ R 3 and the viewing direction d ∈ S 2 .
F ( · , · ) is typically parameterized witha multi-layer perceptron (MLP).
To render an image under the given camera pose, the pixel coloris obtained via volume rendering along the corresponding ray with near and far bounds.
Owingto NeRF’s strong performance in 3D reconstruction and novel view synthesis,
recent attempts
on3D-aware image synthesis propose to introduce the generative neural radiance field (G-NeRF) [29],
G ( x , d , z ) =
( c , σ ) , to the generator.
It improves the 3D consistency of synthesized images acrossdifferent camera views, due to the underlying geometry encoded in G-NeRF.
Geometry extraction from monocular images.
Geometry extraction from single view attemptsto predict 3D information, such as depth, normal, and albedo, from monocular photographs.
Unsupervised extraction is made possible by differentiable rendering, which enables the gradientcalculation of 3D objects through images during the rendering process.
It can be formulated intoan autoencoding process based on differentiable rendering to learn the geometry factors in anunsupervised manner.
The geometry encoder Φ( · ) maps the image into geometry factors, e.g., depth,normal, albedo, and lighting.
The decomposed factors are then used to reconstruct the input imagethrough a differentiable renderer.
3.2 Geometry-aware discrimination
Recall that we introduce a geometry branch to the discriminator to provide 3D supervision signalsfor the generator.
In this part, we present how the geometry branch is incorporated to build ageometry-aware discriminator for improving 3D-aware image synthesis.
Learning a geometry-aware discriminator with real images.
Recent attempts on 3D-aware GANsinherit a 2D discriminator which only distinguishes images in 2D space, suffering from the limited3D supervision for the generator.
To make the maximum use of the 3D information encoded in the2D images and prevent the discriminator from distinguishing images without awareness of geometry,we assign an geometry extraction task beyond bi-class domain classification, which is to derive thegeometry information from the given real image I r .
To achieve this, we introduce a geometry branchΦ( · ) and supervise the discriminator with an extra autoencoding training objective:
[Equation]
where Ψ( · ) represents a differentiable renderer, and l r is a function that measures image distance.
Performing geometry-aware discrimination on fake images.
As discussed above, the 3D-awaregenerator built upon G-NeRF [2, 3, 5, 9, 23–25, 29, 30, 34, 36] can synthesize consistent 3D-awareimages owing to the underlying geometry encoded in G-NeRF rather than the supervision signal fromthe discriminator.
In such a case, the underlying geometry cannot be accurate despite the good qualityof the synthesized 2D images.
To alleviate the issue of the imperfect shape encoded in G-NeRF, weleverage the intrinsic geometry characteristics of a single image and give geometry-aware supervisionto the generator.
Concretely, we first ask the discriminator to factorize the geometry information of synthesized images.
And then the geometry k extracted by Φ( · ) is regarded as a pseudo label tosupervise the corresponding geometry ˆ k encoded in G-NeRF in a loop-back manner.
The imagegeneration process is formulated as I f
= G ( z , ξ ) , where z and ξ are the sampled latent code andcamera pose.
An extra training objective is included to perform geometry-aware discrimination:
[Equation]
) where I f is the fake image synthesized by the generator, and l f denotes a loss function that measuresthe difference between the geometries.
G k represents geometry extraction from G-NeRF.
Loss functions.
To summarize, with the purpose of making the discriminator geometry-aware, thegenerator and the discriminator are jointly trained with:
[Equation]
[Equation]
where f ( t ) =
− log(1 + exp( − t ))
[20] is the negated logistic loss, and the third term in Eq.
(7) isthe gradient penalty.
λ , λ fgeo and λ rgeo represent weights for different loss terms.
θ G and θ D denotethe parameters of the generator and GeoD, respectively.
3.3 Implementation details
Geometry branch in GeoD.
Considering the complexity of contents in the image, we employdifferent architectures of geometry branch for objects and scenes.
The geometry branch for objects isbuilt upon Unsup3d [33] and trained unsupervisedly, with the assumption that objects are symmetric.
In this branch, the encoder decomposes the input image I into six factors Φ( I ) =
( d , n , a , l , v , c ) ,including a depth map d , a normal map n , an albedo map a , a global lighting direction l , a viewpointv , and confidence maps c .
All factors except the viewpoint are estimated under the canonical view.
During the reconstruction process, an image I c under the canonical view will first be reconstructedfrom d , n , a , and l using Lambertian shading model.
Then, a reprojection operator warps I c fromthe canonical view to viewpoint v given the depth map d , which gives us a reconstructed image
ˆ I .
Considering the symmetry assumption, a second reconstructed image ˆ I ′ is obtained from flippeddepth and albedo.
To weaken the influence of asymmetric parts, confidence maps, which indicate thepixel-wise symmetry score, are used as the weighting factors of pixel-wise reconstruction loss.
Thereconstruction loss for geometry branch of objects on real data is thus given by:
[Equation]
where l 1 and l p denote the pixel-wise l 1 loss and the perceptual loss [12], respectively.
λ flip representsthe weighting factor of loss term.
The geometry ( e.g. , normal) used for generator supervision isobtained by warping the geometry under the canonical view to the one under viewpoint v .
Geometry branch for scenes is built on Li et al.’s
The network is constructed in a cascademanner with two stages.
Each stage contains two encoders.
One encoder will output albedo a , normaln , roughness r and depth d , while the other one aims at predicting spatially-varying lighting l .
Thefollowing differentiable rendering layer then renders the diffuse image I d and the specular image I sbased on these outputs.
a , n , r , d , I d , I s along with the original image will become the inputs ofthe encoders in the next stage.
Because of the large diversity of geometries, materials, and lighting,ground-truth data for each decomposed component is available for training.
Therefore, the trainingloss for geometry branch of scenes is the sum of l 2 loss between each predicted factor q and itsground-truth data ˜ q :
[Equation]
where l 2 and λ q denote the pixel-wise l 2 loss and the weighting factor for each loss term.
Geometry extraction from G-NeRF.
As discussed above, the underlying geometry encoding in
G-NeRF can be leveraged as an implicit constraint for synthesizing 3D-aware images.
Our approachaims at supervising the underlying geometry by GeoD and the normal is chosen as the geometryrepresentation to pass the 3D information during the discrimination process.
We first obtain the depthmap d regarding different camera rays by performing volume rendering [19] on the depth-axis:
d
[Equation]
r
[Equation]
=
N (cid:88)
i
[Equation] [Equation]
11) where δ i =
||
x i +1 − x i || 2 is the distance between adjacent sampled points, and z [ x i ] denotes thedepth value of each point x i .
Then the tangent map, t , is derived from the depth map along u, vdirections [33].
Here, we take the u direction as an example:
[Equation]
where p x,y =
( x, y, 1) T is the homogeneous coordinate of image pixel ( x, y ) , and K is the cameraintrinsic matrix.
And then the normal map ˆ n can be obtained by performing the outer product on thetangent map ˆ n = t uu,v ⊗ t vu,v , followed by the point-wise normalization operation.
Experiments
4.1 Settings
Datasets.
We evaluate the proposed GeoD on three real-world unstructured datasets, including
FFHQ [14], AFHQ cat [4], and LSUN bedroom [35].
FFHQ contains unique 70K high-resolutionreal images of human faces.
All images are aligned and cropped following [14].
AFHQ cat includesaround 5K images of various cat faces in different poses.
We use the cat face detector to get thelandmarks and align images following [14].
Images are then cropped to keep the face in the center.
There are about 3M images in the LSUN bedroom.
The images are captured in various camera views.
We use center-cropping to preprocess the images.
Baselines.
We compare against three state-of-the-art methods in 3D-aware image synthesis, includingπ -GAN [2], StyleNeRF
[9] and VolumeGAN [34].
Baselines are either released by the authors ortrained with official implementations.
Training.
We inherit the generators from the baselines and build GeoD on baselines’ discriminator.
We follow the training protocol of the baselines.
For human and cat faces, GeoD is trained fromscratch along with the original GAN pipeline.
As a result of the complexity of indoor scenes, it isdifficult to learn reasonable geometries from monocular images only.
Therefore, for GeoD of scenes,we pretrain the geometry branch on synthetic data [18] and NYU dataset [21] following Li et al.
The resolution of the geometry branch in GeoD is 64 × 64 for faces and 256 × 256 for scenes.
Imagesare resized to satisfy this requirement.
The entire training ensures the discriminator to see 25000Kreal images.
Due to the expensive rendering process, π -GAN is trained with images of resolution64 × 64, and VolumeGAN on LSUN bedroom is trained on 128 × 128 images.
Other experiments areconducted on the resolution of 256 × 256.
More details are available in Supplementary Material .
Evaluation metrics.
We use three metrics to evaluate the performance, including Fréchet Inception
Distance (FID) [10], Scale-Invariant Depth Error (SIDE) [6], and Reprojection Error (RE) [34].
FID is calculated on 50K real images and 50K fake images rendered with random latent codesand camera views, reflecting the quality and diversity of the generated images.
SIDE evaluates thegenerated geometries’ accuracy on 10K images by comparing the ground-truth depth and the depthmap rendered from the generator.
We adopt the pretrained models, LeMul [11] for faces and DPT [27]for scenes, as the depth estimators to extract the depth map from 2D images as ground truths.
Sincethe resolution of NeRF is different from the image resolution in StyleNeRF and VolumeGAN, weresize the ground-truth depth to NeRF
’s resolution for comparison.
RE evaluates the consistency ofsynthesized images on 10K pairs.
We randomly sample two views, and render the image and its depthmap under each view given the same latent code.
The depth map will be resized to image size if theresolutions of two do not match.
Then, we warp the image from one view to the other view using itsdepth, and calculate the error between the warped image and the image rendered under that view.
[Figure]
[Table]
4.2 Main results
2D image quality and 3D shape.
Tab. 1 reports the FID scores evaluated on different models anddatasets.
With our GeoD, the FID scores are better than or on par with the baselines’ in most cases,indicating better image quality and diversity.
This is also reflected in the qualitative results in Figs.
2to 4.
For StyleNeRF on LSUN bedroom and VolumeGAN on AFHQ cat, the FID scores are slightlyworse than the baselines’.
We conjecture the reason is that these baselines tend to learn a flat shapeas shown in Figs.
3 and 4, and thus they reduce the 3D-aware image generation problem to a 2Dimage generation task, leading to lower FID scores.
Our method, however, keeps the image qualitycomparable with or even better than the baselines while producing correct underlying geometries.
Analysis on geometries.
We analyze the underlying geometry of the generated images from twoperspectives, i.e. , the accuracy of the geometry and the consistency across different views, whichare reflected by SIDE and RE in Tab. 1.
Here, for StyleNeRF [9], SIDE and RE are evaluated onlyon the foreground objects of FFHQ and AFHQ cat.
Lower SIDE and RE values on all datasetsand architectures indicate the achievement of better geometries with GeoD. This is also reflected
S t y l e
N e R F
S t y l e
N e R F + G e o
D
FFHQ
AFHQ Cat
LSUN Bedroom in the qualitative comparisons in Figs.
2 to 4.
For each sample, we uniformly sample three viewsand show the normal map and the mesh at each view.
π -GAN generates noisy shapes on threedatasets.
With our GeoD, the pits are all gone and the generated shapes are more vivid, resultingin better accuracy and consistency scores.
StyleNeRF and VolumeGAN apply NeRF in a lowerresolution than the image resolution, and tend to generate flat shapes as shown in Figs.
3 and 4.
Especially on AFHQ cat and LSUN bedroom datasets, their generated geometries are nearly orcompletely flat planes.
By using GeoD as the discriminator, the resulting geometry is much morethree-dimensional.
VolumeGAN combines the volume representation and neural radiance fieldsfor rendering, and therefore the generated shape is finer than that of StyleNeRF.
Fig.
5 visualizesthe geometries extracted from the 3D-aware generator and GeoD. GeoD successfully extracts vividgeometries and thus is able to guide the generator to produce fine geometries.
4.3 Ablation study
[Table] [Table]
Fine
[Table]
We conduct ablation studies to analyze theeffectiveness of GeoD under three settings:trained from scratch, pretrained, and finetuned.
π -GAN is chosen as the backboneand all experiments are done on FFHQ64 × 64.
In the first setting, GeoD is trainedtogether with the generator from scratch.
In the second setting, we first pretrain thegeometry branch of GeoD on FFHQ datasetuntil it converges.
Then, the geometry branch of GeoD will be freezed during the entire trainingof GAN.
In the third setting, the geometry branch of GeoD will first be trained offline on FFHQdataset for half of the iterations that it converges.
After that, the weights will be used to initialize thegeometry branch of GeoD in the training of the whole framework.
Results are presented in Tab. 2.
Pretrained geometry branch in GeoD provides better supervision of geometry at the beginning, hencethe accuracy of the geometry is better.
However, too strong geometry supervision from the startmay dominate the supervision signal and thus leads to a sub-optimal solution for RGB synthesis.
Therefore, the setting that GeoD is trained from scratch achieves better image quality and consistency.
V o l u m e G A
N
V o l u m e G A
N + G e o D
FFHQ
AFHQ Cat consistency.
For this purpose, we equip the discriminator with a third branch, learned with the task of novel view synthesis .
Such a design improves the multi-view consistency (RE) without sacrificing 2D image quality (FID) and 3D shape (SIDE).
[Table]
Besides the underlying geometry,multi-view consistency is alsoimportant for 3D-aware imagesynthesis.
In this section, weshow that GeoD is a generalframework such that it is alsoapplicable to improve the multiview consistency with a thirdnovel view synthesis task.
Specially, we ask the generator tosynthesize N multi-view images{ I i } N − 1 i =0 , where { I i } N
− 2 i = aretreated as the source images.
These source images are thenused to reconstruct an image,
I recN − 1 , which shares the sameview as I N − 1 .
The differencebetween I recN −
and I N − 1 reflects 267 the degree of consistency and in return guides the generator to be more consistent across differentviews.
We instantiate the consistency branch with IBRNet [31], a 3D reconstruction model pretrainedon multiple scene reconstruction datasets.
In the experiment, we set N to be 6 and work on LSUNbedroom dataset.
We choose StyleNeRF and VolumeGAN as baselines, because they face more severeinconsistency issues as convolutional layers and upsampling operators are introduced to compensatethe low-resolution NeRF inside for high-resolution synthesis.
Tab.
3 reports the evaluation on modelswith or without the consistency branch.
With consistency branch, FID scores are on par with baselines’or experience a slightly degradation.
While the geometries become better given the supervision fromthe consistency branch, which is indicated by the SIDE metric.
The consistency of the generatedimages is also improved.
The results demonstrate the necessity of introducing a consistency branchin the discriminator to enhance the multi-view consistency of the generator.
[Figure]
[Figure]
4.5 Application of GeoD in GAN inversion
One of the potential applications of GeoD is to help better geometry reconstruction from a real image.
To extract the underlying shape from a real image, GAN inversion is performed on the 3D-awaregenerator following [2].
As shown in Fig.
6, the generator of π -GAN trained with a 2D discriminatorreconstructs geometries with bumpy surfaces, resulting in inconsistent novel view synthesis.
Withour geometry-aware discriminator, the reconstructed shape becomes smooth and realistic, and theimages synthesized under different viewpoints are more consistent with each other.
Conclusion
In this paper, we propose a new geometry-aware discriminator, GeoD , to improve 3D-aware imagesynthesis.
GeoD not only provides supervision on 2D images but also guides the learning of theunderlying geometry.
With such a discriminator, 3D-aware generators produce high-fidelity imageswith better underlying geometry.
Besides, we further extend our GeoD to improve the mult-viewconsistency of the generator.
Experimental results demonstrate the effectiveness of our proposeddiscriminator from both the image quality and the geometry perspectives.
We hope our work canbring more attention to the research on effective discriminators.
Discussion.
Despite the high quality achieved in both the 2D image and the geometry, the qualityof the generated geometries is influenced by the performance of the geometry branch in GeoD. Forexample, geometry branch for scenes requires labeled data for training.
However, these labels arenot available in the current dataset for synthesis.
Consequently, training the geometry branch on thesame dataset as the generator is impossible, and the one pretrained on other datasets is adopted.
Thedomain gap between the two datasets degrades the quality of supervision signals on geometries, andthus leads to a sub-optimal solution for the generator.
A more efficient geometry branch or a novelfine-tuning method can be proposed to alleviate the problem.