S TYLE N E RF: A S TYLE - BASED 3D-A WARE G ENERATOR FOR H IGH - RESOLUTION I MAGE S YNTHESIS
A BSTRACT
We propose StyleNeRF , a 3D-aware generative model for photo-realistic highresolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images.
Existing approaches either cannot synthesize highresolution images with ﬁne details or yield noticeable 3D-inconsistent artifacts.
In addition, many of them lack control over style attributes and explicit 3D camera poses.
StyleNeRF integrates the neural radiance ﬁeld (NeRF) into a style-based generator to tackle the aforementioned challenges, i e., improving rendering efﬁciency and 3D consistency for high-resolution image generation.
We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the ﬁrst issue.
To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss.
With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality.
StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views.
It also supports challenging tasks, including style mixing and semantic editing. Code and pre-trained models are available at: https: /github. com facebookresearch
/StyleNeRF.
I NTRODUCTION
Photo-realistic free-view image synthesis is a long-standing problem in computer vision and computer graphics.
Traditional graphics pipeline requires production-quality 3D models, computationally expensive rendering and manual work, making it challenging to apply to large-scale real-world scenes.
In the meantime, Generative Adversarial Networks (GANs, Goodfellow et al., 2014) can be trained on a large number of unstructured images to generate high-quality images.
However, most GAN models operate in 2D space and lack the 3D understanding of the input, which results in their inability to synthesize images of the same scene with 3D consistency and direct camera control.
Natural images are the 2D projection of the 3D world.
Hence, recent works on generative models (Schwarz et al., 2020; Chan et al., 2021) enforce 3D structures by incorporating a neural radiance ﬁeld (NeRF, Mildenhall et al., 2020).
However, these methods cannot synthesize high-resolution images with delicate details due to the computationally expensive rendering process of NeRF.
Furthermore, the slow rendering process leads to inefﬁcient training and makes these models unsuitable for interactive applications.
GIRAFFE (Niemeyer & Geiger, 2021b) combines NeRF with a CNN- based renderer, which has the potential to synthesize high-resolution images.
However, this method falls short of 3D-consistent image generation and so far has not shown high-resolution results.
We propose StyleNeRF , a new 3D-aware generative model for high-resolution 3D consistent image synthesis at interactive rates.
It also allows control of the 3D camera pose and enables control of speciﬁc style attributes.
StyleNeRF incorporates 3D scene representations into a style-based generative model.
To prevent the expensive direct color image rendering from the original NeRF approach, we only use NeRF to produce a low-resolution feature map and upsample it progressively to high resolution.
To improve 3D consistency, we propose several designs, including a desirable upsampler that achieves high consistency while mitigating artifacts in the outputs, a novel regularization term that forces the output to match the rendering result of the original NeRF and ﬁxing the issues of view direction condition and noise injection.
StyleNeRF is trained using unstructured real-world images.
A progressive training strategy signiﬁcantly improves the stability of learning real geometry.
We evaluate StyleNeRF on various challenging datasets.
StyleNeRF can synthesize photo-realistic 1024 2 images at interactive rates while achieving high multi-view consistency.
None of the existing methods can achieve both characteristics.
Additionally, StyleNeRF enables direct control on styles.
R ELATED W ORK
Neural Implicit Fields Representing 3D scenes as neural implicit ﬁelds has increasingly gained much attention.
Michalkiewicz et al.
; Mescheder et al.
; Park et al.
; Peng et al.
(2020) predict neural implicit ﬁelds with 3D supervision.
Some of them (Sitzmann et al., 2019;
Niemeyer et al., 2019) assume that the ray color only lies on the geometry surface and propose differentiable renderers to learn a neural implicit surface representation.
NeRF and and similar works (Lombardi et al., 2019; Mildenhall et al., 2020; Liu et al., 2020; Zhang et al., 2020) utilize a volume rendering technique to render neural implicit volume representations for novel view synthesis.
In this work, we focus on generative NeRF.
Unlike the discussed methods, which require posed multi-view images, our approach only needs unstructured single-view images for training.
Image Synthesis with GANs Starting from Goodfellow et al.
(2014), GANs have demonstrated high-quality results (Durugkar et al., 2017; Mordido et al., 2018;
Doan et al., 2019; Zhang et al., 2019;
Brock et al., 2018; Karras et al., 2018).
StyleGANs (Karras et al., 2019; 2020b) achieve
SOTA quality and support different levels of style control.
Karras et al.
(2021) solve the “texture sticking” problem of 2D GANs in generating animations with 2D transformations.
Some methods (H¨ark¨onen et al., 2020; Tewari et al., 2020a; Shen et al., 2020; Abdal et al., 2020; Tewari et al., 2020b; Leimk¨uhler & Drettakis, 2021; Shoshan et al., 2021) leverage disentangled properties in the latent space to enable explicit controls, most of which focus on faces.
While these methods can synthesize face poses parameterized by two angles, extending them to general objects and controlling 3D cameras is not easy.
Chen et al.
(2021a) proposed to generate segmentation maps from implicit ﬁelds to enable 3D control.
However, it requires 3D meshes for pre-training.
In contrast, our work can synthesize images for general objects , enabling explicit 3D camera control.
3D-Aware GANs
Recently, 3D representations have been integrated into 2D generative models to enable camera control.
Voxel-based GANs (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; 2020) lack ﬁne details in the output due to resolution restriction.
Radiance ﬁelds-based methods (Schwarz et al., 2020;
Chan et al., 2021; Niemeyer & Geiger, 2021a) achieve higher quality but have difﬁculties in training on high-resolution images ( 512 2 and beyond) due to the expensive rendering process.
GIRAFFE (Niemeyer & Geiger, 2021b) improves the training and rendering efﬁciency by combining NeRF with a CNN-based renderer; GSN (DeVries et al., 2021) models a locally conditional NeRF with a similar renderer for unconstrained indoor scene generation.
However, they both produce severe view-inconsistent artifacts due to their network designs (e.g., 3 × 3 Conv and upsampler).
In contrast, our method can effectively preserve view consistency in image synthesis.
M ETHOD 3. I MAGE SYNTHESIS AS NEURAL IMPLICIT FIELD RENDERING
Style-based Generative Neural Radiance Field We start by modeling a 3D scene as neural radiance ﬁeld (NeRF, Mildenhall et al., 2020).
It is typically parameterized as multilayer perceptrons
(MLPs), which takes the position x ∈ R 3 and viewing direction d ∈ S 2 as input, and predicts the density σ ( x ) ∈ R + and view-dependent color c ( x , d ) ∈ R 3 .
To model high-frequency details, follwing NeRF (Mildenhall et al., 2020), we map each dimension of x and d with Fourier features :
[Equation]
We formalize StyleNeRF representations by conditioning NeRF with style vectors w as follows:
[Equation]
z
[Equation]
Z
[Equation]
Similar as StyleGAN2 (Karras et al., 2020b), f is a mapping network that maps noise vectors from the spherical Gaussian space Z to the style space W ; g i w ( . ) is the i th layer MLP whose weight matrix is modulated by the input style vector w .
φ n w ( x ) is the n -th layer feature of that point.
We then use the extracted features to predict the density and color, respectively:
[Equation]
where h σ and h c can be a linear projection or 2-layer MLPs.
Different from the original NeRF, we assume n c > n σ for Equation (3) as the visual appearance generally needs more capacity to model than the geometry.
The ﬁrst min( n σ , n c ) layers are shared in the network.
Volume Rendering Image synthesis is modeled as volume rendering from a given camera pose p ∈ P .
For simplicity, we assume a camera is located on the unit sphere pointing to the origin with a ﬁxed ﬁeld of view (FOV).
We sample the camera’s pitch & yaw from a uniform or Gaussian distribution.
To render an image
I ∈
R H
× W × 3 , we shoot a camera ray r ( t ) =
o + t d ( o is the camera origin) for each pixel, and then calculate the color using the volume rendering equation:
I NeRF w
[Equation]
dt, where p w [Equation] t
[Equation]
exp
[Equation]
[Equation]
In practice, the above equation is discretized by accumulating sampled points along the ray.
Following NeRF (Mildenhall et al., 2020), stratiﬁed and hierarchical sampling are applied for more accurate discrete approximation to the continuous implicit function.
Challenges Compared to 2D generative models (e.g., StyleGANs (Karras et al., 2019; 2020b)), the images generated by NeRF-based models have 3D consistency, which is guaranteed by modeling the image synthesis as a physics process, and the neural 3D scene representation is invariant across different viewpoints.
However, the drawbacks are apparent: these models cost much more computation to render an image at the exact resolution.
For example, 2D GANs are 100 ∼times more efﬁcient to generate a 1024 2 image than NeRF-based models.
Furthermore, NeRF consumes much more memory to cache the intermediate results for gradient back-propagation during training, making it difﬁcult to train on high-resolution images.
Both of these restrict the scope of applying NeRF-based models in high-quality image synthesis, especially at the training stage when calculating the objective function over the whole image is crucial.
3. A PPROXIMATION FOR HIGH - RESOLUTION IMAGE GENERATION
In this section, we propose how to improve the efﬁciency of StyleNeRF by taking inspiration from 2D
GANs.
We observe that the image generation of 2D GANs (e.g., StyleGANs) is fast due to two main reasons: (1) each pixel only takes single forward pass through the network; (2) image features are generated progressively from coarse to ﬁne, and the feature maps with higher resolutions typically have a smaller number of channels to save memory.
In StyleNeRF , the ﬁrst point can be partially achieved by early aggregating the features into the 2D space before the ﬁnal colors are computed.
In this way, each pixel is assigned with a feature vector,
Furthermore, it only needs to pass through a network once rather than calling the network multiple times for all sampled points on the ray as NeRF does.
We approximate Equation (4) as:
[Equation]
where φ n,n σ w [Equation] A
[Equation]
= g n w ◦ g n − 1 w
[Equation]
The deﬁnitions of A ( . ) and φ n,n σ w ( . ) can be extended to the operations on a set of rays, each ray processed independently.
Next, instead of using volume rendering to render a high-resolution feature map directly, we can employ NeRF to generate a downsampled feature map at a low resolution and then employ upsampling in 2D space to progressively increase into the required high resolution.
We take two adjacent resolutions as an example.
Suppose R L ∈ R H/ 2 ×
W/
and R H ∈
R H
× W are the corresponding rays of the pixels in the low- and high-resolution images, respectively.
To approximate the high-resolution feature map, we can up-sample in the low-resolution feature space:
[Equation]
Recursively inserting Upsample operators enables efﬁcient high-resolution image synthesis as the computationally expensive volume rendering only needs to generate a low-resolution feature map.
The efﬁciency is further improved when using fewer channels for higher resolution.
While early aggregation and upsampling operations can accelerate the rendering process for highresolution image synthesis, they would come with scariﬁcation to the inherent consistency of NeRF.
There are two reasons why they introduce inconsistency.
First, the resulting model contains nonlinear transformations to capture spurious correlations in 2D observation, mainly when substantial ambiguity exists.
For example, our training data are unstructured single-view images without sufﬁcient multi-view supervision.
Second, such a pixel-space operation like up-sampling would compromise 3D consistency.
Therefore, na ¨ ıve model designs would lead to severe multi-view inconsistent outputs (e.g., when moving the camera to render images, hairs are constantly changing).
In the following, we propose several designs and choices to alleviate the inconsistency in the outputs.
3. P RESERVING 3D CONSISTENCY
Upsampler design Up-sampling in 2D space causes multi-view inconsistency in general; however, the speciﬁc design choice of the upsampler determines how much such inconsistency is introduced.
As our model is directly derived from NeRF, MLP ( 1 × 1 Conv ) is the basic building block.
With MLPs, however, pixel-wise learnable upsamplers such as pixelshufﬂe (Shi et al., 2016) or LIIF (Chen et al., 2021b) produce “chessboard” or “texture sticking” artifacts due to its tendency of relying on the image coordinates implicitly.
In the meanwhile, Karras et al.
(2019; 2020b; 2021) proposed to use non-learnable upsamplers that interpolate the feature map with pre-deﬁned lowpass ﬁlters (e.g. bilinear interpolation).
While these upsamplers can produce smoother outputs, we observed non-removable “bubble” artifacts in both the feature maps and output images.
We conjecture it is due to the lack of local variations when combining MLPs with ﬁxed low-pass ﬁlters.
We achieve the balance between consistency and image quality by combining these two approaches (see
Figure 2).
For any input feature map X ∈ R N × N × D :
Upsample
[Equation]
where ψ θ : R D → R 4 D is a learnable 2-layer MLP, and K is a ﬁxed blur kernel (Zhang, 2019).
NeRF path regularization We propose a new regularization term to enforce 3D consistency, which regularizes the model output to match the original path (Equation (4)).
In this way, the ﬁnal outputs can be closer to the NeRF results, which have multi-view consistency.
This is implemented by sub-sampling pixels on the output and comparing them against those generated by NeRF:
[Equation]
where S is the set of randomly sampled pixels; R in and R out are the corresponding rays of the pixels in the low-resolution image generated via NeRF and high-resolution output of StyleNeRF.
Remove view direction condition Predicting colors with view direction condition was suggested by the original NeRF for modeling view-dependent effects and was by default applied in most follow-up works (Chan et al., 2021; Niemeyer & Geiger, 2021b).
However, this design would give the model additional freedom to capture spurious correlations and dataset bias, especially if only a single-view target is provided.
We, therefore, remove the view direction in color prediction, which improves the synthesis consistency (See Figure 8).
Fix 2D noise injection Existing studies (Karras et al., 2019; 2020b;
Feng et al., 2020) have showed that injecting per-pixel noise can increase the model’s capability of modeling stochastic variation (e.g. hairs, stubble).
Nevertheless, such 2D noise only exists in the image plane, which will not change in a 3D consistent way when the camera moves.
To preserve 3D consistency, our default solution is to trade the model’s capability of capturing variation by removing the noise injection as
Karras et al.
(2021) does.
Optionally, we also propose a novel geometry-aware noise injection based on the estimated surface from StyleNeRF.
See Appendix A.4 for more details.
3. S TYLE N E RF A RCHITECTURE
In this section, we describe the network architecture and the learning procedure of StyleNeRF .
Mapping Network Following StyleGAN2, latent codes are sampled from standard Gaussian and processed by a mapping network.
Finally, the output vectors are broadcast to the synthesis networks.
Synthesis Network Considering that our training images generally have unbounded background, we choose NeRF++ (Zhang et al., 2020), a variant of NeRF, as the StyleNeRF backbone.
NeRF++ consists of a foreground NeRF in a unit sphere and a background NeRF represented with inverted sphere parameterization.
As shown in Figure 3, two MLPs are used to predict the density where the has fewer than the foreground
Then a shared MLP is for
Each
Discriminator & Objectives
We use the same discriminator as StyleGAN2.
Following previous works (Chan et al., 2021; Niemeyer & Geiger, 2021b), StyleNeRF adopts a non-saturating GAN objective with R1 regularization (Mescheder et al., 2018).
A new NeRF path regularization is employed to enforce 3D consistency.
The ﬁnal loss function is deﬁned as follows ( D is the discriminator and
G is the generator including the mapping and synthesis networks):
[Equation]
where f ( u )
=
− log(1 + exp( − u )) , and p data is the data distribution.
We set β = 0 .
2 and λ = 0 .
Progressive Training We train StyleNeRF
progressively from low to high resolution, which makes the training more stable and efﬁcient.
We observed in the experiments that were directly training for the highest resolution easily makes the model fail to capture the object geometry.
We suspect it is because both Equations (5) and (6) are just approximations to the original NeRF.
Therefore, inspired by Karras et al.
(2017), we propose a new three -stage progressive training strategy:
For the ﬁrst Timages, we train StyleNeRF without approximation at low-resolution; then, during T 1 ∼ T 2 images, both the generator and discriminator linearly increase the output resolutions until reaching the target resolution; At last, we ﬁx the architecture and continue training the model at the highest resolution until T 3 images.
Please refer to the Appendix A.5 for more details.
E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Datasets We evaluate StyleNeRF on four high-resolution unstructured real datasets: FFHQ (Karras et al., 2019), MetFaces (Karras et al., 2020a), AFHQ (Choi et al., 2020) and CompCars (Yang et al., 2015).
The dataset details are described in Appendix B.
Baselines We compare our approach with a voxel-based method, HoloGAN (Nguyen-Phuoc et al., 2019), and three radiance ﬁeld-based methods: GRAF (Schwarz et al., 2020), π -GAN (Chan et al., 2021) and GIRAFFE (Niemeyer & Geiger, 2021b).
As most of the baselines are restricted to low resolutions, we made the comparison at 256 2 pixels for fairness.
We also report the results of the state-of-the-art 2D GAN (Karras et al., 2020b) for reference.
See more details in Appendix C.
Conﬁgurations All datasets except MetFaces are trained progressively.
However, the MetFaces dataset is too small to train stably, so we ﬁnetune from the pretrained model for FFHQ at the highest resolution.
By default, we train 64 images per batch, and set T 1 = 500 k, T 2 = 5000 k and T 3 = 25000 k images, respectively.
The input resolution is ﬁxed 32 2 for all experiments.
4. R ESULTS
Qualitative comparison We evaluate our approach and the baselines on three datasets: FFHQ, AFHQ, and CompCars.
Target images are resized to 256 2 .
We render each object in a sequence and sample four viewpoints shown counterclockwise in Figure 4.
While all baselines can generate images under direct camera control, HoloGAN, GARF, and π -GAN fail to learn geometry correctly and thus produce severe artifacts.
GIRAFFE synthesizes images in better quality; however, it produces 3D inconsistent artifacts: the shape and appearance in the output change constantly when the camera moves.
We believe it is due to the wrong choice of 3 × 3 Conv layers.
Compared to the baselines, StyleNeRF achieves the best visual quality with high 3D consistency across views.
Please see more results in the supplemental video.
Quantitative comparison We measure the visual quality of image generation by the Frechet Inception Distance (FID,
Heusel et al., 2017) and Kernal Inception
Distance (KID, Bi´nkowski et al., 2018) in Table 1.
Across all three datasets,
StyleNeRF consistently outperforms the baselines by signiﬁcant gains in terms of FID and KID and largely reduces the performance gap between the 3D-aware GANs and the SOTA 2D GAN (i.e., StyleGAN2).
Note that while 2D GANs can achieve high quality for each image, they cannot synthesize images of the same scene with 3D consistency.
Table 1 also shows the speed comparison over different image resolutions on FFHQ.
StyleNeRF enables rendering at interactive rates and achieves signiﬁcant speed-up over voxel and pure NeRF-based methods, and is comparable to GIRAFFE and StyleGAN2.
In addition, we include comparison on the consistency of synthesis quantitatively in Appendix D.
High-resolution synthesis Unlike the baseline models, our method can generate high-resolution images ( 512 2 and beyond).
Figure 5 shows an uncurated set of images rendered by StyleNeRF .
We also show more results of StyleNeRF and report the quantitative results of the SOTA 2D GAN (StyleGAN2 (Karras et al., 2020b)) for reference in the Appendix D.
4. C ONTROLLABLE I MAGE S YNTHESIS
Explicit camera control Our method can synthesize novel views with direct camera control and generalize to extreme camera poses, which starkly differs from the training camera
pose distribution.
Figure 6 shows our results with extreme camera poses, such as zoom-in and -out, and steep view angles.
The rendered images maintain good consistency given different camera poses.
[Table]
Style mixing and interpolation Figure 7 shows the results of style mixing and interpolation.
As shown in style mixing experiments, copying styles before 2D aggregation affects geometry aspects (shape of noses, glasses, etc.), while copying those after 2D aggregation brings changes in appearance (colors of skins, eyes, hairs, etc.), which indicates clear disentangled styles of geometry and appearance.
In the style interpolation results, the smooth interpolation between two different styles without visual artifacts further demonstrates that the style space is semantically learned.
Style inversion and editing We can also do inverse rendering tasks with a learned StyleNeRF model.
We ﬁrst pre-train a camera pose predictor in a selfsupervised manner.
Then given an input image, we use this predictor to estimate the camera pose and optimize the styles via back-propagation to ﬁnd the best styles that match the target image.
We can further perform semantic editing.
For instance, we can optimize the styles according to text snippet using a CLIP loss (Radford et al., 2021; Patashnik et al., 2021).
An example is shown in Figure 7.
4. A BLATION S TUDIES
Figure 8 (a) shows a typical result without progressive training.
Despite not directly affecting the quality of each single image, the model fails to learn the correct shape, e.g. the face should be in convex shape but is predicted as concave surfaces.
This leads to severe 3D inconsistent artifacts when the camera moves (see the red circles in Figure 8 (a))
As shown in Figure 8 (b), without NeRF-path regularization, the model sometimes gets stuck learning geometry in 3D and produces a “ﬂat shape” output.
Figure 8 (c) demonstrates an example of the result with view direction as a condition.
The modeling of view-dependent effects introduces an ambiguity between 3D shape and radiance and thus leads to a degenerate solution without multi-view consistency (see the red circles in Figure 8 (c)).
4. L IMITATIONS AND F UTURE WORK
While StyleNeRF can efﬁciently synthesize photo-realistic high-resolution images in high multiview consistency with explicit camera control, it sacriﬁces some properties that pure NeRF-based al edit with CLIP scores, we
“a hair” as methods have.
An example is shown in Figure 8 (d), where we extract the underlying geometry with marching cube from the learned density.
Although StyleNeRF is able to recover a coarse geometry, it captures less details compared to the pure NeRF-based models such as π -GAN (Chan et al.
Additionally, StyleNeRF only empirically preserves multi-view consistency, but it does not guarantee strict 3D consistency.
Further exploration and theoretical analysis are needed.
C ONCLUSION
We proposed a 3D-aware generative model, StyleNeRF , for efﬁcient high-resolution image generation with high 3D consistency, which allows control over explicit 3D camera poses and style attributes.
Our experiments have demonstrated that StyleNeRF can synthesize photo-realistic 1024images at interactive rates and outperforms previous 3D-aware generative methods.
E THICS S TATEMENT
Our work focuses on technical development, i.e., synthesizing high-quality images with user control.
Our approach can be used for movie post-production, gaming, helping artists reduce workload, generating synthetic data to develop machine learning techniques, etc.
Note that our approach is not biased towards any speciﬁc gender, race, region, or social class.
It works equally well irrespective of the difference in subjects.
However, the ability of generative models, including our approach, to synthesize images at a quality that some might ﬁnd difﬁcult to differentiate from source images raises essential concerns about different forms of disinformation, such as generating fake images or videos.
Therefore, we believe the image synthesized using our approach must present itself as synthetic.
We also believe it is essential to develop appropriate privacy-preserving techniques and large-scale authenticity assessment, such as ﬁngerprinting, forensics, and other veriﬁcation techniques to identify synthesized images.
Such safeguarding measures would reduce the potential for misuse.
We also hope that the high-quality images produced by our approach could foster the development of the forgery mentioned above detection and veriﬁcation systems.
Finally, we believe that a robust public conversation is essential to creating a set of appropriate regulations and laws that would alleviate the risks of misusing these techniques while promoting their positive effects on technology development.
R EPRODUCIBILITY S TATEMENT
We assure that all the results shown in the paper and supplemental materials can be reproduced.
Furthermore, we will open-source our code together with pre-trained checkpoints.
To reproduce our results, we provide the preprocessing procedures of each dataset and implementation details in the main paper and Appendix.
A CKNOWLEDGEMENTS
Christian Theobalt was supported by ERC Consolidator Grant 4DReply (770784).
Lingjie Liu was supported by Lise Meitner Postdoctoral Fellowship.
A A DDITIONAL M ODEL D ETAILS A. I MPLICIT F IELDS M ODELING
Modeling backgournd with NeRF++ Most unstructured images used in this work have a complex and unbounded background, especially for the images taken from 360 degree directions (e.g., images in the CompCars dataset).
Therefore, it is inefﬁcient and difﬁcult to model the entire embedded scene in the image within a ﬁxed bounding box.
Instead, following NeRF++ (Zhang et al., 2020), we partition the whole scene into foreground and background where the background is modeled with an additional network that takes as inputs the background latent codes and a 3D point x = ( x, y, z ) transformed using inverted sphere parameterization:
[Equation]
Then, we uniformly sample G background points in an inverse depth range: [1 /R, 0) where R = 2 .in
our experiments to represent where the background starts.
Hierarchical volume sampling Similar to NeRF (Mildenhall et al., 2020), we adopt an efﬁcient hierarchical sampling strategy for the foreground network learning.
We ﬁrst uniformly sample N points between the near and far planes and then perform importance sampling of M points based on the estimated density distribution in the coarse sampling stage.
Different from NeRF (Mildenhall et al., 2020), we use a single network for coarse and ﬁne sampling to predict the color and density.
A. C AMERA
Camera setup We assume that each image is rendered by a camera with a ﬁxed intrinsic matrix in a pose sampled from a pre-deﬁned camera pose distribution.
The intrinsics are determined by ﬁeldof-view (FOV) and image resolution, normalized to [ − 1 , 1] regardless of the actual resolution.
The camera is located on the unit sphere, pointing at the world origin where the target object is located.
Then the camera pose p ( θ, φ ) is the function of pitch ( θ ) and yaw ( φ ).
Typically, the cameras in real data do not follow the simple distributions that we assumed, which is the main problem for all 3D-aware GANs (including the proposed model) to learn the correct geometry.
To tackle this issue,
Niemeyer & Geiger (2021a) proposed to learn a neural network to generate camera simultaneously poses jointly with scene representations.
However, our initial attempts of applying similar methods to StyleNeRF did not work, and the camera generator quickly diverged.
Instead, we manually set the parameters φ and θ from either a Gaussian or uniform distribution depending on the datasets.
Camera predictor
Once StyleNeRF is trained, we can optionally train an additional camera predictor in a selfsupervised manner.
More speciﬁcally, we ﬁrst randomly sample a camera pose and the latent codes to render the output image.
Then, a CNN-based encoder (backbone initialized with a pretrained ResNet18) reads the image and predicts the input camera parameters based on a smoothed
L1 loss function.
The camera predictor can be directly used on natural images, in particular for style inversion tasks (see Figure 7) as our early exploration shows that it is non-trivial to train the camera pose from random through latent optimization.
A. D ERIVATION OF A PPROXIMATION
The following shows the detailed derivation of Equation (6).
For simplicity, we omit the condition on style vectors ω in the equation.
[Figure]
[Equation]
In the above derivation
[Equation]
we treat the volume rendering integral as an expectation.
Therefore, if p ( t ) is close to an impulse function which is near zero everywhere outside a small region near the object surface, locally h c and φ n c ,n σ can be seen as linear functions.
Because the expectation of a linear function in an interval is equal to the value of the function evaluated at the middle point of the interval, the accuracy of the proposed approximation is well justiﬁed under this assumption.
A. D ETAILS ABOUT THE NOISE INJECTION
As mentioned in the paper, injecting per-pixel noise improves the generation quality of GANs.
However, naively adding such noise as in Karras et al.
(2019) is not feasible for our purpose because such noise exists in the pixel space, and it does not move when the viewpoint changes.
On the other hand, the output gets noisy if we inject random noise for each frame separately.
To avoid such inconsistency, the default setup in our experiments is to remove the noise injection as similarly done in (Karras et al., 2021) which works reasonably but has less ability to model local stochastic variations, resulting in lower visual quality (in terms of metrics like FID, KID, etc).
As an alternative, to improve synthesizing quality and keep multi-view consistency, we design a geometry-aware method for noise injection.
Precisely, for each feature layer at the resolution of
N 2 , we extract the underlying geometry using marching-cube based on the predicted density.
Next, we set the volume resolution N 3 accordingly.
Then, we assign independent Gaussian noise at each vertex of the extracted mesh and render the noise map via rasterization from the same viewpoint.
Finally, we inject the resulting noise to each layer before non-linear activation.
Since this process requires a proper density function to start with, we keep the standard per-pixel noise unchanged during training and only apply such geometry-aware noise at inference time.
Compared to models without noise injection, although the proposed approach can capture more stochastic details, it is relatively slower (due to multi-scale marching-cube and rasterization), and the quality of the learned geometry constrains the ﬁnal output.
A. P ROGRESSIVE T RAINING
As described in the main paper, it is essential to train StyleNeRF progressively from low to high resolutions to learn good geometry.
We consider two types of progressive training:
Progressive Growing
Following Karras et al.
(2017), we start from a shallow network for lowresolution, and progressively fade in layers.
Since the parameters of the newly added layers are random, it is essential to have a linear interpolation with low-resolution output to stabilize training.
Progressive Up-sampling Instead of growing new layers, another option is to train StyleNeRF by progressively inserting upsample operations during training as similarly done in Chan et al.
In this case, we do not need to fade in layers linearly.
In both cases, the discriminator correspondingly grows from low to high resolutions progressively as described in Karras et al.
Figure 11 illustrates progressive training for the generator and the discriminator.
In practice, we found that both methods worked similarly in terms of visual quality, while progressive growing achieves a better speed advantage for low-resolution images.
Therefore, our results are reported by training with progressive growing.
A. H YPERPARAMETERS
We reuse the same architecture and parameters of StyleGAN2 for the mapping network ( 8 fully connected layers, 100 × lower learning rate) and discriminator.
In addition, both the latent and style dimensions are set to 512 .
For the foreground and background ﬁelds, before predicting the density, we use the style-based MLPs with 256 and 128 hidden units, respectively.
The Fourier feature dimension is set to L = 10 (Equation (1)) for both ﬁelds.
For layers after 2D aggregation, we follow the ﬂexible layer conﬁguration used by StyleGAN2, which decreases the feature sizes from 512 (for 32 2 ) until 32 (for 1024 2 ).
We keep most of the training conﬁgurations unchanged from StyleGAN2.
A. I MPLEMENTATION
We implement our model based on the ofﬁcial Pytorch implementation of StyleGAN2-ADA 1 .
By default, we train the StyleNeRF model by going through 25000 k images with a minibatch = 64 .
All models are trained on 8 Tesla V100 GPUs for about three days.
[Table]
B D ATASET D ETAILS
FFHQ ( https://github.com/NVlabs/ffhq-dataset ) contains 70k images of real human faces in resolution of 1024 2 .
We assume the human face to be captured at the origin.
In the training stage, we sample the pitch and yaw of the camera from the Gaussian distribution.
AFHQ ( https://github.com/clovaai/stargan-v2# animal-faces-hq-dataset-afhq ) contains in total 15k images of animal faces including cat, dog and wild three categories in resolution of 512 2 .
We directly merge all training images without using the label information.
Similar to the FFHQ setting, we sample the pitch and yaw from the Gaussian distribution.
MetFaces ( https://github.com/NVlabs/metfaces-dataset ) contains 1336 images of faces extracted from artworks.
The original resolution is 1024 2 , and we resize it to 512 2 for our main experiments.
As mentioned in the paper, we ﬁnetune StyleNeRF
from the FFHQ checkpoint due to the small size of MetFaces.
CompCars ( http://mmlab.ie.cuhk.edu.hk/datasets/comp_cars/ ) contains 136726 images capturing the entire cars with different styles.
The original dataset contains images with different aspect ratios.
We preprocess the dataset by center cropping and resizing them into 256 2 .
In the training stage, we sample the camera 360 degree uniformly.
C B ASELINE D ETAILS
HoloGAN (Nguyen-Phuoc et al., 2019)
We train HoloGAN on top of the ofﬁcial implementation.
Speciﬁcally, HoloGAN is trained in resolution 256 2 with Adam optimizer with an initial learning rate of 5e-5.
We train each HoloGAN model for 50 epochs.
The learning rate linearly decays between epoch 25 and 50.
As original HoloGAN only supports training on images of resolution 64and 128 2 , to synthesize images of resolution 256 2 we follow the same adaptation scheme as used in GIRAFFE (Niemeyer & Geiger, 2021b) that add one convolution layer with AdaIn (Huang &
Belongie, 2017) and leaky ReLU activation on top of the ofﬁcial synthesis network.
GRAF (Schwarz et al., 2020)
We adopt the ofﬁcial implementation 3 and retrain GRAF models on FFHQ, AFHQ and preprocessed CompCars dataset in resolution 256 2 .
Following the default setting, all models are trained at the target resolution using the patch-based discriminator.
π -GAN (Chan et al., 2021)
Same as GRAF, we use the ofﬁcial implementation 4 and retrain π -GAN models on these three datasets: FFHQ, AFHQ and CompCars.
Due to the high cost of running π -GAN in high-resolution, we follow the same pipeline, which progressively increases the resolution from 32 2 until 128 2 , and we render the ﬁnal outputs in 256 2 by sampling more pixels.
GIRAFFE (Niemeyer & Geiger, 2021b)
For FFHQ dataset, we directly use their pretrained models of resolution 256 2 .
For AFHQ and CompCars dataset, we retrain GIRAFFE with the ofﬁcial Giraffe implementation 5 .
We change the default random crop preprocessing to center crop to compare with other approaches.
[Table]
[Table]
D A DDITIONAL R ESULTS
Comparison to StyleGAN2 We also provide quantitative comparisons on high-resolution image synthesis against StyleGAN2 and a variant that adopts comparable settings with our default model.
As shown in Table 2, there is still a gap to the best 2D-GAN models.
Nevertheless, the proposed
StyleNeRF reaches similar or even better results as 2D models in similar architectures.
This implies the quality drops are mainly from less powerful architecture (e.g., no noise injection, no 3 ×convolutions).
As future work, more exploration can be done to close this gap.
Quantitative comparison on 3D Consistency We conduct quantitative comparisons to measure the multi-view consistency of the synthesized images from StyleNeRF and two baselines ( π -GAN,
GIRAFFE).
We do not employ the metric used in GRAF (Schwarz et al., 2020) because the calculation of that metric requires 3D ground-truth geometry, and thus it can only be applied to synthetic data, not real data where the ground-truth geometry is not available.
Instead, we propose a new method to measure multi-view consistency without the requirement of 3D shapes.
First for each model, we randomly sample 1000 seeds, and for each seed, we generate a sequence of 9 images following the same trajectory.
Next, we take four images as conditioning, and predict pseudo ground truth images given the remaining ﬁve camera poses using a pre-trained
IBRNet (Wang et al., 2021), a recently proposed image-based rendering method for generic view interpolation and can generalize to novel scenes.
Then we measure the typical image reconstruction scores (PSNR, SSIM, and LPIPS (Zhang et al., 2018)) between the pseudo targets and the remaining images.
Since the IBRNet model is ﬁxed for all the methods, the reconstruction scores can be used as a metric to measure the multi-view consistency of different methods (the better the reconstruction scores are, the higher consistency the synthesized images are).
We tested on the FFHQ 256 2 and AFHQ 256 2 and reported the results in Table 3.
StyleNeRF achieves much better consistency than GIRAFFE, while it is comparable to π -GAN, the SoTA pure
NeRF-based model, in terms of multi-view consistency.
Meanwhile, StyleNeRF can generate highresolution images efﬁciently, which cannot be achieved by existing pure NeRF-based models.
3D Reconstruction We also include the COLMAP (Sch¨onberger & Frahm, 2016) reconstruction example of the proposed StyleNeRF on FFHQ to validate the 3D consistency of the model’s output.
As shown in Figure 12, given a style, we sample 36 camera poses to generate images and obtain the reconstructed point clouds from COLMAP with default parameters and no known camera poses.
Ablation on upsamplers We test different upsampling operators quantitatively on FFHQ 512 2 and report the FID scores in Table 4.
We also include the additional result of using pixelshufﬂe without applying the blur kernel as described in Equation (6).
The FID results show that the proposed upsampler achieves the best quality compared to other operators.
Additional Visual Results We show additional results of generating images with different camera views as follows.
Please also refer to the supplemental video, which shows more results.
[Figure]