Original: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
Revised: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields

Original: Abstract
Revised: Abstract

Original: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images.
Revised: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images.

Original: However, these methods focus on 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes.
Revised: However, these methods focus on 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes.

Original: In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors.
Revised: In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors.

Original: At its core is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to a given 3D Morphable Model (3DMM) mesh.
Revised: At its core is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to a given 3D Morphable Model (3DMM) mesh.

Original: To achieve accurate control over fine-grained 3D face shapes of the synthesized image, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis algorithm.
Revised: To achieve accurate control over fine-grained 3D face shapes of the synthesized image, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis algorithm.

Original: Experiments validate the effectiveness of the proposed method, which can generate high-fidelity face images and shows more precise 3D controllability than state-ofthe-art 2D-based controllable face synthesis methods.
Revised: Experiments validate the effectiveness of the proposed method, which can generate high-fidelity face images and shows more precise 3D controllability than state-ofthe-art 2D-based controllable face synthesis methods.

Original: Find code and more demo at https://keqiangsun.github.io/projects/cgof.
Revised: Find code and more demo at https://keqiangsun.github.io/projects/cgof.

Original:  Introduction
Revised:  Introduction

Original: Recent success of Generative Adversarial Networks (GANs) [13] has led to tremendous progress in face image synthesis.
Revised: Recent success of Generative Adversarial Networks (GANs) [12] has led to tremendous progress in face image synthesis.

Original: State-of-the-art methods, such as StyleGAN [21, 22, 20], are capable of generating photo-realistic face images.
Revised: State-of-the-art methods, such as StyleGAN [19, 20, 18], are capable of generating photo-realistic face images.

Original: Apart from photo-realism, being able to control the appearance of the generated images is also key in many real-world applications, such as face animation, reenactment, and free-viewpoint rendering.
Revised: Apart from photo-realism, being able to control the appearance of the generated images is also key in many real-world applications, such as face animation, reenactment, and free-viewpoint rendering.

Original: Early works on controllable face synthesis rely on external attribute annotations to learn an attribute-guided face image generation model [27, 11, 55].
Revised: Early works on controllable face synthesis rely on external attribute annotations to learn an attribute-guided face image generation model [25, 10, 53].

Original: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [26], can only provide abstract semantic-level guidance on the generation, and the generated faces often lack 3D geometric consistency.
Revised: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [24], can only provide abstract semantic-level guidance on the generation, and the generated faces often lack 3D geometric consistency.

Original: Moreover, it is often much harder to obtain low-level geometric annotations beyond semantic labels for direct 3D supervision.
Revised: Moreover, it is often much harder to obtain low-level geometric annotations beyond semantic labels for direct 3D supervision.

Original: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as 3D Morphable Models (3DMMs) [2, 38], into StyleGAN-based synthesis models, allowing for more precise 3D control over the generated images, including facial expressions and head poses [8, 52, 39].
Revised: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as 3D Morphable Models (3DMMs) [2, 36], into StyleGAN-based synthesis models, allowing for more precise 3D control over the generated images, including facial expressions and head poses [7, 50, 37].

Original: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces under large expression and pose variations due to the lack of a 3D representation, as shown in Fig.
Revised: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces under large expression and pose variations due to the lack of a 3D representation, as shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: With the advances in differentiable rendering and neural 3D representations, a recent line of work has explored photo-realistic 3D face generation using only 2D image collections as training data [49, 5, 45, 14, 37, 4, 57, 9].
Revised: With the advances in differentiable rendering and neural 3D representations, a recent line of work has explored photo-realistic 3D face generation using only 2D image collections as training data [47, 4, 43, 13, 35, 3, 55, 8].

Original: Neural Radiance Fields (NeRFs)
Revised: Neural Radiance Fields (NeRFs)

Original: [31], in particular, have enabled 3D generative models, such as pi-GAN [5] and StyleNeRF [14], to synthesize high-fidelity, 3D consistent faces, by training with 2D images only.
Revised: [29], in particular, have enabled 3D generative models, such as pi-GAN [4] and StyleNeRF [13], to synthesize high-fidelity, 3D consistent faces, by training with 2D images only.

Original: However, these models are purely generative and do not support precise 3D control over the generated faces, such as facial expressions.
Revised: However, these models are purely generative and do not support precise 3D control over the generated faces, such as facial expressions.

Original: In this work, we aim to connect these two groups of research—controllable face synthesis and 3D generative models—and present a NeRF-based 3D conditional face synthesis model that can generate high-fidelity face images with precise 3D control over the 3D shape, expression, and pose of the generated faces, by leveraging a parametric 3DMM face model.
Revised: In this work, we aim to connect these two groups of research—controllable face synthesis and 3D generative models—and present a NeRF-based 3D conditional face synthesis model that can generate high-fidelity face images with precise 3D control over the 3D shape, expression, and pose of the generated faces, by leveraging a parametric 3DMM face model.

Original: Imposing precise 3D conditions on an implicit neural radiance field is non-trivial.
Revised: Imposing precise 3D conditions on an implicit neural radiance field is non-trivial.

Original: A naive baseline solution is to enforce that the input 3DMM parameters can be reproduced from the generated face image via a 3DMM reconstruction model, similar to [8].
Revised: A naive baseline solution is to enforce that the input 3DMM parameters can be reproduced from the generated face image via a 3DMM reconstruction model, similar to [7].

Original: Unfortunately, this 3DMM parameter reconstruction loss only provides indirect supervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3D control, as shown in Fig.
Revised: Unfortunately, this 3DMM parameter reconstruction loss only provides indirect supervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3D control, as shown in Fig.

Original: We seek to impose explicit 3D condition directly on the NeRF volume.
Revised: We seek to impose explicit 3D condition directly on the NeRF volume.

Original: To this end, we propose a conditional Generative Occupancy Field (cGOF).
Revised: To this end, we propose a conditional Generative Occupancy Field (cGOF).

Original: It consists of a mesh-guided volume sampling procedure and a complimentary density regularizer that gradually concentrates the volume density around a given mesh.
Revised: It consists of a mesh-guided volume sampling procedure and a complimentary density regularizer that gradually concentrates the volume density around a given mesh.

Original: In order to achieve fine-grained 3D control, such as expression, we further introduce a 3D landmark loss and a volume warping loss.
Revised: In order to achieve fine-grained 3D control, such as expression, we further introduce a 3D landmark loss and a volume warping loss.

Original: The main contributions are as follows.
Revised: The main contributions are as follows.

Original: We propose a novel controllable 3D face synthesis method that learns to generate high-fidelity face images with precise controllability over the 3D shape, expression, and poses, given only a collection of 2D images as training data.
Revised: We propose a novel controllable 3D face synthesis method that learns to generate high-fidelity face images with precise controllability over the 3D shape, expression, and poses, given only a collection of 2D images as training data.

Original: This is achieved by a novel conditional Generative Occupancy Field representation and a set of volumetric losses that effectively condition the generated NeRF volumes on a parametric 3D face model.
Revised: This is achieved by a novel conditional Generative Occupancy Field representation and a set of volumetric losses that effectively condition the generated NeRF volumes on a parametric 3D face model.

Original:  Related Work
Revised:  Related Work

Original: Controllable Generative Adversarial Networks.
Revised: Controllable Generative Adversarial Networks.

Original: Generative Adversarial Nets (GANs) [13] gained popularity over the last decade due to their remarkable image generation ability [42, 19, 21, 22].
Revised: Generative Adversarial Nets (GANs) [12] gained popularity over the last decade due to their remarkable image generation ability [40, 17, 19, 20].

Original: Prior works have studied disentangled representation learning in generative models [16, 52, 51, 23, 46, 1, 32, 6, 7, 25].
Revised: Prior works have studied disentangled representation learning in generative models [14, 50, 49, 21, 44, 1, 30, 5, 6, 23].

Original: Existing work on controllable facial image synthesis [40, 52, 51, 8, 39] relies on 2D image-based generative models, which does not guarantee 3D consistency in the generated images.
Revised: Existing work on controllable facial image synthesis [38, 50, 49, 7, 37] relies on 2D image-based generative models, which does not guarantee 3D consistency in the generated images.

Original: 3D-Aware GAN.
Revised: 3D-Aware GAN.

Original: Another line of work in 3D-Aware generative models has looked into disentangling 3D geometric information from 2D images, and learning to generate various 3D structures, such as voxels [56, 60, 12, 32, 33, 28], meshes [49, 24], and Neural Radiance Fields (NeRFs)
Revised: Another line of work in 3D-Aware generative models has looked into disentangling 3D geometric information from 2D images, and learning to generate various 3D structures, such as voxels [54, 58, 11, 30, 31, 26], meshes [47, 22], and Neural Radiance Fields (NeRFs)

Original: [5, 58, 14, 44, 36, 35], by training on image collections using differentiable rendering.
Revised: [4, 56, 13, 42, 34, 33], by training on image collections using differentiable rendering.

Original: The key idea is to use a discriminator that encourages images rendered from random viewpoints sampled from a prior pose distribution to be indistinguishable from real images.
Revised: The key idea is to use a discriminator that encourages images rendered from random viewpoints sampled from a prior pose distribution to be indistinguishable from real images.

Original: Among these works, pi-GAN has demonstrated high-fidelity 3D synthesis results adopting a NeRF representation.
Revised: Among these works, pi-GAN has demonstrated high-fidelity 3D synthesis results adopting a NeRF representation.

Original: SofGAN [6] z
Revised: SofGAN [5] z

Original: MappingNetwork
Revised: MappingNetwork

Original: [Figure]
Revised: [Figure]

Original: cGOF Generator
Revised: cGOF Generator

Original: [Figure]
Revised: [Figure]

Original: Radiance Integration 3DMM Landmarks 𝐥
Revised: Radiance Integration 3DMM Landmarks 𝐥

Original: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′
Revised: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′

Original: Displacement 𝐹 !"
Revised: Displacement 𝐹 !"

Original: Warped from 𝑧′ NeRF 𝑧 NeRF
Revised: Warped from 𝑧′ NeRF 𝑧 NeRF

Original: Real Image 𝐼
Revised: Real Image 𝐼

Original: Generated Image 𝐼 #
Revised: Generated Image 𝐼 #

Original: Generated Shape ξ Pose
Revised: Generated Shape ξ Pose

Original: Noise ℒ "#$ ℒ %&’($ ℒ !"
Revised: Noise ℒ "#$ ℒ %&’($ ℒ !"

Original: #$ ℒ %&’() ℛ *+((,- ℛ .
Revised: #$ ℒ %&’() ℛ *+((,- ℛ .

Original: ℒ %&’(* ℒ %&’(𝐜
Revised: ℒ %&’(* ℒ %&’(𝐜

Original: 3D Priors for Face Generation.
Revised: 3D Priors for Face Generation.

Original: To learn controllable 3D face synthesis models, researchers have leveraged prior face models.
Revised: To learn controllable 3D face synthesis models, researchers have leveraged prior face models.

Original: 3D Morphable Models (3DMMs) [38, 2] is a widely-used parametric face model, which represents the human face as a set of PCA bases derived from 3D scans.
Revised: 3D Morphable Models (3DMMs) [36, 2] is a widely-used parametric face model, which represents the human face as a set of PCA bases derived from 3D scans.

Original: Many previous works [8, 52, 6, 39] have proposed to incorporate 3D priors of 3DMM into generative models to enable controllable 3D face synthesis.
Revised: Many previous works [7, 50, 5, 37] have proposed to incorporate 3D priors of 3DMM into generative models to enable controllable 3D face synthesis.

Original: In particular, DiscoFaceGAN [8], PIE [51] and
Revised: In particular, DiscoFaceGAN [7], PIE [49] and

Original: StyleRig [52] leverage 3DMM to establish losses to ensure 3D consistency, but still use a 2D imagebased StyleGAN generator.
Revised: StyleRig [50] leverage 3DMM to establish losses to ensure 3D consistency, but still use a 2D imagebased StyleGAN generator.

Original: HeadNeRF [18] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated multi-view datasets.
Revised: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated multi-view datasets.

Original: Our method is trained on unannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.
Revised: Our method is trained on unannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.

Original:  Method
Revised:  Method

Original: Given a collection of single-view real-world face images Y , where each face is assumed to appear only once without multiple views, the goal of this work is to learn a controllable 3D face synthesis model that can generate face images with desired 3D shape, expression, and pose.
Revised: Given a collection of single-view real-world face images Y , where each face is assumed to appear only once without multiple views, the goal of this work is to learn a controllable 3D face synthesis model that can generate face images with desired 3D shape, expression, and pose.

Original: To achieve this, we propose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric 3D Morphable Model (3DMM) [38].
Revised: To achieve this, we propose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric 3D Morphable Model (3DMM) [36].

Original: At the core of our method is a novel conditional Generative
Revised: At the core of our method is a novel conditional Generative

Original: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D control based on the 3DMM, as illustrated in Figure 2.
Revised: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D control based on the 3DMM, as illustrated in Figure 2.

Original: 3.1 Preliminaries on NeRF and pi-GAN
Revised: 3.1 Preliminaries on NeRF and pi-GAN

Original: Neural Radiance Field (NeRF).
Revised: Neural Radiance Field (NeRF).

Original: NeRF [31] represents a 3D scene as a radiance field parametrized by a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color
Revised: NeRF [29] represents a 3D scene as a radiance field parametrized by a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color

Original: c ∈ Rfor every 3D location, given its xyz coordinates and the viewing direction as input.
Revised: c ∈ Rfor every 3D location, given its xyz coordinates and the viewing direction as input.

Original: To render a 2D image of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3D volume, and evaluate the density values and radiance colors at a number of sampled points along each ray.
Revised: To render a 2D image of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3D volume, and evaluate the density values and radiance colors at a number of sampled points along each ray.

Original: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =
Revised: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =

Original: o + t d , where o denotes the camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .
Revised: o + t d , where o denotes the camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .

Original: For each 3D sample point x i , we query the NeRF network f to obtain its σ i and radiance color c i :
Revised: For each 3D sample point x i , we query the NeRF network f to obtain its σ i and radiance color c i :

Original: ( σ i , c i ) = f
Revised: ( σ i , c i ) = f

Original: ( x i , d ) .
Revised: ( x i , d ) .

Original: Using volume rendering [29, 31], the final color of the pixel is given by ˆ C ( r ) =
Revised: Using volume rendering [27, 29], the final color of the pixel is given by ˆ C ( r ) =

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: T i (1 − exp ( − σ i δ i ))
Revised: T i (1 − exp ( − σ i δ i ))

Original: c
Revised: c

Original: i , where
Revised: i , where

Original: T i = exp( − i − 1 (cid:88)
Revised: T i = exp( − i − 1 (cid:88)

Original: j = σ j δ j ) , (1) and δ i
Revised: j = σ j δ j ) , (1) and δ i

Original: =
Revised: =

Original: t i +1 − t
Revised: t i +1 − t

Original: i is the distance between adjacent samples.
Revised: i is the distance between adjacent samples.

Original: To optimize a NeRF on a 3D scene, the photometric loss is computed between the rendered pixels and the ground-truth images captured from a dense set of viewpoints.
Revised: To optimize a NeRF on a 3D scene, the photometric loss is computed between the rendered pixels and the ground-truth images captured from a dense set of viewpoints.

Original: Generative Radiance Field.
Revised: Generative Radiance Field.

Original: Several follow-up works have extended this representation to a generative framework [5, 14, 9, 57].
Revised: Several follow-up works have extended this representation to a generative framework [4, 13, 8, 55].

Original: Instead of using a multi-view reconstruction loss, they train a discriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.
Revised: Instead of using a multi-view reconstruction loss, they train a discriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.

Original: Our method is built on top of pi-GAN [5], a generative radiance field model based on a SIRENmodulated NeRF representation [47].
Revised: Our method is built on top of pi-GAN [4], a generative radiance field model based on a SIRENmodulated NeRF representation [45].

Original: During training, images are rendered from random viewpoints ξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).
Revised: During training, images are rendered from random viewpoints ξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).

Original: To model geometry and appearance variations, a random noise code z is sampled from a standard normal distribution and mapped to a number of frequencies γ i and phase shifts
Revised: To model geometry and appearance variations, a random noise code z is sampled from a standard normal distribution and mapped to a number of frequencies γ i and phase shifts

Original: β i through a mapping network.
Revised: β i through a mapping network.

Original: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before the sinusoidal activations.
Revised: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before the sinusoidal activations.

Original: The rendered images I g =
Revised: The rendered images I g =

Original: G ( z , ξ ) and the real images I randomly sampled from the training dataset are then passed into the discriminator D , and the model is trained using a non-saturating GAN loss with R1 regularization following [30]:
Revised: G ( z , ξ ) and the real images I randomly sampled from the training dataset are then passed into the discriminator D , and the model is trained using a non-saturating GAN loss with R1 regularization following [28]:

Original: [Equation]
Revised: [Equation]

Original: (2) where f ( u ) =
Revised: (2) where f ( u ) =

Original: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.
Revised: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.

Original: 3.2 Controllable 3D Face Synthesis
Revised: 3.2 Controllable 3D Face Synthesis

Original: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [38], which is a parametric morphable face model, where the shape, expression, texture, and other factors of a face are modeled by a set of PCA bases and coefficients ˜
Revised: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [36], which is a parametric morphable face model, where the shape, expression, texture, and other factors of a face are modeled by a set of PCA bases and coefficients ˜

Original: z
Revised: z

Original: =
Revised: =

Original: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.
Revised: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.

Original: Our goal is to condition the generated face image I g on a set of 3DMM parameters
Revised: Our goal is to condition the generated face image I g on a set of 3DMM parameters

Original: ˜ z
Revised: ˜ z

Original: as well as a camera pose ξ , which specifies the desired configuration of the generated face image.
Revised: as well as a camera pose ξ , which specifies the desired configuration of the generated face image.

Original: Baseline.
Revised: Baseline.

Original: To enforce such conditioning, following [8], we make use of an off-the-shelf pre-trained and fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2D image [10].
Revised: To enforce such conditioning, following [7], we make use of an off-the-shelf pre-trained and fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2D image [9].

Original: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures the generated face images commit to the input condition.
Revised: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures the generated face images commit to the input condition.

Original: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which is later passed into the pi-GAN generator G to produce a face image I g .
Revised: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which is later passed into the pi-GAN generator G to produce a face image I g .

Original: Unlike [8], which additionally trains a set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the 3DMM coefficients of all the training images follow a normal distribution and simply standardize them to obtain the corresponding normalized noise code z
Revised: Unlike [7], which additionally trains a set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the 3DMM coefficients of all the training images follow a normal distribution and simply standardize them to obtain the corresponding normalized noise code z

Original: = τ (˜ z ) =
Revised: = τ (˜ z ) =

Original: ˜
Revised: ˜

Original: L − 1 (˜ z − ˜ µ ) , where ˜ µ and ˜ L are the mean and Cholesky decomposition component of covariance matrix of the 3DMM coefficients estimated from all the training images using the 3DMM reconstruction model R .
Revised: L − 1 (˜ z − ˜ µ ) , where ˜ µ and ˜ L are the mean and Cholesky decomposition component of covariance matrix of the 3DMM coefficients estimated from all the training images using the 3DMM reconstruction model R .

Original: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalized parameters and the input noise:
Revised: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalized parameters and the input noise:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMM parameter reconstruction loss.
Revised: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMM parameter reconstruction loss.

Original: Example results of this baseline model can be seen in Fig.
Revised: Example results of this baseline model can be seen in Fig.

Original: Challenges.
Revised: Challenges.

Original: There are several issues with this baseline model.
Revised: There are several issues with this baseline model.

Original: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstruction model R .
Revised: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstruction model R .

Original: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real faces precisely.
Revised: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real faces precisely.

Original: Since our model generates a 3D representation for rendering, a more effective solution is
Revised: Since our model generates a 3D representation for rendering, a more effective solution is

Original: [Figure]
Revised: [Figure]

Original: to impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.
Revised: to impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.

Original: However, the radiance field representation is highly over-parametrized [59].
Revised: However, the radiance field representation is highly over-parametrized [57].

Original: The color of a pixel is computed by integrating the radiance along the entire camera ray, whereas in many cases in reality, it should only reflect the color of the intersection point on an opaque surface.
Revised: The color of a pixel is computed by integrating the radiance along the entire camera ray, whereas in many cases in reality, it should only reflect the color of the intersection point on an opaque surface.

Original: Incorporating losses directly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its converging to an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-view consistency, as shown in Fig.
Revised: Incorporating losses directly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its converging to an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-view consistency, as shown in Fig.

Original: 3.3 Conditional Generative Occupancy Field
Revised: 3.3 Conditional Generative Occupancy Field

Original: In order to impose precise 3D control on the NeRF volume, we first identify an effective way of enforcing the volume density to converge to the surface of human faces.
Revised: In order to impose precise 3D control on the NeRF volume, we first identify an effective way of enforcing the volume density to converge to the surface of human faces.

Original: Inspired by [57], we propose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volume sampler (MgS) and a distance-aware volume density regularizer.
Revised: Inspired by [55], we propose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volume sampler (MgS) and a distance-aware volume density regularizer.

Original: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume sampling procedure.
Revised: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume sampling procedure.

Original: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜
Revised: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜

Original: z
Revised: z

Original: = τ − 1 ( z ) by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .
Revised: = τ − 1 ( z ) by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .

Original: With these coefficients, we can sample a 3D face mesh M in from the 3DMM Λ :
Revised: With these coefficients, we can sample a 3D face mesh M in from the 3DMM Λ :

Original: [Equation]
Revised: [Equation]

Original: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape and expression in 3DMM respectively.
Revised: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape and expression in 3DMM respectively.

Original: During rendering, this mesh will serve as a 3D condition, allowing us to apply importance sampling around the surface region, and suppress the volume densities far away from the surface.
Revised: During rendering, this mesh will serve as a 3D condition, allowing us to apply importance sampling around the surface region, and suppress the volume densities far away from the surface.

Original: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh
Revised: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh

Original: M in along the ray direction by rendering a depth map, taking into account the perspective projection.
Revised: M in along the ray direction by rendering a depth map, taking into account the perspective projection.

Original: This allows us to sample points along the ray r ( t )
Revised: This allows us to sample points along the ray r ( t )

Original: =
Revised: =

Original: o + t d within a set of uniformly spaced intervals around the surface intersection point of the ray and the input mesh:
Revised: o + t d within a set of uniformly spaced intervals around the surface intersection point of the ray and the input mesh:

Original: [Equation]
Revised: [Equation]

Original: , where i = 1 , 2 , . .
Revised: , where i = 1 , 2 , . .

Original: N
Revised: N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 5) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of the sampling region.
Revised: 5) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of the sampling region.

Original: We gradually shrink this margin from 0 .
Revised: We gradually shrink this margin from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 05 of the volume range.
Revised: 05 of the volume range.

Original: Furthermore, we suppress the volume density in the space far away from the given mesh using a distance-aware volume density regularizer.
Revised: Furthermore, we suppress the volume density in the space far away from the given mesh using a distance-aware volume density regularizer.

Original: Specifically, we additionally sample a sparse set of points along each ray, and denote the absolute distances from these sampling points to the mesh surface as d i = | t i − t m | , i = 1 , . . .
Revised: Specifically, we additionally sample a sparse set of points along each ray, and denote the absolute distances from these sampling points to the mesh surface as d i = | t i − t m | , i = 1 , . . .

Original: , N vol .
Revised: , N vol .

Original: The distance-aware volume density regularizer R d is defined as:
Revised: The distance-aware volume density regularizer R d is defined as:

Original: [Equation]
Revised: [Equation]

Original: where σ i is the volume density evaluated at those sample points, and α is an inverse temperature parameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .
Revised: where σ i is the volume density evaluated at those sample points, and α is an inverse temperature parameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .

Original: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .
Revised: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .

Original: The mesh-guided volume sampler together with the volume density regularizer effectively constrains the generated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.
Revised: The mesh-guided volume sampler together with the volume density regularizer effectively constrains the generated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.

Original: 3.4 Auxiliary Losses for Fine-grained 3D Control
Revised: 3.4 Auxiliary Losses for Fine-grained 3D Control

Original: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMM mesh.
Revised: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMM mesh.

Original: However, since 3DMM is only a low-dimensional PCA model of human faces without capturing the fine-grained shape and texture details of real faces, enforcing the generated 3D faces to fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does not guarantee precise control over the geometric details, especially under large facial expression changes.
Revised: However, since 3DMM is only a low-dimensional PCA model of human faces without capturing the fine-grained shape and texture details of real faces, enforcing the generated 3D faces to fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does not guarantee precise control over the geometric details, especially under large facial expression changes.

Original: To enable precise geometric control, we further introduce two auxiliary loss terms.
Revised: To enable precise geometric control, we further introduce two auxiliary loss terms.

Original: 3D Landmark Loss encourages the semantically important facial landmarks to closely follow the input condition.
Revised: 3D Landmark Loss encourages the semantically important facial landmarks to closely follow the input condition.

Original: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68 facial landmarks [43, 10].
Revised: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68 facial landmarks [41, 9].

Original: Let { l k } N k k
Revised: Let { l k } N k k

Original: =1 denote the 3D locations of these landmarks on the input 3DMM mesh.
Revised: =1 denote the 3D locations of these landmarks on the input 3DMM mesh.

Original: In order to find the same set of landmarks on the generated image, we extract 3D landmarks { ˆ l k } N k k
Revised: In order to find the same set of landmarks on the generated image, we extract 3D landmarks { ˆ l k } N k k

Original: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2D projections.
Revised: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2D projections.

Original: We then render the depth values of NeRF at these 2D locations using volume rendering and back-project them to 3D to obtain the 3D landmark locations of the generated volume { ˆ l ′ k } .
Revised: We then render the depth values of NeRF at these 2D locations using volume rendering and back-project them to 3D to obtain the 3D landmark locations of the generated volume { ˆ l ′ k } .

Original: To avoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) when calculating loss.
Revised: To avoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) when calculating loss.

Original: The landmark loss is defined among these three sets of 3D landmarks:
Revised: The landmark loss is defined among these three sets of 3D landmarks:

Original: [Equation]
Revised: [Equation]

Original: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the input mesh at these semantically important landmark locations, which is particularly helpful in achieving precise expression control, as validated in Tab. 2.
Revised: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the input mesh at these semantically important landmark locations, which is particularly helpful in achieving precise expression control, as validated in Tab. 2.

Original: Volume Warping Loss enforces that the NeRF volume generated with a different expression code should be consistent with the original volume warped by a warping field induced from the two 3DMM meshes.
Revised: Volume Warping Loss enforces that the NeRF volume generated with a different expression code should be consistent with the original volume warped by a warping field induced from the two 3DMM meshes.

Original: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled during training, M ′ =
Revised: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled during training, M ′ =

Original: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp , and V and V ′ be their vertices respectively.
Revised: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp , and V and V ′ be their vertices respectively.

Original: We first compute the vertex displacements denoted by
Revised: We first compute the vertex displacements denoted by

Original: ∆ V = V ′ − V , and render them into a 2D displacement map
Revised: ∆ V = V ′ − V , and render them into a 2D displacement map

Original: F ∆ V ∈ R H
Revised: F ∆ V ∈ R H

Original: × W × 3 , where each pixel
Revised: × W × 3 , where each pixel

Original: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .
Revised: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .

Original: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities and radiance colors ( σ i , c i ) =
Revised: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities and radiance colors ( σ i , c i ) =

Original: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.
Revised: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.

Original: The superscript ( u, v ) is dropped for simplicity.
Revised: The superscript ( u, v ) is dropped for simplicity.

Original: We warp all the 3D sample points along each ray using the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D points x ′ i
Revised: We warp all the 3D sample points along each ray using the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D points x ′ i

Original: = x i + F ∆ V (assuming that all the points are already close to the mesh surface).
Revised: = x i + F ∆ V (assuming that all the points are already close to the mesh surface).

Original: We then query the second NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densities and radiance colors ( σ ′
Revised: We then query the second NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densities and radiance colors ( σ ′

Original: i , c ′ i ) =
Revised: i , c ′ i ) =

Original: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.
Revised: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.

Original: In addition, we also enforce the images rendered from the two radiance fields to be consistent.
Revised: In addition, we also enforce the images rendered from the two radiance fields to be consistent.

Original: We integrate the radiance colors c
Revised: We integrate the radiance colors c

Original: i of the warped points with their densities σ ′
Revised: i of the warped points with their densities σ ′

Original: i using Eq.
Revised: i using Eq.

Original: (1), and obtain an image
Revised: (1), and obtain an image

Original: ˆ I g (with all pixels).
Revised: ˆ I g (with all pixels).

Original: This image ˆ I g rendered from the second radiance field is encouraged to be identical to the original image
Revised: This image ˆ I g rendered from the second radiance field is encouraged to be identical to the original image

Original: I g .
Revised: I g .

Original: The final warping loss is thus composed of three terms:
Revised: The final warping loss is thus composed of three terms:

Original: [Equation]
Revised: [Equation]

Original: where β d , β c and β I are the balancing weights.
Revised: where β d , β c and β I are the balancing weights.

Original: 3.5 Neural Renderer and Overall Training Objective
Revised: 3.5 Neural Renderer and Overall Training Objective

Original: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.
Revised: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.

Original: To improve the image quality without introducing too much computational overhead, we leverage an auxiliary neural renderer
Revised: To improve the image quality without introducing too much computational overhead, we leverage an auxiliary neural renderer

Original: [Table]
Revised: [Table]

Original: Similar to RegNeRF [34], we further impose a smoothness regularizer R smooth on the generated NeRF volume by penalizing the L2 differences of neighboring pixels in both the depth maps and normal maps (referred to as R depthsmooth and R norm smooth below).
Revised: Similar to RegNeRF [32], we further impose a smoothness regularizer R smooth on the generated NeRF volume by penalizing the L2 differences of neighboring pixels in both the depth maps and normal maps (referred to as R depthsmooth and R norm smooth below).

Original: The final training objective is thus given by
Revised: The final training objective is thus given by

Original: [Equation]
Revised: [Equation]

Original: (9) where the λ ’s specify the weights for each term.
Revised: (9) where the λ ’s specify the weights for each term.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We train our model on the CelebA [26] dataset, which consists of 200 k celebrity face images with various poses, expressions, and lighting conditions.
Revised: We train our model on the CelebA [24] dataset, which consists of 200 k celebrity face images with various poses, expressions, and lighting conditions.

Original: We use the cropped images and do not make use of their facial attribute annotations .
Revised: We use the cropped images and do not make use of their facial attribute annotations .

Original: The neural renderer module was originally trained on FFHQ dataset, which uses a slightly different cropping scheme.
Revised: The neural renderer module was originally trained on FFHQ dataset, which uses a slightly different cropping scheme.

Original: Therefore, we finetune their pre-trained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.
Revised: Therefore, we finetune their pre-trained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.

Original: Implementation Details.
Revised: Implementation Details.

Original: We provide implementation details in the supplementary material, including network architectures and hyper-parameters.
Revised: We provide implementation details in the supplementary material, including network architectures and hyper-parameters.

Original: Here, we highlight a few important ones.
Revised: Here, we highlight a few important ones.

Original: We build our model on top of the implementation of pi-GAN [5], which uses a SIREN -based NeRF representation.
Revised: We build our model on top of the implementation of pi-GAN [4], which uses a SIREN -based NeRF representation.

Original: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for the density regularizer.
Revised: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for the density regularizer.

Original: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.
Revised: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.

Original: Metrics.
Revised: Metrics.

Original: (1) Disentanglement Score (DS) .
Revised: (1) Disentanglement Score (DS) .

Original: Following [8], we evaluate 3D controllability using a 3DMM Disentanglement Score.
Revised: Following [7], we evaluate 3D controllability using a 3DMM Disentanglement Score.

Original: To measure the DS on one disentangled property z i ( e.g . shape, expression or pose), we randomly sample a set of z i from a normal distribution while keeping other codes fixed { z j } , j ̸ = i , and render images.
Revised: To measure the DS on one disentangled property z i ( e.g . shape, expression or pose), we randomly sample a set of z i from a normal distribution while keeping other codes fixed { z j } , j ̸ = i , and render images.

Original: We re-estimate the 3DMM parameters
Revised: We re-estimate the 3DMM parameters

Original: ˆ z i and { ˆ z j } from these rendered images and compute DS i = (cid:81) j,j ̸
Revised: ˆ z i and { ˆ z j } from these rendered images and compute DS i = (cid:81) j,j ̸

Original: = i σ ˆ z
Revised: = i σ ˆ z

Original: i /σ
Revised: i /σ

Original: ˆ z j , where σ ˆ z i and σ ˆ z j denote variance of the reconstructed parameters of the measured property
Revised: ˆ z j , where σ ˆ z i and σ ˆ z j denote variance of the reconstructed parameters of the measured property

Original: ˆ z
Revised: ˆ z

Original: i and the rest ˆ z j , respectively.
Revised: i and the rest ˆ z j , respectively.

Original: A higher DS score indicates that the model achieves more precise controllability on that property while keeping the others unchanged.
Revised: A higher DS score indicates that the model achieves more precise controllability on that property while keeping the others unchanged.

Original: (2) Chamfer Distance (CD) .
Revised: (2) Chamfer Distance (CD) .

Original: We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely the generated 3D face follows the condition.
Revised: We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely the generated 3D face follows the condition.

Original: (3) Landmark Distance (LD).
Revised: (3) Landmark Distance (LD).

Original: To evaluate the fine-grained controllability of our model, we detect the 3D landmarks in the generated images using [10] and compute the Euclidean distance between the detected landmarks and the ones extracted from the input 3DMM vertices.
Revised: To evaluate the fine-grained controllability of our model, we detect the 3D landmarks in the generated images using [9] and compute the Euclidean distance between the detected landmarks and the ones extracted from the input 3DMM vertices.

Original: (4) Landmark Correlation (LC).
Revised: (4) Landmark Correlation (LC).

Original: We measure the precision of expression control, by computing the 3D landmark displacements of two faces generated with a pair of random expression codes, and reporting the correlation score between these displacements and those obtained from the two 3DMM meshes.
Revised: We measure the precision of expression control, by computing the 3D landmark displacements of two faces generated with a pair of random expression codes, and reporting the correlation score between these displacements and those obtained from the two 3DMM meshes.

Original: (5) Frechet Inception Distance (FID) .
Revised: (5) Frechet Inception Distance (FID) .

Original: The quality of the generated face images is measured using Frechet Inception Distance [17] with an ImageNet-pretrained Inception-V3 [50] feature extractor.
Revised: The quality of the generated face images is measured using Frechet Inception Distance [15] with an ImageNet-pretrained Inception-V3 [48] feature extractor.

Original: We align images with 68 landmarks [43, 48, 41] to avoid domain shift.
Revised: We align images with 68 landmarks [41, 46, 39] to avoid domain shift.

Original: 4.1 Qualitative Evaluation
Revised: 4.1 Qualitative Evaluation

Original: Fig.
Revised: Fig.

Original: 4 shows some example images generated by our proposed cGOF model.
Revised: 4 shows some example images generated by our proposed cGOF model.

Original: In particular, we visualize the generated faces by varying only one of the controlled properties at a time, including
Revised: In particular, we visualize the generated faces by varying only one of the controlled properties at a time, including

Original: [Figure]
Revised: [Figure]

Original: facial expression, head pose, and identity–combining shape and texture as in [8].
Revised: facial expression, head pose, and identity–combining shape and texture as in [7].

Original: Our model is able to generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner, even in the presence of large expression and pose variations.
Revised: Our model is able to generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner, even in the presence of large expression and pose variations.

Original: In contrast, existing methods tend to produce inconsistent face images as shown in Fig.
Revised: In contrast, existing methods tend to produce inconsistent face images as shown in Fig.

Original: Please refer to the supplementary materials for more qualitative results and side-by-side comparisons against state-of-the-art methods.
Revised: Please refer to the supplementary materials for more qualitative results and side-by-side comparisons against state-of-the-art methods.

Original: 4.2 Quantitative Evaluation
Revised: 4.2 Quantitative Evaluation

Original: We compare our method to several state-of-the-art methods.
Revised: We compare our method to several state-of-the-art methods.

Original: E4E [53] and GAN-Control [46] are purely 2D controllable face image synthesis methods, which are based on facial attribute annotations.
Revised: E4E [51] and GAN-Control [44] are purely 2D controllable face image synthesis methods, which are based on facial attribute annotations.

Original: DiscoFaceGAN [8], PIE [51] and StyleRig [52] leverage 3DMM priors but still use a 2D image-based
Revised: DiscoFaceGAN [7], PIE [49] and StyleRig [50] leverage 3DMM priors but still use a 2D image-based

Original: StyleGAN generator.
Revised: StyleGAN generator.

Original: HeadNeRF [18] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated datasets, whereas our method is a 3D generative model trained on unannotated single-view images only.
Revised: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated datasets, whereas our method is a 3D generative model trained on unannotated single-view images only.

Original: We compare the controllability accuracy using the Disentanglement Score (see Col 3 to 5 of Tab. 1).
Revised: We compare the controllability accuracy using the Disentanglement Score (see Col 3 to 5 of Tab. 1).

Original: Our method achieves significantly better disentanglement and more precise control in terms of shapes,
Revised: Our method achieves significantly better disentanglement and more precise control in terms of shapes,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +ℒ $%&’# (
Revised: +ℒ $%&’# (

Original: [Equation]
Revised: [Equation]

Original: ℒ (%)*+ )* + MgS +
Revised: ℒ (%)*+ )* + MgS +

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: #’$- +
Revised: #’$- +

Original: [Equation]
Revised: [Equation]

Original: (-/ +
Revised: (-/ +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: $)
Revised: $)

Original: [Equation]
Revised: [Equation]

Original: *+ gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM.
Revised: *+ gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM.

Original: Refer to Sec.
Revised: Refer to Sec.

Original: 4.3 for more details.
Revised: 4.3 for more details.

Original: ∗ Note that we do not use the depth loss in the final model, which compares the depths rendered from NeRF and 3DMM.
Revised: ∗ Note that we do not use the depth loss in the final model, which compares the depths rendered from NeRF and 3DMM.

Original: expressions, and poses, compared to existing methods.
Revised: expressions, and poses, compared to existing methods.

Original: Note that since PIE and StyleRig do not have open-sourced code, we compute the score on the 167 generated images provided on the PIE project page 2 .
Revised: Note that since PIE and StyleRig do not have open-sourced code, we compute the score on the 167 generated images provided on the PIE project page 2 .

Original: We also compare the FID scores in Col 6 of Tab. 1.
Revised: We also compare the FID scores in Col 6 of Tab. 1.

Original: Our model achieves plausible image quality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.
Revised: Our model achieves plausible image quality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: We conduct ablation studies to validate the effectiveness of each component in our model by adding them one by one, and evaluating the generated images with the four metrics specified above.
Revised: We conduct ablation studies to validate the effectiveness of each component in our model by adding them one by one, and evaluating the generated images with the four metrics specified above.

Original: Tab.and Fig.
Revised: Tab.and Fig.

Original: 5 illustrate the studies and show that all the components improve the precision of 3D controllable image generation.
Revised: 5 illustrate the studies and show that all the components improve the precision of 3D controllable image generation.

Original: In particular, the mesh-guided volume sampler (MgS) significantly improves the effectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).
Revised: In particular, the mesh-guided volume sampler (MgS) significantly improves the effectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).

Original: The 3D landmark loss L ldmk (denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) further enhance the control over fine-grained geometric details.
Revised: The 3D landmark loss L ldmk (denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) further enhance the control over fine-grained geometric details.

Original: We also report FIDs of the outputs of the cGOF with the image resolution of 128 × 128 .
Revised: We also report FIDs of the outputs of the cGOF with the image resolution of 128 × 128 .

Original: Overall, although the geometric regularizers lead to slightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability by large margins (indicated by the improvement in other metrics).
Revised: Overall, although the geometric regularizers lead to slightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability by large margins (indicated by the improvement in other metrics).

Original:  Conclusions
Revised:  Conclusions

Original: We have presented a controllable 3D face synthesis method that can generate high-fidelity face images based on a conditional 3DMM mesh.
Revised: We have presented a controllable 3D face synthesis method that can generate high-fidelity face images based on a conditional 3DMM mesh.

Original: We propose a novel conditional Generative Occupancy Field representation and a set of 3D losses that effectively impose 3D conditions directly on the generated
Revised: We propose a novel conditional Generative Occupancy Field representation and a set of 3D losses that effectively impose 3D conditions directly on the generated

Original: NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, and pose of the generated faces than state-of-the-art 2D counterparts.
Revised: NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, and pose of the generated faces than state-of-the-art 2D counterparts.

Original: Limitations.
Revised: Limitations.

Original: The proposed method still lacks precise texture and illumination control over the generated face images.
Revised: The proposed method still lacks precise texture and illumination control over the generated face images.

Original: It also relies heavily on the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair.
Revised: It also relies heavily on the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: The proposed work might be used to generate fake facial images for imposture or to create synthesized facial images to hide the true identities of cybercriminals.
Revised: The proposed work might be used to generate fake facial images for imposture or to create synthesized facial images to hide the true identities of cybercriminals.

Original: Defensive algorithms for recognizing such synthesized facial images should be accordingly developed.
Revised: Defensive algorithms for recognizing such synthesized facial images should be accordingly developed.

Original:  Acknowledgements
Revised:  Acknowledgements

Original: This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14204021, 14207319).
Revised: This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14204021, 14207319).

Original: We would like to thank Yu Deng for providing the evaluation code of the Disentanglement Score, and
Revised: We would like to thank Yu Deng for providing the evaluation code of the Disentanglement Score, and

Original: Jianzhu Guo for providing the pre-trained face reconstruction model.
Revised: Jianzhu Guo for providing the pre-trained face reconstruction model.

Original: We are also indebted to thank
Revised: We are also indebted to thank

Original: Xingang Pan, Han Zhou, KwanYee Lin, Jingtan Piao, and Hang Zhou for their insightful discussions.Checklist
Revised: Xingang Pan, Han Zhou, KwanYee Lin, Jingtan Piao, and Hang Zhou for their insightful discussions.

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Sec. 6.

Original: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Sec.(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Sec.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sec.(b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Sec.3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We release the code and instructions at https://keqiangsun.github.io/projects/cgof.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Sec.

Original: 4 and the supplementary materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Sec.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] The license of the assets will be released together with our code.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] We are using only public, standard benchmark datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: We did not use crowdsourcing or conduct research with human subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: Appendices A Implementation Details

Original: Network Architectures.

Original: This project is based on the released code of pi-GAN [5] and

Original: Deep3DFaceRecon [10].

Original: We use the same architectures for the generator and discriminator, as well as the main training process from the pi-GAN.

Original: To implement our proposed conditional Generative

Original: Occupancy Field (cGOF), we integrate a deep 3D face reconstruction model Deep3DFaceRecon [10] to reconstruct 3DMM parameters from the generated images for the 3DMM reconstruction parameter loss L recon .

Original: We follow [10] to adopt the popular 2009 Basel Face Model [38] for shape and texture bases, and use the expression bases of [15], built from FaceWarehouse [3].

Original: The expression components are the PCA model of the offsets between the expression meshes and the neutral meshes of individual persons.

Original: Coordinate System Alignment.

Original: We align the 3DMM meshes to pi-GAN to impose the 3D losses.

Original: We first use the pre-trained pi-GAN to generate multi-view images of various face instances in the canonical viewpoint and use the Deep3DFaceRecon to reconstruct 3DMM meshes M for the generated faces, on which we define a set of landmarks l 3D .

Original: Let K recon and E recon be the intrinsic and extrinsic matrices for 3DMM, and K pigan and E pigan the counterparts for pi-GAN respectively.

Original: Let T R2P be the transformation matrix from 3DMM to pi-GAN that we would like to estimate.

Original: We directly optimize this transformation matrix by minimizing the difference of 2D projections of a set of landmarks obtained from the original mesh M and the transformed mesh M ′ =

Original: T R2P · M , denoted as l 2D and l ′ 2D :

Original: [Equation]

Original: where l

Original: [Equation]

Original: [Equation]

Original: Background Modeling.

Original: We model the background by setting the weight of the last sample point along each ray to be w N = 1 − (cid:80) N − 1 i =1 w i , where w i =

Original: T i · (1 − exp ( − σ i δ i )) is weight of the rest of the sample points along the ray, including both the mesh-guided samples and the volume samples.

Original: Hyper-Parameters and Volume Sampling.

Original: Tab. 3 summarizes all hyper-parameters.

Original: We sample 12 coarse points evenly in the volume to and obtain N vol = 12 fine points as introduced in [31].

Original: Only

Original: N vol = 12 fine points and the N surf

Original: = 12 points sampled around the 3DMM input mesh are used for rendering and optimization with gradient backpropagation.

Original: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.

Original: To determine loss weights λ s, we start from the original Pi-GAN and add the proposed components one-by-one, as in Tab. 2 in the main paper.

Original: For each component, we first initialize the hyperparameter lambda as 1 , and empirically find a reasonable value before taking a fine-grained search.

Original: B Additional Quantitative Results B.1 Further Comparison with DiscoFaceGAN

Original: In Tab. 1 in the main paper, we have compared with DiscoFaceGAN [8] in terms of Disentangle

Original: Scores.

Original: To directly compare the 3DMM conditioning methods proposed in our paper with those in the DiscoFaceGAN [8], we introduce the conditioning methods of DiscoFaceGAN, i.e . imitative loss, and contrastive loss, into Pi-GAN to build the “Pi-GAN + DiscoFaceGAN” model.

Original: Comparison results are shown in the Tab. 4.

Original: Row 1 shows the Disentangle Scores of the officially released DiscoFaceGAN model concerning shapes, expressions, and poses.

Original: Row 2 corresponds to the constructed “Pi-GAN + DiscoFaceGAN” method.

Original: Comparing Row 1 and 2 , we find the Pi-GAN backbone enhances the disentanglement of head pose and face shape while harming the expression disentanglement.

Original: Row 3 indicates the performance of our method, which outperforms the “Pi-GAN +

Original: DiscoFaceGAN” model in all metrics by a large margin, demonstrating the efficiency of the proposed conditioning method against the DiscoFaceGAN [8].

Original: [Table]

Original: B.2 User Study

Original: We conduct a user study to add to the comparison.

Original: We follow the experiment setting of Tab. 1 in the main paper.

Original: Specifically, we provide the control results of five methods

Original: [Table]

Original: and ask a total of 21 users to rank the results according to their image quality and the controlling effects.

Original: Tab. 5 summarizes the results, where the "Average Ranking / Average Score / Ranking 1st Ratio" are reported for each method with respect to concerning factors,

Original: i

Original: [Table]

Original: e . identity, expression, pose.

Original: Our model achieves the highest average ranking and scores for all aspects

Original: [Table]

Original: indicating our model produces more perceptually compelling results and achieves better 3D controllability than other counterparts.

Original: C Additional Qualitative Results C.1 Additional Results with Pose Variations

Original: We present large pose results for GAN-Control [46] in Fig.

Original: 6, DiscoFaceGAN [8] in Fig.

Original: 7, HeadNeRF [18] in Fig.

Original: 8 and pi-GAN [5] in Fig. 9.

Original: Previous methods fail to generate plausible face images when the camera pose gets larger ( e.g . > 60 ◦ ), including NeRF-based methods (HeadNeRF and pi-GAN).

Original: Nevertheless, as shown in Fig. 10, our method produces plausible 3D consistent face images even in extremely large poses.

Original: Note that, to get rid of the view-dependent effect, we follow [14, 18] and remove the dependence of the radiance colors on the viewing direction by setting the view direction to a constant (0 , 0 , − 1) when evaluating the radiance colors of the sample points.

Original: C.2 Additional Results with Expression Variations

Original: We present more results on the expression control, comparing our method against two state-ofthe-art controllable face synthesis methods, one attribute-guided GAN-Control [46], and the other 3DMM-guided DiscoFaceGAN [8].

Original: Figs.

Original: 11 to 13 show the generated faces using GAN-Control,

Original: DiscoFaceGAN and our method respectively.

Original: For each figure, the first column shows a reference image, columns 2 to 5 show images generated with mild expressions, and columns 6 to 9 show images generated with wilder expressions.

Original: Each row corresponds to the same person, and each column corresponds to the same expression.

Original: [Table]

Original: [Table]

Original: , we can see that GAN-Control fails to preserve the identity as well as other factors of the face image ( e.g . background) when changing only the facial expression code.

Original: A few examples are highlighted in red.

Original: Moreover, we observe that with the original range of the expression parameters, the model results in only a small variation of expressions, whereas with increased perturbations, it leads to much more significant shape and identity inconsistencies.

Original: We also notice the “smiling” attribute tends to strongly correlate with the “female” and “long-hair” attributes.

Original: In Fig.

Original: 12, we can see that the DiscoFaceGAN fails to impose consistent expression control over the faces.

Original: The blue boxes highlight a few examples, where the same expression code produces different expressions in different faces.

Original: Moreover, the images generated with wild expressions may appear unnatural, such as ‘exp 5’ in Fig.

Original: In Fig. 13, we show that our model generates compelling photo-realistic face images with highly consistent, precise expression control.

Original: In each row, only expressions change while other properties remain unchanged, such as identity (shape and texture), hair, and background.

Original: In each column, all instances follow the same expression.

Original: In addition, in Fig.

Original: 14, we present examples of images generated by our model with out-of-distribution expressions, such as frowning, pouting, curling lips, smirking etc .

Original: Despite images with such expressions hardly existing in the training data, our model is still able to generate highly plausible images.

Original: Reference

Original: [Figure]

Original: Expressions

Original: [Figure]

Original: Reference

Original: Common Expressions

Original: Wilder Expressions exp exp exp exp exp exp exp exp

Original: [Figure]

Original: Reference

Original: [Figure]