Original: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
Revised: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images withsome levels of controllability, e.g., controlling the shapes, expressions, textures, andposes of the generated face images.
Revised: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images.

Original: However, these methods focus on 2D imagegenerative models, which are prone to producing inconsistent face images underlarge expression and pose changes.
Revised: However, these methods focus on 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes.

Original: In this paper, we propose a new NeRF-basedconditional 3D face synthesis framework, which enables 3D controllability overthe generated face images by imposing explicit 3D conditions from 3D face priors.
Revised: In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors.

Original: At its core is a conditional Generative Occupancy Field (cGOF) that effectivelyenforces the shape of the generated face to commit to a given 3D Morphable Model(3DMM) mesh.
Revised: At its core is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to a given 3D Morphable Model (3DMM) mesh.

Original: To achieve accurate control over fine-grained 3D face shapes ofthe synthesized image, we additionally incorporate a 3D landmark loss as well as avolume warping loss into our synthesis algorithm.
Revised: To achieve accurate control over fine-grained 3D face shapes of the synthesized image, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis algorithm.

Original: Experiments validate the effectiveness of the proposed method, which can generate high-fidelity face images andshows more precise 3D controllability than state-of-the-art 2D-based controllableface synthesis methods. 
Revised: Experiments validate the effectiveness of the proposed method, which can generate high-fidelity face images and shows more precise 3D controllability than state-ofthe-art 2D-based controllable face synthesis methods. Find code and more demo at https://keqiangsun.github.io/projects/cgof.

Original:  Introduction
Revised:  Introduction

Original: Recent success of Generative Adversarial Networks (GANs) [12] has led to tremendous progressin face image synthesis.
Revised: Recent success of Generative Adversarial Networks (GANs) [13] has led to tremendous progress in face image synthesis.

Original: State-of-the-art methods, such as StyleGAN [19, 20, 18], are capableof generating photo-realistic face images.
Revised: State-of-the-art methods, such as StyleGAN [21, 22, 20], are capable of generating photo-realistic face images.

Original: Apart from photo-realism, being able to control theappearance of the generated images is also key in many real-world applications, such as faceanimation, reenactment and free-viewpoint rendering.
Revised: Apart from photo-realism, being able to control the appearance of the generated images is also key in many real-world applications, such as face animation, reenactment, and free-viewpoint rendering.

Original: Early works on controllable face synthesis relyon external attribute annotations to learn an attribute-guided face image generation model [25, 10, 53].
Revised: Early works on controllable face synthesis rely on external attribute annotations to learn an attribute-guided face image generation model [27, 11, 55].

Original: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [24], canonly provide abstract semantic-level guidance on the generation, and the generated faces often lack 3Dgeometric consistency.
Revised: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [26], can only provide abstract semantic-level guidance on the generation, and the generated faces often lack 3D geometric consistency.

Original: Moreover, it is often much harder to obtain low-level geometric annotationsbeyond semantic labels for direct 3D supervision.
Revised: Moreover, it is often much harder to obtain low-level geometric annotations beyond semantic labels for direct 3D supervision.

Original: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as3D Morphable Models (3DMMs) [2, 36], into StyleGAN-based synthesis models, allowing for moreprecise 3D control over the generated images, including facial expressions and head poses [7, 50, 37].
Revised: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as 3D Morphable Models (3DMMs) [2, 38], into StyleGAN-based synthesis models, allowing for more precise 3D control over the generated images, including facial expressions and head poses [8, 52, 39].

Original: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces underlarge expression and pose variations due to the lack of a 3D representation, as shown in Fig.
Revised: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces under large expression and pose variations due to the lack of a 3D representation, as shown in Fig.

Original:  With the advances in differentiable rendering and neural 3D representations, a recent line of work hasexplored photo-realistic 3D face generation using only 2D image collections as training data [47, 4,43, 13, 35, 3, 55, 8].
Revised: [Figure] With the advances in differentiable rendering and neural 3D representations, a recent line of work has explored photo-realistic 3D face generation using only 2D image collections as training data [49, 5, 45, 14, 37, 4, 57, 9].

Original: Neural Radiance Fields (NeRFs)
Revised: Neural Radiance Fields (NeRFs)

Original: [29], in particular, have enabled 3D generativemodels, such as pi-GAN [4] and StyleNeRF [13], to synthesize high-fidelity, 3D consistent faces,[Figure] by training with 2D images only.
Revised: [31], in particular, have enabled 3D generative models, such as pi-GAN [5] and StyleNeRF [14], to synthesize high-fidelity, 3D consistent faces, by training with 2D images only.

Original: However, these models are purely generative and do not supportprecise 3D control over the generated faces, such as facial expressions.
Revised: However, these models are purely generative and do not support precise 3D control over the generated faces, such as facial expressions.

Original: In this work, we aim to connect these two groups of research—controllable face synthesis and 3Dgenerative models—and present a NeRF-based 3D conditional face synthesis model that can generatehigh-fidelity face images with precise 3D control over the 3D shape, expression, and pose of thegenerated faces, by leveraging a parametric 3DMM face model.
Revised: In this work, we aim to connect these two groups of research—controllable face synthesis and 3D generative models—and present a NeRF-based 3D conditional face synthesis model that can generate high-fidelity face images with precise 3D control over the 3D shape, expression, and pose of the generated faces, by leveraging a parametric 3DMM face model.

Original: Imposing precise 3D conditions onan implicit neural radiance field is non-trivial.
Revised: Imposing precise 3D conditions on an implicit neural radiance field is non-trivial.

Original: A naive baseline solution is to enforce that the input3DMM parameters can be reproduced from the generated face image via a 3DMM reconstructionmodel, similar to [7].
Revised: A naive baseline solution is to enforce that the input 3DMM parameters can be reproduced from the generated face image via a 3DMM reconstruction model, similar to [8].

Original: Unfortunately, this 3DMM parameter reconstruction loss only provides indirectsupervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3Dcontrol, as shown in Fig.
Revised: Unfortunately, this 3DMM parameter reconstruction loss only provides indirect supervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3D control, as shown in Fig.

Original: We seek to impose explicit 3D condition directly on the NeRF volume.
Revised: We seek to impose explicit 3D condition directly on the NeRF volume.

Original: To this end, we proposea conditional Generative Occupancy Field (cGOF).
Revised: To this end, we propose a conditional Generative Occupancy Field (cGOF).

Original: It consists of a mesh-guided volume samplingprocedure and a complimentary density regularizer that gradually concentrates the volume densityaround a given mesh.
Revised: It consists of a mesh-guided volume sampling procedure and a complimentary density regularizer that gradually concentrates the volume density around a given mesh.

Original: In order to achieve fine-grained 3D control, such as expression, we furtherintroduce a 3D landmark loss and a volume warping loss.
Revised: In order to achieve fine-grained 3D control, such as expression, we further introduce a 3D landmark loss and a volume warping loss.

Original: The main contributions are as follows.
Revised: The main contributions are as follows.

Original: We propose a novel controllable 3D face synthesis methodthat learns to generate high-fidelity face images with precise controllability over the 3D shape,expression, and poses, given only a collection of 2D images as training data.
Revised: We propose a novel controllable 3D face synthesis method that learns to generate high-fidelity face images with precise controllability over the 3D shape, expression, and poses, given only a collection of 2D images as training data.

Original: This is achieved bya novel conditional Generative Occupancy Field representation and a set of volumetric losses thateffectively condition the generated NeRF volumes on a parametric 3D face model.
Revised: This is achieved by a novel conditional Generative Occupancy Field representation and a set of volumetric losses that effectively condition the generated NeRF volumes on a parametric 3D face model.

Original:  Related Work
Revised:  Related Work

Original: Controllable Generative Adversarial Networks.
Revised: Controllable Generative Adversarial Networks.

Original: Generative Adversarial Nets (GANs) [12] gainedpopularity over the last decade due to their remarkable image generation ability [40, 17, 19, 20].
Revised: Generative Adversarial Nets (GANs) [13] gained popularity over the last decade due to their remarkable image generation ability [42, 19, 21, 22].

Original: Priorworks have studied disentangled representation learning in generative models [14, 50, 49, 21, 44, 1,30, 5, 6, 23].
Revised: Prior works have studied disentangled representation learning in generative models [16, 52, 51, 23, 46, 1, 32, 6, 7, 25].

Original: Existing work on controllable facial image synthesis [38, 50, 49, 7, 37] relies on 2Dimage-based generative models, which does not guarantee 3D consistency in the generated images.3D-Aware GAN.
Revised: Existing work on controllable facial image synthesis [40, 52, 51, 8, 39] relies on 2D image-based generative models, which does not guarantee 3D consistency in the generated images. 3D-Aware GAN.

Original: Another line of work in 3D-Aware generative models has looked into disentangling3D geometric information from 2D images, and learning to generate various 3D structures, suchas voxels [54, 57, 11, 30, 31, 26], meshes [47, 22], and Neural Radiance Fields (NeRFs)
Revised: Another line of work in 3D-Aware generative models has looked into disentangling 3D geometric information from 2D images, and learning to generate various 3D structures, such as voxels [56, 60, 12, 32, 33, 28], meshes [49, 24], and Neural Radiance Fields (NeRFs)

Original: [4, 13,42, 34, 33], by training on image collections using differentiable rendering.
Revised: [5, 58, 14, 44, 36, 35], by training on image collections using differentiable rendering.

Original: The key idea is touse a discriminator that encourages images rendered from random viewpoints sampled from aprior pose distribution to be indistinguishable from real images.
Revised: The key idea is to use a discriminator that encourages images rendered from random viewpoints sampled from a prior pose distribution to be indistinguishable from real images.

Original: Among these works, pi-GANhas demonstrated high-fidelity 3D synthesis results adopting a NeRF representation.
Revised: Among these works, pi-GAN has demonstrated high-fidelity 3D synthesis results adopting a NeRF representation.

Original: SofGAN [5]ensures viewpoint consistency via the proposed semantic occupancy field but requires 3D mesheswith semantic annotations for pre-training. In this work, we build on top of pi-GAN and directlylearn from unlabeled image collections with a prior 3DMM. z
Revised: SofGAN [6] z

Original: MappingNetwork
Revised: MappingNetwork

Original: [Figure]
Revised: [Figure]

Original: cGOF 
Revised: cGOF Generator

Original: [Figure]
Revised: [Figure]

Original: Radiance Integration [Figure] Landmarks 𝐥
Revised: Radiance Integration 3DMM Landmarks 𝐥

Original: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′
Revised: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′

Original: Displacement 𝐹 !"
Revised: Displacement 𝐹 !"

Original: Warped from 𝑧′ NeRF 𝑧 NeRF
Revised: Warped from 𝑧′ NeRF 𝑧 NeRF

Original: Real Image 𝐼
Revised: Real Image 𝐼

Original: Generated Image 𝐼 #
Revised: Generated Image 𝐼 #

Original: Generated Shape ξ Pose
Revised: Generated Shape ξ Pose

Original: Noise ℒ "#$ ℒ %&’($ ℒ !"
Revised: Noise ℒ "#$ ℒ %&’($ ℒ !"

Original: #$ ℒ %&’() ℛ *+((,- ℛ .
Revised: #$ ℒ %&’() ℛ *+((,- ℛ .

Original: ℒ %&’(* ℒ %&’(𝐜
Revised: ℒ %&’(* ℒ %&’(𝐜

Original: 3D Priors for Face Generation.
Revised: 3D Priors for Face Generation.

Original: To learn controllable 3D face synthesis models, researchers haveleveraged prior face models.
Revised: To learn controllable 3D face synthesis models, researchers have leveraged prior face models.

Original: 3D Morphable Models (3DMMs) [36, 2] is a widely-used parametricface model, which represents the human face as a set of PCA bases derived from 3D scans.
Revised: 3D Morphable Models (3DMMs) [38, 2] is a widely-used parametric face model, which represents the human face as a set of PCA bases derived from 3D scans.

Original: Manyprevious works [7, 50, 5, 37] have proposed to incorporate 3D priors of 3DMM into generativemodels to enable controllable 3D face synthesis.
Revised: Many previous works [8, 52, 6, 39] have proposed to incorporate 3D priors of 3DMM into generative models to enable controllable 3D face synthesis.

Original: In particular, DiscoFaceGAN [7], PIE [49] and
Revised: In particular, DiscoFaceGAN [8], PIE [51] and

Original: StyleRig [50] leverage 3DMM to establish losses to ensure 3D consistency, but still use a 2D imagebased StyleGAN generator.
Revised: StyleRig [52] leverage 3DMM to establish losses to ensure 3D consistency, but still use a 2D imagebased StyleGAN generator.

Original: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM butis trained with a reconstruction loss using annotated multi-view datasets.
Revised: HeadNeRF [18] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated multi-view datasets.

Original: Our method is trained onunannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.
Revised: Our method is trained on unannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.

Original:  Method
Revised:  Method

Original: Given a collection of single-view real-world face images Y , where each face is assumed to appearonly once without multiple views, the goal of this work is to learn a controllable 3D face synthesismodel that can generate face images with desired 3D shape, expression, and pose.
Revised: Given a collection of single-view real-world face images Y , where each face is assumed to appear only once without multiple views, the goal of this work is to learn a controllable 3D face synthesis model that can generate face images with desired 3D shape, expression, and pose.

Original: To achieve this, wepropose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric3D Morphable Model (3DMM) 
Revised: To achieve this, we propose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric 3D Morphable Model (3DMM) [38].

Original: At the core of our method is a novel conditional Generative
Revised: At the core of our method is a novel conditional Generative

Original: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D controlbased on the 3DMM, as illustrated in Figure 2.
Revised: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D control based on the 3DMM, as illustrated in Figure 2.

Original: 3.1 Preliminaries on NeRF and pi-GAN
Revised: 3.1 Preliminaries on NeRF and pi-GAN

Original: Neural Radiance Field (NeRF).
Revised: Neural Radiance Field (NeRF).

Original: NeRF [29] represents a 3D scene as a radiance field parametrizedby a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color c ∈ R 3for every 3D location, given its xyz coordinates and the viewing direction as input.
Revised: NeRF [31] represents a 3D scene as a radiance field parametrized by a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color c ∈ Rfor every 3D location, given its xyz coordinates and the viewing direction as input.

Original: To render a 2Dimage of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3Dvolume, and evaluate the density values and radiance colors at a number of sampled points along eachray.
Revised: To render a 2D image of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3D volume, and evaluate the density values and radiance colors at a number of sampled points along each ray.

Original: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =
Revised: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =

Original: o + t d , where o denotesthe camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .
Revised: o + t d , where o denotes the camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .

Original: For each 3D sample point x i , we query the NeRF network f to obtain its σ i and radiance color c i :( σ i , c i ) = f
Revised: For each 3D sample point x i , we query the NeRF network f to obtain its σ i and radiance color c i : ( σ i , c i ) = f

Original: ( x i , d ) .
Revised: ( x i , d ) .

Original: Using volume rendering [27, 29], the final color of the pixel is given by 
Revised: Using volume rendering [29, 31], the final color of the pixel is given by ˆ C ( r ) = N (cid:88) i =

Original: [Equation]
Revised: T i (1 − exp ( − σ i δ i ))

Original:  where
Revised: c i , where

Revised: T i = exp( − i − 1 (cid:88)

Original: [Equation] 1) and δ i
Revised: j = σ j δ j ) , (1) and δ i

Original: =
Revised: =

Original: t i +1 − t
Revised: t i +1 − t

Original: i is the distance between adjacent samples.
Revised: i is the distance between adjacent samples.

Original: To optimize a NeRF on a 3D scene, thephotometric loss is computed between the rendered pixels and the ground-truth images captured froma dense set of viewpoints.
Revised: To optimize a NeRF on a 3D scene, the photometric loss is computed between the rendered pixels and the ground-truth images captured from a dense set of viewpoints.

Original: Generative Radiance Field.
Revised: Generative Radiance Field.

Original: Several follow-up works have extended this representation to a generative framework [4, 13, 8, 55].
Revised: Several follow-up works have extended this representation to a generative framework [5, 14, 9, 57].

Original: Instead of using a multi-view reconstruction loss, they train adiscriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.
Revised: Instead of using a multi-view reconstruction loss, they train a discriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.

Original: Our method is built on top of pi-GAN [4], a generative radiance field model based on a SIRENmodulated NeRF representation [45].
Revised: Our method is built on top of pi-GAN [5], a generative radiance field model based on a SIRENmodulated NeRF representation [47].

Original: During training, images are rendered from random viewpointsξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).
Revised: During training, images are rendered from random viewpoints ξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).

Original: To modelgeometry and appearance variations, a random noise code z is sampled from a standard normaldistribution and mapped to a number of frequencies γ i and phase shifts
Revised: To model geometry and appearance variations, a random noise code z is sampled from a standard normal distribution and mapped to a number of frequencies γ i and phase shifts

Original: β i through a mapping network.
Revised: β i through a mapping network.

Original: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before thesinusoidal activations.
Revised: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before the sinusoidal activations.

Original: The rendered images I g =
Revised: The rendered images I g =

Original: G ( z , ξ ) and the real images I randomly sampledfrom the training dataset are then passed into the discriminator D , and the model is trained using anon-saturating GAN loss with R1 regularization following [28]:
Revised: G ( z , ξ ) and the real images I randomly sampled from the training dataset are then passed into the discriminator D , and the model is trained using a non-saturating GAN loss with R1 regularization following [30]:

Original: [Equation]
Revised: [Equation]

Original:  where f ( u ) =
Revised: (2) where f ( u ) =

Original: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.
Revised: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.

Original: 3.2 Controllable 3D Face Synthesis
Revised: 3.2 Controllable 3D Face Synthesis

Original: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [36], whichis a parametric morphable face model, where the shape, expression, texture, and other factors of aface are modeled by a set of PCA bases and coefficients ˜
Revised: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [38], which is a parametric morphable face model, where the shape, expression, texture, and other factors of a face are modeled by a set of PCA bases and coefficients ˜

Original: z
Revised: z

Original: =
Revised: =

Original: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.
Revised: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.

Original: Ourgoal is to condition the generated face image I g on a set of 3DMM parameters˜
Revised: Our goal is to condition the generated face image I g on a set of 3DMM parameters

Original:  z as well as a camerapose ξ , which specifies the desired configuration of the generated face image.
Revised: ˜ z as well as a camera pose ξ , which specifies the desired configuration of the generated face image.

Original: Baseline.
Revised: Baseline.

Original: To enforce such conditioning, following [7], we make use of an off-the-shelf pre-trainedand fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2Dimage [9].
Revised: To enforce such conditioning, following [8], we make use of an off-the-shelf pre-trained and fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2D image [10].

Original: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures thegenerated face images commit to the input condition.
Revised: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures the generated face images commit to the input condition.

Original: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which islater passed into the pi-GAN generator G to produce a face image I g .
Revised: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which is later passed into the pi-GAN generator G to produce a face image I g .

Original: Unlike [7], which additionallytrains a set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the3DMM coefficients of all the training images follow a normal distribution and simply standardizethem to obtain the corresponding normalized noise code z
Revised: Unlike [8], which additionally trains a set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the 3DMM coefficients of all the training images follow a normal distribution and simply standardize them to obtain the corresponding normalized noise code z

Original: = τ (˜ z ) =
Revised: = τ (˜ z ) =

Original: ˜
Revised: ˜

Original: L − 1 (˜ z − ˜ µ ) , where ˜ µ and ˜ L arethe mean and Cholesky decomposition component of covariance matrix of the 3DMM coefficientsestimated from all the training images using the 3DMM reconstruction model R .
Revised: L − 1 (˜ z − ˜ µ ) , where ˜ µ and ˜ L are the mean and Cholesky decomposition component of covariance matrix of the 3DMM coefficients estimated from all the training images using the 3DMM reconstruction model R .

Original: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalizedparameters and the input noise:
Revised: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalized parameters and the input noise:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMMparameter reconstruction loss.
Revised: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMM parameter reconstruction loss.

Original: Example results of this baseline model can be seen in Fig.
Revised: Example results of this baseline model can be seen in Fig.

Original: Challenges.
Revised: Challenges.

Original: There are several issues with this baseline model.
Revised: There are several issues with this baseline model.

Original: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstructionmodel R .
Revised: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstruction model R .

Original: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real facesprecisely.
Revised: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real faces precisely.

Original: Since our model generates a 3D representation for rendering, a more effective solution isto impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.
Revised: Since our model generates a 3D representation for rendering, a more effective solution is [Figure] to impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.

Original: However, the radiance field representation is highly over-parametrized [56].
Revised: However, the radiance field representation is highly over-parametrized [59].

Original: The color of a pixel is[Figure] leads poor realism and multi-view computed by integrating the radiance along the entire camera ray, whereas in many cases in reality,it should only reflect the color of the intersection point on an opaque surface.
Revised: The color of a pixel is computed by integrating the radiance along the entire camera ray, whereas in many cases in reality, it should only reflect the color of the intersection point on an opaque surface.

Original: Incorporating lossesdirectly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its convergingto an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-viewconsistency, as shown in Fig.
Revised: Incorporating losses directly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its converging to an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-view consistency, as shown in Fig.

Original: 3.3 Conditional Generative Occupancy Field
Revised: 3.3 Conditional Generative Occupancy Field

Original: In order to impose precise 3D control on the NeRF volume, we first identify an effective way ofenforcing the volume density to converge to the surface of human faces.
Revised: In order to impose precise 3D control on the NeRF volume, we first identify an effective way of enforcing the volume density to converge to the surface of human faces.

Original: Inspired by [55], wepropose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volumesampler (MgS) and a distance-aware volume density regularizer.
Revised: Inspired by [57], we propose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volume sampler (MgS) and a distance-aware volume density regularizer.

Original: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume samplingprocedure.
Revised: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume sampling procedure.

Original: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜
Revised: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜

Original: z
Revised: z

Original: = τ − 1 ( z )by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .
Revised: = τ − 1 ( z ) by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .

Original: With these coefficients, we can samplea 3D face mesh M in from the 3DMM Λ :
Revised: With these coefficients, we can sample a 3D face mesh M in from the 3DMM Λ :

Original: [Equation]
Revised: [Equation]

Original: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape andexpression in 3DMM respectively.
Revised: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape and expression in 3DMM respectively.

Original: During rendering, this mesh will serve as a 3D condition, allowingus to apply importance sampling around the surface region, and suppress the volume densities faraway from the surface.
Revised: During rendering, this mesh will serve as a 3D condition, allowing us to apply importance sampling around the surface region, and suppress the volume densities far away from the surface.

Original: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh
Revised: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh

Original: M in along the ray direction by rendering a depth map, taking into account the perspective projection.
Revised: M in along the ray direction by rendering a depth map, taking into account the perspective projection.

Original: This allows us to sample points along the ray r ( t )
Revised: This allows us to sample points along the ray r ( t )

Original: =
Revised: =

Original: o + t d within a set of uniformly spaced intervalsaround the surface intersection point of the ray and the input mesh:
Revised: o + t d within a set of uniformly spaced intervals around the surface intersection point of the ray and the input mesh:

Original: [Equation]
Revised: [Equation]

Revised: , where i = 1 , 2 , . .

Original: ) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of thesampling region.
Revised: N [Equation] [Equation] 5) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of the sampling region.

Original: We gradually shrink this margin from 0 .
Revised: We gradually shrink this margin from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 05 of the volume range.
Revised: 05 of the volume range.

Original: Furthermore, we suppress the volume density in the space far away from the given mesh using adistance-aware volume density regularizer.
Revised: Furthermore, we suppress the volume density in the space far away from the given mesh using a distance-aware volume density regularizer.

Original: Specifically, we additionally sample a sparse set of pointsalong each ray, and denote the absolute distances from these sampling points to the mesh surface asd i [Equation] i 
Revised: Specifically, we additionally sample a sparse set of points along each ray, and denote the absolute distances from these sampling points to the mesh surface as d i = | t i − t m | , i = 1 , . . .

Original: [Equation]

Original: t m | , i =

Original: [Equation]

Original: . , N vol .
Revised: , N vol .

Original: The distance-aware volume density regularizer R d is defined as:
Revised: The distance-aware volume density regularizer R d is defined as:

Original: [Equation]
Revised: [Equation]

Original: where σ i is the volume density evaluated at those sample points, and α is an inverse temperatureparameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .
Revised: where σ i is the volume density evaluated at those sample points, and α is an inverse temperature parameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .

Original: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .
Revised: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .

Original: Themesh-guided volume sampler together with the volume density regularizer effectively constrains thegenerated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.
Revised: The mesh-guided volume sampler together with the volume density regularizer effectively constrains the generated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.

Original: 3.4 Auxiliary Losses for Fine-grained 3D Control
Revised: 3.4 Auxiliary Losses for Fine-grained 3D Control

Original: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMMmesh.
Revised: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMM mesh.

Original: However, since 3DMM is only a low-dimensional PCA model of human faces withoutcapturing the fine-grained shape and texture details of real faces, enforcing the generated 3D facesto fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does notguarantee precise control over the geometric details, especially under large facial expression changes.
Revised: However, since 3DMM is only a low-dimensional PCA model of human faces without capturing the fine-grained shape and texture details of real faces, enforcing the generated 3D faces to fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does not guarantee precise control over the geometric details, especially under large facial expression changes.

Original: To enable precise geometric control, we further introduce two auxiliary loss terms.3D Landmark Loss encourages the semantically important facial landmarks to closely follow theinput condition.
Revised: To enable precise geometric control, we further introduce two auxiliary loss terms. 3D Landmark Loss encourages the semantically important facial landmarks to closely follow the input condition.

Original: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68 faciallandmarks [41, 9].
Revised: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68 facial landmarks [43, 10].

Original: Let { l k } N k k
Revised: Let { l k } N k k

Original: =1 denote the 3D locations of these landmarks on the input 3DMMmesh.
Revised: =1 denote the 3D locations of these landmarks on the input 3DMM mesh.

Original: In order to find the same set of landmarks on the generated image, we extract 3D landmarks{ ˆ l k } N k k
Revised: In order to find the same set of landmarks on the generated image, we extract 3D landmarks { ˆ l k } N k k

Original: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2Dprojections.
Revised: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2D projections.

Original: We then render the depth values of NeRF at these 2D locations using volume renderingand back-project them to 3D to obtain the 3D landmark locations of the generated volume { ˆ l ′ k } .
Revised: We then render the depth values of NeRF at these 2D locations using volume rendering and back-project them to 3D to obtain the 3D landmark locations of the generated volume { ˆ l ′ k } .

Original: Toavoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) whencalculating loss.
Revised: To avoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) when calculating loss.

Original: The landmark loss is defined among these three sets of 3D landmarks:
Revised: The landmark loss is defined among these three sets of 3D landmarks:

Original: [Equation]
Revised: [Equation]

Original: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the inputmesh at these semantically important landmark locations, which is particularly helpful in achievingprecise expression control, as validated in Tab. 2.
Revised: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the input mesh at these semantically important landmark locations, which is particularly helpful in achieving precise expression control, as validated in Tab. 2.

Original: Volume Warping Loss enforces that the NeRF volume generated with a different expression codeshould be consistent with the original volume warped by a warping field induced from the two 3DMMmeshes.
Revised: Volume Warping Loss enforces that the NeRF volume generated with a different expression code should be consistent with the original volume warped by a warping field induced from the two 3DMM meshes.

Original: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled duringtraining, M ′ =
Revised: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled during training, M ′ =

Original: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp ,and V and V ′ be their vertices respectively.
Revised: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp , and V and V ′ be their vertices respectively.

Original: We first compute the vertex displacements denoted by∆ V = V ′ − V , and render them into a 2D displacement map
Revised: We first compute the vertex displacements denoted by ∆ V = V ′ − V , and render them into a 2D displacement map

Original: F ∆ V ∈ R H
Revised: F ∆ V ∈ R H

Original: × W × 3 , where each pixel
Revised: × W × 3 , where each pixel

Original: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .
Revised: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .

Original: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities andradiance colors ( σ i , c i ) =
Revised: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities and radiance colors ( σ i , c i ) =

Original: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.
Revised: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.

Original: The superscript ( u, v ) is dropped for simplicity.
Revised: The superscript ( u, v ) is dropped for simplicity.

Original: We warp all the 3D sample points along each rayusing the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D pointsx ′ 
Revised: We warp all the 3D sample points along each ray using the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D points x ′ i

Original: i = x i + F ∆ V (assuming that all the points are already close to the mesh surface).
Revised: = x i + F ∆ V (assuming that all the points are already close to the mesh surface).

Original: We then query thesecond NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densitiesand radiance colors ( σ ′
Revised: We then query the second NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densities and radiance colors ( σ ′

Original: i , c ′ i ) =
Revised: i , c ′ i ) =

Original: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.
Revised: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.

Original: In addition, we also enforce the images rendered from the two radiance fields to be consistent.
Revised: In addition, we also enforce the images rendered from the two radiance fields to be consistent.

Original: Weintegrate the radiance colors c
Revised: We integrate the radiance colors c

Original: i of the warped points with their densities σ ′
Revised: i of the warped points with their densities σ ′

Original: i using Eq.
Revised: i using Eq.

Original: (1), and obtainan image
Revised: (1), and obtain an image

Original: ˆ I g (with all pixels).
Revised: ˆ I g (with all pixels).

Original: This image ˆ I g rendered from the second radiance field is encouraged tobe identical to the original image
Revised: This image ˆ I g rendered from the second radiance field is encouraged to be identical to the original image

Original: I g .
Revised: I g .

Original: The final warping loss is thus composed of three terms:
Revised: The final warping loss is thus composed of three terms:

Original: [Equation]
Revised: [Equation]

Original: where β d , β c and β I are the balancing weights.
Revised: where β d , β c and β I are the balancing weights.

Original: 3.5 Neural Renderer and Overall Training Objective
Revised: 3.5 Neural Renderer and Overall Training Objective

Original: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.
Revised: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.

Original: To improve the image qualitywithout introducing too much computational overhead, we leverage an auxiliary neuralrendererintroduced in [52] as a standalone module, which takes in the rendered coarse 2D image and producesa higher resolution version of it ( 512 × 512 ). We take their pre-trained model on the FFHQ datasetand finetune it on the CelebA-HQ dataset. This neural renderersignificantly improves the quality ofthe generated images without compromising much 3D controllability, as shown in Tab. 1.
Revised: To improve the image quality without introducing too much computational overhead, we leverage an auxiliary neural renderer

Original: [Table]
Revised: [Table]

Original: Similar to RegNeRF [32], we further impose a smoothness regularizer R smooth on the generated NeRFvolume by penalizing the L2 differences of neighboring pixels in both the depth maps and normalmaps (referred to as R depthsmooth and R normsmooth below).
Revised: Similar to RegNeRF [34], we further impose a smoothness regularizer R smooth on the generated NeRF volume by penalizing the L2 differences of neighboring pixels in both the depth maps and normal maps (referred to as R depthsmooth and R norm smooth below).

Original: The final training objective is thus given by
Revised: The final training objective is thus given by

Original: [Equation]
Revised: [Equation]

Original: (9) where the λ ’s specify the weights for each term.
Revised: (9) where the λ ’s specify the weights for each term.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We train our model on the CelebA [24] dataset, which consists of 200 k celebrity faceimages with various poses, expressions, and lighting conditions.
Revised: We train our model on the CelebA [26] dataset, which consists of 200 k celebrity face images with various poses, expressions, and lighting conditions.

Original: We use the cropped images anddo not make use of their facial attribute annotations .
Revised: We use the cropped images and do not make use of their facial attribute annotations .

Original: The neural renderer module was originallytrained on FFHQ dataset, which uses a slightly different cropping scheme.
Revised: The neural renderer module was originally trained on FFHQ dataset, which uses a slightly different cropping scheme.

Original: Therefore, we finetunetheir pre-trained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.
Revised: Therefore, we finetune their pre-trained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.

Original: Implementation Details.
Revised: Implementation Details.

Original: We provide implementation details in the supplementary material, includingnetwork architectures and hyper-parameters.
Revised: We provide implementation details in the supplementary material, including network architectures and hyper-parameters.

Original: Here, we highlight a few important ones.
Revised: Here, we highlight a few important ones.

Original: We build ourmodel on top of the implementation of pi-GAN [4], which uses a SIREN -based NeRF representation.
Revised: We build our model on top of the implementation of pi-GAN [5], which uses a SIREN -based NeRF representation.

Original: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for thedensity regularizer.
Revised: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for the density regularizer.

Original: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.
Revised: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.

Original: Metrics.
Revised: Metrics.

Original: (1) Disentanglement Score (DS) .
Revised: (1) Disentanglement Score (DS) .

Original: Following [7], we evaluate 3D controllability using a3DMM Disentanglement Score.
Revised: Following [8], we evaluate 3D controllability using a 3DMM Disentanglement Score.

Original: To measure the DS on one disentangled property z i ( e.g . shape,expression or pose), we randomly sample a set of z i from a normal distribution while keepingother codes fixed { z j } , j ̸ = i , and render images.
Revised: To measure the DS on one disentangled property z i ( e.g . shape, expression or pose), we randomly sample a set of z i from a normal distribution while keeping other codes fixed { z j } , j ̸ = i , and render images.

Original: We re-estimate the 3DMM parameters
Revised: We re-estimate the 3DMM parameters

Original: ˆ z i and{ ˆ z j } from these rendered images and compute DS i = (cid:81) j,j ̸
Revised: ˆ z i and { ˆ z j } from these rendered images and compute DS i = (cid:81) j,j ̸

Original: = i σ ˆ z
Revised: = i σ ˆ z

Original: i /σ
Revised: i /σ

Original: ˆ z j , where σ ˆ z i and σ ˆ z j denotevariance of the reconstructed parameters of the measured property
Revised: ˆ z j , where σ ˆ z i and σ ˆ z j denote variance of the reconstructed parameters of the measured property

Original: ˆ z
Revised: ˆ z

Original: i and the rest ˆ z j , respectively.
Revised: i and the rest ˆ z j , respectively.

Original: Ahigher DS score indicates that the model achieves more precise controllability on that property whilekeeping the others unchanged.
Revised: A higher DS score indicates that the model achieves more precise controllability on that property while keeping the others unchanged.

Original: (2) Chamfer Distance (CD) .
Revised: (2) Chamfer Distance (CD) .

Original: We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely thegenerated 3D face follows the condition.
Revised: We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely the generated 3D face follows the condition.

Original: (3) Landmark Distance (LD).
Revised: (3) Landmark Distance (LD).

Original: To evaluate the fine-grainedcontrollability of our model, we detect the 3D landmarks in the generated images using [9] andcompute the Euclidean distance between the detected landmarks and the ones extracted from the input3DMM vertices.
Revised: To evaluate the fine-grained controllability of our model, we detect the 3D landmarks in the generated images using [10] and compute the Euclidean distance between the detected landmarks and the ones extracted from the input 3DMM vertices.

Original: (4) Landmark Correlation (LC).
Revised: (4) Landmark Correlation (LC).

Original: We measure the precision of expression control, bycomputing the 3D landmark displacements of two faces generated with a pair of random expressioncodes, and reporting the correlation score between these displacements and those obtained from thetwo 3DMM meshes.
Revised: We measure the precision of expression control, by computing the 3D landmark displacements of two faces generated with a pair of random expression codes, and reporting the correlation score between these displacements and those obtained from the two 3DMM meshes.

Original: (5) Frechet Inception Distance (FID) .
Revised: (5) Frechet Inception Distance (FID) .

Original: The quality of the generated face imagesis measured using Frechet Inception Distance [15] with an ImageNet-pretrained Inception-V3 [48]feature extractor.
Revised: The quality of the generated face images is measured using Frechet Inception Distance [17] with an ImageNet-pretrained Inception-V3 [50] feature extractor.

Original: We align images with 68 landmarks [41, 46, 39] to avoid domain shift.
Revised: We align images with 68 landmarks [43, 48, 41] to avoid domain shift.

Original: 4.1 Qualitative Evaluation
Revised: 4.1 Qualitative Evaluation

Original: [Figure]

Original: Fig.
Revised: Fig.

Original: 4 shows some example images generated by our proposed cGOF model.
Revised: 4 shows some example images generated by our proposed cGOF model.

Original: In particular, wevisualize the generated faces by varying only one of the controlled properties at a time, includingfacial expression, head pose, and identity–combining shape and texture as in [7].
Revised: In particular, we visualize the generated faces by varying only one of the controlled properties at a time, including [Figure] facial expression, head pose, and identity–combining shape and texture as in [8].

Original: Our model is ableto generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner,even in the presence of large expression and pose variations.
Revised: Our model is able to generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner, even in the presence of large expression and pose variations.

Original: In contrast, existing methods tend to[Figure] produce inconsistent face images as shown in Fig.
Revised: In contrast, existing methods tend to produce inconsistent face images as shown in Fig.

Original: Please refer to the supplementary materials formore qualitative results and side-by-side comparisons against state-of-the-art methods.
Revised: Please refer to the supplementary materials for more qualitative results and side-by-side comparisons against state-of-the-art methods.

Original: 4.2 Quantitative Evaluation
Revised: 4.2 Quantitative Evaluation

Original: We compare our method to several state-of-the-art methods.
Revised: We compare our method to several state-of-the-art methods.

Original: E4E [51] and GAN-Control [44] arepurely 2D controllable face image synthesis methods, which are based on facial attribute annotations.
Revised: E4E [53] and GAN-Control [46] are purely 2D controllable face image synthesis methods, which are based on facial attribute annotations.

Original: DiscoFaceGAN [7], PIE [49] and StyleRig [50] leverage 3DMM priors but still use a 2D image-based
Revised: DiscoFaceGAN [8], PIE [51] and StyleRig [52] leverage 3DMM priors but still use a 2D image-based

Original: StyleGAN generator.
Revised: StyleGAN generator.

Original: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM but is trainedwith a reconstruction loss using annotated datasets, whereas our method is a 3D generative modeltrained on unannotated single-view images only.
Revised: HeadNeRF [18] uses a 3D NeRF representation as well as 3DMM but is trained with a reconstruction loss using annotated datasets, whereas our method is a 3D generative model trained on unannotated single-view images only.

Original: We compare the controllability accuracy using the Disentanglement Score (see Col 3 to 5 of Tab. 1).
Revised: We compare the controllability accuracy using the Disentanglement Score (see Col 3 to 5 of Tab. 1).

Original: Our method achieves significantly better disentanglement and more precise control in terms of shapes,expressions, and poses, compared to existing methods.
Revised: Our method achieves significantly better disentanglement and more precise control in terms of shapes,

Original: Note that since PIE and StyleRig do not haveopen-sourced code, we compute the score on the 167 generated images provided on the PIE project

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: +ℒ $%&’# (

Original: [Table]
Revised: [Equation]

Original: ( +ℒ (%)*+ )
Revised: ℒ (%)*+ )* + MgS +

Original: [Table]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: #’$- +

Revised: [Equation]

Revised: (-/ +

Revised: [Equation]

Original: + MgS +ℛ ( +ℛ ,-’’*+#’$- +ℒ .(-/ +ℒ 0"$) +ℛ ,-’’*+(%)*+ gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM.
Revised: [Equation] $) [Equation] *+ gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM.

Original: Refer to Sec.
Revised: Refer to Sec.

Original: 4.3 for more details.
Revised: 4.3 for more details.

Original: ∗ Note that we do not use the depth loss in the final model, which compares the depths rendered from NeRF and 3DMM. 
Revised: ∗ Note that we do not use the depth loss in the final model, which compares the depths rendered from NeRF and 3DMM. expressions, and poses, compared to existing methods.

Original:  page 1 .
Revised: Note that since PIE and StyleRig do not have open-sourced code, we compute the score on the 167 generated images provided on the PIE project page 2 .

Original: We also compare the FID scores in Col 6 of Tab. 1.
Revised: We also compare the FID scores in Col 6 of Tab. 1.

Original: Our model achieves plausible imagequality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.
Revised: Our model achieves plausible image quality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: We conduct ablation studies to validate the effectiveness of each component in our model by addingthem one by one, and evaluating the generated images with the four metrics specified above.
Revised: We conduct ablation studies to validate the effectiveness of each component in our model by adding them one by one, and evaluating the generated images with the four metrics specified above.

Original: Tab. 2and Fig.
Revised: Tab.and Fig.

Original: 5 illustrate the studies and show that all the components improve the precision of 3Dcontrollable image generation.
Revised: 5 illustrate the studies and show that all the components improve the precision of 3D controllable image generation.

Original: In particular, the mesh-guided volume sampler (MgS) significantlyimproves the effectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).
Revised: In particular, the mesh-guided volume sampler (MgS) significantly improves the effectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).

Original: The 3D landmarkloss L ldmk (denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) furtherenhance the control over fine-grained geometric details.
Revised: The 3D landmark loss L ldmk (denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) further enhance the control over fine-grained geometric details.

Original: We also report FIDs of the outputs of thecGOF with the image resolution of 128 × 128 .
Revised: We also report FIDs of the outputs of the cGOF with the image resolution of 128 × 128 .

Original: Overall, although the geometric regularizers lead toslightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability bylarge margins (indicated by the improvement in other metrics).
Revised: Overall, although the geometric regularizers lead to slightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability by large margins (indicated by the improvement in other metrics).

Original:  Conclusions
Revised:  Conclusions

Original: We have presented a controllable 3D face synthesis method that can generate high-fidelity face imagesbased on a conditional 3DMM mesh.
Revised: We have presented a controllable 3D face synthesis method that can generate high-fidelity face images based on a conditional 3DMM mesh.

Original: We propose a novel conditional Generative Occupancy Fieldrepresentation and a set of 3D losses that effectively impose 3D conditions directly on the generated
Revised: We propose a novel conditional Generative Occupancy Field representation and a set of 3D losses that effectively impose 3D conditions directly on the generated

Original: NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, andpose of the generated faces than state-of-the-art 2D counterparts.
Revised: NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, and pose of the generated faces than state-of-the-art 2D counterparts.

Original: Limitations.
Revised: Limitations.

Original: The proposed methodstill lacks precise texture and illumination control over the generated face images.
Revised: The proposed method still lacks precise texture and illumination control over the generated face images.

Original: It also relies heavilyon the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair.
Revised: It also relies heavily on the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: The proposed work might be used to generate fake facial images for imposture orto create synthesized facial images to hide the true identities of cybercriminals.
Revised: The proposed work might be used to generate fake facial images for imposture or to create synthesized facial images to hide the true identities of cybercriminals.

Original: Defensive algorithmsfor recognizing such synthesized facial images should be accordingly developed.  
Revised: Defensive algorithms for recognizing such synthesized facial images should be accordingly developed.  Acknowledgements

Revised: This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14204021, 14207319).

Revised: We would like to thank Yu Deng for providing the evaluation code of the Disentanglement Score, and

Revised: Jianzhu Guo for providing the pre-trained face reconstruction model.

Revised: We are also indebted to thank

Revised: Xingang Pan, Han Zhou, KwanYee Lin, Jingtan Piao, and Hang Zhou for their insightful discussions.

Revised: Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [Yes] See Sec. 6.

Revised: • Did you include the license to the code and datasets?

Revised: [No] The code and the data are proprietary.

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: Please do not modify the questions and only use the provided macros for your answers.

Revised: Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] See Sec.(c)

Revised: Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See Sec.(d)

Revised: Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] See Sec.(b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See Sec.3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] We release the code and instructions at https://keqiangsun.github.io/projects/cgof.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See Sec.

Revised: 4 and the supplementary materials.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Sec.4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [Yes] The license of the assets will be released together with our code.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [No] We are using only public, standard benchmark datasets.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A]

Revised: We did not use crowdsourcing or conduct research with human subjects.

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: Appendices A Implementation Details

Revised: Network Architectures.

Revised: This project is based on the released code of pi-GAN [5] and

Revised: Deep3DFaceRecon [10].

Revised: We use the same architectures for the generator and discriminator, as well as the main training process from the pi-GAN.

Revised: To implement our proposed conditional Generative

Revised: Occupancy Field (cGOF), we integrate a deep 3D face reconstruction model Deep3DFaceRecon [10] to reconstruct 3DMM parameters from the generated images for the 3DMM reconstruction parameter loss L recon .

Revised: We follow [10] to adopt the popular 2009 Basel Face Model [38] for shape and texture bases, and use the expression bases of [15], built from FaceWarehouse [3].

Revised: The expression components are the PCA model of the offsets between the expression meshes and the neutral meshes of individual persons.

Revised: Coordinate System Alignment.

Revised: We align the 3DMM meshes to pi-GAN to impose the 3D losses.

Revised: We first use the pre-trained pi-GAN to generate multi-view images of various face instances in the canonical viewpoint and use the Deep3DFaceRecon to reconstruct 3DMM meshes M for the generated faces, on which we define a set of landmarks l 3D .

Revised: Let K recon and E recon be the intrinsic and extrinsic matrices for 3DMM, and K pigan and E pigan the counterparts for pi-GAN respectively.

Revised: Let T R2P be the transformation matrix from 3DMM to pi-GAN that we would like to estimate.

Revised: We directly optimize this transformation matrix by minimizing the difference of 2D projections of a set of landmarks obtained from the original mesh M and the transformed mesh M ′ =

Revised: T R2P · M , denoted as l 2D and l ′ 2D :

Revised: [Equation]

Revised: where l

Revised: [Equation]

Revised: [Equation]

Revised: Background Modeling.

Revised: We model the background by setting the weight of the last sample point along each ray to be w N = 1 − (cid:80) N − 1 i =1 w i , where w i =

Revised: T i · (1 − exp ( − σ i δ i )) is weight of the rest of the sample points along the ray, including both the mesh-guided samples and the volume samples.

Revised: Hyper-Parameters and Volume Sampling.

Revised: Tab. 3 summarizes all hyper-parameters.

Revised: We sample 12 coarse points evenly in the volume to and obtain N vol = 12 fine points as introduced in [31].

Revised: Only

Revised: N vol = 12 fine points and the N surf

Revised: = 12 points sampled around the 3DMM input mesh are used for rendering and optimization with gradient backpropagation.

Revised: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.

Revised: To determine loss weights λ s, we start from the original Pi-GAN and add the proposed components one-by-one, as in Tab. 2 in the main paper.

Revised: For each component, we first initialize the hyperparameter lambda as 1 , and empirically find a reasonable value before taking a fine-grained search.

Revised: B Additional Quantitative Results B.1 Further Comparison with DiscoFaceGAN

Revised: In Tab. 1 in the main paper, we have compared with DiscoFaceGAN [8] in terms of Disentangle

Revised: Scores.

Revised: To directly compare the 3DMM conditioning methods proposed in our paper with those in the DiscoFaceGAN [8], we introduce the conditioning methods of DiscoFaceGAN, i.e . imitative loss, and contrastive loss, into Pi-GAN to build the “Pi-GAN + DiscoFaceGAN” model.

Revised: Comparison results are shown in the Tab. 4.

Revised: Row 1 shows the Disentangle Scores of the officially released DiscoFaceGAN model concerning shapes, expressions, and poses.

Revised: Row 2 corresponds to the constructed “Pi-GAN + DiscoFaceGAN” method.

Revised: Comparing Row 1 and 2 , we find the Pi-GAN backbone enhances the disentanglement of head pose and face shape while harming the expression disentanglement.

Revised: Row 3 indicates the performance of our method, which outperforms the “Pi-GAN +

Revised: DiscoFaceGAN” model in all metrics by a large margin, demonstrating the efficiency of the proposed conditioning method against the DiscoFaceGAN [8].

Revised: [Table]

Revised: B.2 User Study

Revised: We conduct a user study to add to the comparison.

Revised: We follow the experiment setting of Tab. 1 in the main paper.

Revised: Specifically, we provide the control results of five methods

Revised: [Table]

Revised: and ask a total of 21 users to rank the results according to their image quality and the controlling effects.

Revised: Tab. 5 summarizes the results, where the "Average Ranking / Average Score / Ranking 1st Ratio" are reported for each method with respect to concerning factors,

Revised: i

Revised: [Table]

Revised: e . identity, expression, pose.

Revised: Our model achieves the highest average ranking and scores for all aspects

Revised: [Table]

Revised: indicating our model produces more perceptually compelling results and achieves better 3D controllability than other counterparts.

Revised: C Additional Qualitative Results C.1 Additional Results with Pose Variations

Revised: We present large pose results for GAN-Control [46] in Fig.

Revised: 6, DiscoFaceGAN [8] in Fig.

Revised: 7, HeadNeRF [18] in Fig.

Revised: 8 and pi-GAN [5] in Fig. 9.

Revised: Previous methods fail to generate plausible face images when the camera pose gets larger ( e.g . > 60 ◦ ), including NeRF-based methods (HeadNeRF and pi-GAN).

Revised: Nevertheless, as shown in Fig. 10, our method produces plausible 3D consistent face images even in extremely large poses.

Revised: Note that, to get rid of the view-dependent effect, we follow [14, 18] and remove the dependence of the radiance colors on the viewing direction by setting the view direction to a constant (0 , 0 , − 1) when evaluating the radiance colors of the sample points.

Revised: C.2 Additional Results with Expression Variations

Revised: We present more results on the expression control, comparing our method against two state-ofthe-art controllable face synthesis methods, one attribute-guided GAN-Control [46], and the other 3DMM-guided DiscoFaceGAN [8].

Revised: Figs.

Revised: 11 to 13 show the generated faces using GAN-Control,

Revised: DiscoFaceGAN and our method respectively.

Revised: For each figure, the first column shows a reference image, columns 2 to 5 show images generated with mild expressions, and columns 6 to 9 show images generated with wilder expressions.

Revised: Each row corresponds to the same person, and each column corresponds to the same expression.

Revised: [Table]

Revised: [Table]

Revised: , we can see that GAN-Control fails to preserve the identity as well as other factors of the face image ( e.g . background) when changing only the facial expression code.

Revised: A few examples are highlighted in red.

Revised: Moreover, we observe that with the original range of the expression parameters, the model results in only a small variation of expressions, whereas with increased perturbations, it leads to much more significant shape and identity inconsistencies.

Revised: We also notice the “smiling” attribute tends to strongly correlate with the “female” and “long-hair” attributes.

Revised: In Fig.

Revised: 12, we can see that the DiscoFaceGAN fails to impose consistent expression control over the faces.

Revised: The blue boxes highlight a few examples, where the same expression code produces different expressions in different faces.

Revised: Moreover, the images generated with wild expressions may appear unnatural, such as ‘exp 5’ in Fig.

Revised: In Fig. 13, we show that our model generates compelling photo-realistic face images with highly consistent, precise expression control.

Revised: In each row, only expressions change while other properties remain unchanged, such as identity (shape and texture), hair, and background.

Revised: In each column, all instances follow the same expression.

Revised: In addition, in Fig.

Revised: 14, we present examples of images generated by our model with out-of-distribution expressions, such as frowning, pouting, curling lips, smirking etc .

Revised: Despite images with such expressions hardly existing in the training data, our model is still able to generate highly plausible images.

Revised: Reference

Revised: [Figure]

Revised: Expressions

Revised: [Figure]

Revised: Reference

Revised: Common Expressions

Revised: Wilder Expressions exp exp exp exp exp exp exp exp

Revised: [Figure]

Revised: Reference

Revised: [Figure]