Original: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
Revised: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images withsome levels of controllability, e.g., controlling the shapes, expressions, textures, andposes of the generated face images.
Revised: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images withsome levels of controllability, e.g., controlling the shapes, expressions, textures, andposes of the generated face images.

Original: However, these methods focus on 2D imagegenerative models, which are prone to producing inconsistent face images underlarge expression and pose changes.
Revised: However, these methods focus on 2D imagegenerative models, which are prone to producing inconsistent face images underlarge expression and pose changes.

Original: In this paper, we propose a new NeRF-basedconditional 3D face synthesis framework, which enables 3D controllability overthe generated face images by imposing explicit 3D conditions from 3D face priors.
Revised: In this paper, we propose a new NeRF-basedconditional 3D face synthesis framework, which enables 3D controllability overthe generated face images by imposing explicit 3D conditions from 3D face priors.

Original: At its core is a conditional Generative Occupancy Field (cGOF) that effectivelyenforces the shape of the generated face to commit to a given 3D Morphable Model(3DMM) mesh.
Revised: At its core is a conditional Generative Occupancy Field (cGOF) that effectivelyenforces the shape of the generated face to commit to a given 3D Morphable Model(3DMM) mesh.

Original: To achieve accurate control over fine-grained 3D face shapes ofthe synthesized image, we additionally incorporate a 3D landmark loss as wellas a volume warping loss into our synthesis algorithm.
Revised: To achieve accurate control over fine-grained 3D face shapes ofthe synthesized image, we additionally incorporate a 3D landmark loss as well as avolume warping loss into our synthesis algorithm.

Original: Experiments validate theeffectiveness of the proposed method, which is able to generate high-fidelity faceimages and shows more precise 3D controllability than state-of-the-art 2D-basedcontrollable face synthesis methods.
Revised: Experiments validate the effectiveness of the proposed method, which can generate high-fidelity face images andshows more precise 3D controllability than state-of-the-art 2D-based controllableface synthesis methods.

Original:  Introduction
Revised:  Introduction

Original: Recent success of Generative Adversarial Networks (GANs) [13] has led to tremendous progressin face image synthesis.
Revised: Recent success of Generative Adversarial Networks (GANs) [12] has led to tremendous progressin face image synthesis.

Original: State-of-the-art methods, such as StyleGAN [20, 21, 19], are capableof generating photo-realistic face images.
Revised: State-of-the-art methods, such as StyleGAN [19, 20, 18], are capableof generating photo-realistic face images.

Original: Apart from photo-realism, being able to control theappearance of the generated images is also key in many real-world applications, such as faceanimation, reenactment and free-viewpoint rendering.
Revised: Apart from photo-realism, being able to control theappearance of the generated images is also key in many real-world applications, such as faceanimation, reenactment and free-viewpoint rendering.

Original: Early works on controllable face synthesis relyon external attribute annotations to learn an attribute-guided face image generation model [26, 11, 51].
Revised: Early works on controllable face synthesis relyon external attribute annotations to learn an attribute-guided face image generation model [25, 10, 53].

Original: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [25], canonly providean abstract semantic-level guidance on the generation, and the generated faces oftenlack 3D geometric consistency.
Revised: However, these attributes, such as “big nose”, “chubby” and “smiling” in CelebA dataset [24], canonly provide abstract semantic-level guidance on the generation, and the generated faces often lack 3Dgeometric consistency.

Original: Moreover, it is often much harder to obtain low-level geometricannotations beyond semantic labels for direct 3D supervision.
Revised: Moreover, it is often much harder to obtain low-level geometric annotationsbeyond semantic labels for direct 3D supervision.

Original: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as3D Morphable Models (3DMMs) [2, 37], into StyleGAN-based synthesis models, allowing for moreprecise 3D control over the generated images, including facial expressions and head poses [8, 48, 38].
Revised: Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as3D Morphable Models (3DMMs) [2, 36], into StyleGAN-based synthesis models, allowing for moreprecise 3D control over the generated images, including facial expressions and head poses [7, 50, 37].

Original: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces underlarge expression and pose variations due to the lack of a 3D representation, as shown in Fig.
Revised: Despite their impressive image quality, these models still tend to produce 3D inconsistent faces underlarge expression and pose variations due to the lack of a 3D representation, as shown in Fig.

Original: With the advances in differentiable rendering and neural 3D representations, a recent line of work hasexplored photo-realistic 3D face generation using only 2D image collections as training data [45, 5,42, 14, 36, 4, 53, 9].
Revised: With the advances in differentiable rendering and neural 3D representations, a recent line of work hasexplored photo-realistic 3D face generation using only 2D image collections as training data [47, 4,43, 13, 35, 3, 55, 8].

Original: Neural Radiance Fields (NeRFs) [30], in particular, have enabled 3D generativemodels, such as pi-GAN [5] and StyleNeRF [14], to synthesize high-fidelity, 3D consistent faces,
Revised: Neural Radiance Fields (NeRFs) [29], in particular, have enabled 3D generativemodels, such as pi-GAN [4] and StyleNeRF [13], to synthesize high-fidelity, 3D consistent faces,

Original: [Figure]
Revised: [Figure]

Original: by training with 2D images only.
Revised: by training with 2D images only.

Original: However, these models are purely generative and do not supportprecise 3D control over the generated faces, such as facial expressions.
Revised: However, these models are purely generative and do not supportprecise 3D control over the generated faces, such as facial expressions.

Original: In this work, we aim to connect these two groups of research—controllable face synthesis and 3Dgenerative models—and present a NeRF-based 3D conditional face synthesis model that can generatehigh-fidelity face images with precise 3D control over the 3D shape, expression, and pose of thegenerated faces, by leveraging a parametric 3DMM face model.
Revised: In this work, we aim to connect these two groups of research—controllable face synthesis and 3Dgenerative models—and present a NeRF-based 3D conditional face synthesis model that can generatehigh-fidelity face images with precise 3D control over the 3D shape, expression, and pose of thegenerated faces, by leveraging a parametric 3DMM face model.

Original: Imposing precise 3D conditions onan implicit neural radiance field is non-trivial.
Revised: Imposing precise 3D conditions onan implicit neural radiance field is non-trivial.

Original: A naive baseline solution is to enforce that the input3DMM parameters can be reproduced from the generated face image via a 3DMM reconstructionmodel, similar to [8].
Revised: A naive baseline solution is to enforce that the input3DMM parameters can be reproduced from the generated face image via a 3DMM reconstructionmodel, similar to [7].

Original: Unfortunately, this 3DMM parameter reconstruction loss only provides indirectsupervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3Dcontrol, as shown in Fig.
Revised: Unfortunately, this 3DMM parameter reconstruction loss only provides indirectsupervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3Dcontrol, as shown in Fig.

Original: We seek to impose explicit 3D condition directly on the NeRF volume.
Revised: We seek to impose explicit 3D condition directly on the NeRF volume.

Original: To this end, we proposeconditional Generative Occupancy Field (cGOF).
Revised: To this end, we proposea conditional Generative Occupancy Field (cGOF).

Original: It consists of a mesh-guided volume samplingprocedure and a complimentary density regularizer that gradually concentrate the volume densityaround a given mesh.
Revised: It consists of a mesh-guided volume samplingprocedure and a complimentary density regularizer that gradually concentrates the volume densityaround a given mesh.

Original: In order to achieve fine-grained 3D control, such as expression, we furtherintroduce a 3D landmark loss and a volume warping loss.
Revised: In order to achieve fine-grained 3D control, such as expression, we furtherintroduce a 3D landmark loss and a volume warping loss.

Original: The main contributions are as follows.
Revised: The main contributions are as follows.

Original: We propose a novel controllable 3D face synthesis methodthat learns to generate high-fidelity face images with precise controllability over the 3D shape,expression, and pose, given only a collection of 2D images as training data.
Revised: We propose a novel controllable 3D face synthesis methodthat learns to generate high-fidelity face images with precise controllability over the 3D shape,expression, and poses, given only a collection of 2D images as training data.

Original: This is achieved bya novel conditional Generative Occupancy Field representation and a set of volumetric losses thateffectively condition the generated NeRF volumes on a parametric 3D face model.
Revised: This is achieved bya novel conditional Generative Occupancy Field representation and a set of volumetric losses thateffectively condition the generated NeRF volumes on a parametric 3D face model.

Original:  Related Work
Revised:  Related Work

Original: Controllable Generative Adversarial Networks.
Revised: Controllable Generative Adversarial Networks.

Original: Generative Adversarial Nets (GANs) [13] gainedpopularity over the last decade due to their remarkable image generation ability [39, 18, 20, 21].
Revised: Generative Adversarial Nets (GANs) [12] gainedpopularity over the last decade due to their remarkable image generation ability [40, 17, 19, 20].

Original: Many prior works have studied disentangled representation learning in generative models [15, 48, 47,22, 43, 1, 31, 6, 7, 24].
Revised: Priorworks have studied disentangled representation learning in generative models [14, 50, 49, 21, 44, 1,30, 5, 6, 23].

Original: Existing work on controllable facial image synthesis [48, 47, 8] relies on 2Dimage-based generative models, which does not guarantee 3D consistency in the generated images.3D-Aware GAN.
Revised: Existing work on controllable facial image synthesis [38, 50, 49, 7, 37] relies on 2Dimage-based generative models, which does not guarantee 3D consistency in the generated images.3D-Aware GAN.

Original: Another line of work in 3D-Aware generative models has looked into disentangling3D geometric information from 2D images, and learn to generate various 3D structures, such asvoxels [52, 55, 12, 31, 32, 27], meshes [45, 23], and Neural Radiance Fields (NeRFs)
Revised: Another line of work in 3D-Aware generative models has looked into disentangling3D geometric information from 2D images, and learning to generate various 3D structures, suchas voxels [54, 57, 11, 30, 31, 26], meshes [47, 22], and Neural Radiance Fields (NeRFs)

Original: [5, 14, 41,35, 34], by training on image collections using differentiable rendering.
Revised: [4, 13,42, 34, 33], by training on image collections using differentiable rendering.

Original: The key idea is to use adiscriminator that encourages images rendered from random viewpoints sampled from a prior posedistribution are indistinguishable from real images.
Revised: The key idea is touse a discriminator that encourages images rendered from random viewpoints sampled from aprior pose distribution to be indistinguishable from real images.

Original: Among these works, pi-GAN has demonstratedhigh-fidelity 3D synthesis results adopting a NeRF representation.
Revised: Among these works, pi-GANhas demonstrated high-fidelity 3D synthesis results adopting a NeRF representation.

Original: SofGAN [6] ensures viewpointconsistency via the proposed semantic occupancy field but requires 3D meshes with semanticannotations for pre-training.
Revised: SofGAN [5]ensures viewpoint consistency via the proposed semantic occupancy field but requires 3D mesheswith semantic annotations for pre-training.

Original: In this work, we build on top of pi-GAN and directly learn fromunlabeled image collections with a prior 3DMM.
Revised: In this work, we build on top of pi-GAN and directlylearn from unlabeled image collections with a prior 3DMM.

Original: z
Revised: z

Original: MappingNetwork
Revised: MappingNetwork

Original: [Figure]
Revised: [Figure]

Original: cGOFGenerator
Revised: cGOF

Original: [Figure]
Revised: [Figure]

Original: Radiance Integration
Revised: Radiance Integration

Original: [Figure]
Revised: [Figure]

Original: Landmarks 𝐥
Revised: Landmarks 𝐥

Original: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′
Revised: NeRF Landmarks "𝐥 𝑧′ NeRF 𝑉 𝑉′

Original: Displacement 𝐹 !"
Revised: Displacement 𝐹 !"

Original: Warped from 𝑧′ NeRF 𝑧 NeRF
Revised: Warped from 𝑧′ NeRF 𝑧 NeRF

Original: Real Image 𝐼
Revised: Real Image 𝐼

Original: Generated Image 𝐼 #
Revised: Generated Image 𝐼 #

Original: Generated Shape ξ Pose
Revised: Generated Shape ξ Pose

Original: Noise ℒ "#$ ℒ %&’($ ℒ !"
Revised: Noise ℒ "#$ ℒ %&’($ ℒ !"

Original: #$ ℒ %&’() ℛ *+((,- ℛ .
Revised: #$ ℒ %&’() ℛ *+((,- ℛ .

Original: ℒ %&’(* ℒ %&’(𝐜
Revised: ℒ %&’(* ℒ %&’(𝐜

Original: 3D Priors for Face Generation.
Revised: 3D Priors for Face Generation.

Original: To learn controllable 3D face synthesis models, researchers haveleveraged prior face models.
Revised: To learn controllable 3D face synthesis models, researchers haveleveraged prior face models.

Original: 3D Morphable Models (3DMMs) [37, 3] iswidely-used a  parametricface model, which represents  human face as a set of PCA bases derived from 3D scans.
Revised: 3D Morphable Models (3DMMs) [36, 2] is a widely-used parametricface model, which represents the human face as a set of PCA bases derived from 3D scans.

Original: Many previousworks [8, 48, 6] have proposed to incorporate 3D priors of 3DMM into generative models to enablecontrollable 3D face synthesis.
Revised: Manyprevious works [7, 50, 5, 37] have proposed to incorporate 3D priors of 3DMM into generativemodels to enable controllable 3D face synthesis.

Original: In particular, DiscoFaceGAN [8], PIE [47] and StyleRig [48] leverage3DMM to establish losses to ensure 3D consistency, but still use a 2D image-based StyleGANgenerator.
Revised: In particular, DiscoFaceGAN [7], PIE [49] and StyleRig [50] leverage 3DMM to establish losses to ensure 3D consistency, but still use a 2D imagebased StyleGAN generator.

Original: HeadNeRF [17] uses a 3D NeRF representation as well as 3DMM but is trained witha reconstruction loss using annotated multi-view datasets.
Revised: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM butis trained with a reconstruction loss using annotated multi-view datasets.

Original: Our method is trained on unannotatedsingle-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.
Revised: Our method is trained onunannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.

Original:  Method
Revised:  Method

Original: Given a collection of single-view real-world face images Y , where each face is assumed to appearonly once without multiple views, the goal of this work is to learn a controllable 3D face synthesismodel that can generate face images with desired 3D shape, expression, and pose.
Revised: Given a collection of single-view real-world face images Y , where each face is assumed to appearonly once without multiple views, the goal of this work is to learn a controllable 3D face synthesismodel that can generate face images with desired 3D shape, expression, and pose.

Original: To achieve this, wepropose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric3D Morphable Model (3DMM)[37].
Revised: To achieve this, wepropose a NeRF-based conditional 3D generative model G and incorporate priors from a parametric3D Morphable Model (3DMM)

Original: At the core of our method is a novel conditional Generative
Revised: At the core of our method is a novel conditional Generative

Original: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D controlbased on the 3DMM, as illustrated in Figure 2.
Revised: Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D controlbased on the 3DMM, as illustrated in Figure 2.

Original: 3.1 Preliminaries on NeRF and pi-GAN
Revised: 3.1 Preliminaries on NeRF and pi-GAN

Original: Neural Radiance Field (NeRF).
Revised: Neural Radiance Field (NeRF).

Original: NeRF [30] represents a 3D scene as a radiance field parametrizedby a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color c ∈ R 3for every 3D location, given its xyz coordinates and the viewing direction as input.
Revised: NeRF [29] represents a 3D scene as a radiance field parametrizedby a Multi-Layer Perceptron (MLP) that predicts a density value σ ∈ R and a radiance color c ∈ R 3for every 3D location, given its xyz coordinates and the viewing direction as input.

Original: To render a 2Dimage of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3Dvolume, and evaluate the density values and radiance colors at a number of sampled points along eachray.
Revised: To render a 2Dimage of the scene from an arbitrary camera viewpoint ξ , we cast rays through the pixels into the 3Dvolume, and evaluate the density values and radiance colors at a number of sampled points along eachray.

Original: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =
Revised: Specifically, for each pixel, the camera ray r can be expressed as r ( t ) =

Original: o + t d , where o denotesthe camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .
Revised: o + t d , where o denotesthe camera origin, d the direction, and t defines a sample point within near and far bounds t n and t f .

Original: For each 3D sampling point x i , we query the NeRF network f to obtain its σ i and radiance color c i :( σ i , c i ) =
Revised: For each 3D sample point x i , we query the NeRF network f to obtain its σ i and radiance color c i :( σ i , c i ) =

Original: f
Revised: f

Original: ( x i , d ) .
Revised: ( x i , d ) .

Original: Using volume rendering [28, 30], the final color of the pixel is given by
Revised: Using volume rendering [27, 29], the final color of the pixel is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: 1) and δ i = t i +1 − t i is the distance between adjacent samples.

Original: To optimize a NeRF on a 3D scene, the photometric loss is computed by comparing the renderedpixels with the ground-truth images captured from a dense set of viewpoints.
Revised: To optimize a NeRF on a 3D scene, thephotometric loss is computed between the rendered pixels and the ground-truth images captured froma dense set of viewpoints.

Original: Generative Radiance Field.
Revised: Generative Radiance Field.

Original: Several follow-up works have extended this representation to a generative framework [5, 14, 9, 53].
Revised: Several follow-up works have extended this representation to a generative framework [4, 13, 8, 55].

Original: Instead of using a multi-view reconstruction loss, they train adiscriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.
Revised: Instead of using a multi-view reconstruction loss, they train adiscriminator that encourages images rendered from randomly sampled viewpoints to be indistinguishable from real images, allowing the model to be trained with only single-view image collections.

Original: Our method is built on top of pi-GAN [5], a generative radiance field model based on a SIRENmodulated NeRF representation [44].
Revised: Our method is built on top of pi-GAN [4], a generative radiance field model based on a SIRENmodulated NeRF representation [45].

Original: During training, images are rendered from random viewpointsξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).
Revised: During training, images are rendered from random viewpointsξ sampled from a predefined pose distribution p ξ (estimated from the training dataset).

Original: To modelgeometry and appearance variations, a random noise code z is sampled from a standard normaldistribution and mapped to a number of frequencies γ i and phase shifts
Revised: To modelgeometry and appearance variations, a random noise code z is sampled from a standard normaldistribution and mapped to a number of frequencies γ i and phase shifts

Original: β i through a mapping network.
Revised: β i through a mapping network.

Original: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before thesinusoidal activations.
Revised: These encodings are used to modulate the outputs of each layer in the NeRF MLP f before thesinusoidal activations.

Original: The rendered images I g =
Revised: The rendered images I g =

Original: G ( z , ξ ) and the real images I randomly sampledfrom the training dataset are then passed into the discriminator D , and the model is trained using anon-saturating GAN loss with R1 regularization following [29]:
Revised: G ( z , ξ ) and the real images I randomly sampledfrom the training dataset are then passed into the discriminator D , and the model is trained using anon-saturating GAN loss with R1 regularization following [28]:

Original: [Equation]
Revised: [Equation]

Original: where f ( u )
Revised: where f ( u )

Original: =
Revised: =

Original: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.
Revised: − log(1 + exp( − u )) and λ is a hyperparameter balancing the regularization term.

Original: .2 Controllable 3D Face Synthesis
Revised: 3.2 Controllable 3D Face Synthesis

Original: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [37], whichis a parametric morphable face model, where the shape, expression, pose and other factors of a faceare modeled by a set of PCA bases and coefficients ˜
Revised: In order to learn a controllable 3D face synthesis model, we leverage priors of a 3DMM [36], whichis a parametric morphable face model, where the shape, expression, texture, and other factors of aface are modeled by a set of PCA bases and coefficients ˜

Original: z
Revised: z

Original: =
Revised: =

Original: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.
Revised: (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) respectively.

Original: Our goalis to condition the generated face images I g on a set of 3DMM parameters ˜
Revised: Ourgoal is to condition the generated face image I g on a set of 3DMM parameters ˜

Original: z as well as a camera poseξ , which specify the desired configuration of the generated face image.
Revised: z as well as a camerapose ξ , which specifies the desired configuration of the generated face image.

Original: Baseline.
Revised: Baseline.

Original: To enforce such conditioning, following [8], we make use of an off-the-shelf pretrainedand fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2Dimage [10].
Revised: To enforce such conditioning, following [7], we make use of an off-the-shelf pre-trainedand fixed 3DMM reconstruction model R that predicts a set of 3DMM parameters from a single 2Dimage [9].

Original: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures thegenerated face imagesto commit to the input condition.
Revised: This allows us to incorporate a 3DMM parameter reconstruction loss that ensures thegenerated face images commit to the input condition.

Original: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which islater passed into the pi-GAN generator G to produce a face image I g .
Revised: Specifically, we sample a random noise z ∈ R d from a standard normal distribution p z , which is laterpassed into the pi-GAN generator G to produce a face image I g .

Original: Unlike [8], which additionallytrains a set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the3DMM coefficients of all the training images follow a normal distribution and simply standardizethem with their mean and standard deviation to obtain the corresponding normalized noise codez =
Revised: Unlike [7], which additionally trainsa set of VAEs to map the noise z to meaningful 3DMM coefficients ˜ z , we assume that the 3DMMcoefficients of all the training images follow a normal distribution and simply standardize them toobtain the corresponding normalized noise code z

Original:  τ (˜ z ) =
Revised: = τ (˜ z ) =

Original: (˜ z − ˜ µ ) / ˜ σ , where ˜ µ and ˜ σ are the mean and standard deviation of the 3DMM coefficientsestimated from all the training images using the 3DMM reconstruction model R .
Revised: (˜ z − ˜ µ ) / ˜ σ , where ˜ µ and ˜ σ are the meanand standard deviation of the 3DMM coefficients estimated from all the training images using the3DMM reconstruction model R .

Original: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalizedparameters and the input noise:
Revised: During training, we predict 3DMM parameters from the generated image using the reconstruction model R , normalize it and minimize a reconstruction loss between the predicted normalizedparameters and the input noise:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMMparameter reconstruction loss.
Revised: This results in a baseline model, where the conditioning is implicitly imposed through the 3DMMparameter reconstruction loss.

Original: Example results of this baseline model can be seen in Fig.
Revised: Example results of this baseline model can be seen in Fig.

Original: Challenges.
Revised: Challenges.

Original: There are several issues with this baseline model.
Revised: There are several issues with this baseline model.

Original: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstructionmodel R .
Revised: First, the effectiveness of the conditioning depends heavily on the accuracy and generalization performance of the 3DMM reconstructionmodel R .

Original: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real facesprecisely.
Revised: Fundamentally, these parameters only describe an extremely abstract, semantic representation of human faces, which is insufficient to govern the fine-grained geometric details in real facesprecisely.

Original: Since our model generates a 3D representation for rendering, a more effective solution isto impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.
Revised: Since our model generates a 3D representation for rendering, a more effective solution isto impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images.

Original: However, the radiance field representation is highly over-parametrized [54].
Revised: However, the radiance field representation is highly over-parametrized [56].

Original: The color of a pixel iscomputed by integrating the radiance along the entire camera ray, whereas in many cases in reality,
Revised: The color of a pixel is

Original: [Figure]
Revised: [Figure]

Original: with a depth loss leads to ambiguous surfaces indicated by high variance of the rendering ) Rendering semi-transparent volumes leadsto poor realism and multi-view consistency. it should only reflect the color of the intersection point on a opaque surface.
Revised: leads poor realism and multi-view computed by integrating the radiance along the entire camera ray, whereas in many cases in reality,it should only reflect the color of the intersection point on an opaque surface.

Original: Incorporating lossesdirectly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its convergingto an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-viewconsistency, as shown in Fig.3 (a).
Revised: Incorporating lossesdirectly on this over-parametrized 3D volume ( e.g . a depth loss) tends to hamper its convergingto an opaque surface, resulting in a semi-transparent volume that no longer guarantees multi-viewconsistency, as shown in Fig.

Original: 3.3 Conditional Generative Occupancy Field
Revised: 3.3 Conditional Generative Occupancy Field

Original: In order to impose precise 3D control on the NeRF volume, we first identify an effective way ofenforcing the volume density to converge to the surface of human faces.
Revised: In order to impose precise 3D control on the NeRF volume, we first identify an effective way ofenforcing the volume density to converge to the surface of human faces.

Original: Inspired by [53], wepropose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volumesampler (MgS) and a distance-aware volume density regularizer.
Revised: Inspired by [55], wepropose conditional Generative Occupancy Fields (cGOF), which consists of a mesh-guided volumesampler (MgS) and a distance-aware volume density regularizer.

Original: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume samplingprocedure.
Revised: The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume samplingprocedure.

Original: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜
Revised: Given a random noise z , we first obtain the corresponding 3DMM coefficients ˜

Original: z
Revised: z

Original: = τ − 1 ( z )by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .
Revised: = τ − 1 ( z )by denormalization, which consists of (˜ z shape , ˜ z exp , ˜ z tex , ˜ z else ) .

Original: With these coefficients, we can samplea 3D face mesh M in from the 3DMM Λ :
Revised: With these coefficients, we can samplea 3D face mesh M in from the 3DMM Λ :

Original: [Equation]
Revised: [Equation]

Original: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape andexpression in 3DMM respectively.
Revised: where M mean is the mean shape, and B shape and B exp are the deformation bases for shape andexpression in 3DMM respectively.

Original: During rendering, this mesh will serve as  3D condition, allowingus to apply importance sampling around the surface region, and suppress the volume densities faraway from the surface.
Revised: During rendering, this mesh will serve as a 3D condition, allowingus to apply importance sampling around the surface region, and suppress the volume densities faraway from the surface.

Original: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh
Revised: Specifically, for each pixel within the face region, we find its distance t m to the input 3DMM mesh

Original: M in along the ray direction by rendering a depth map, taking into account the perspective projection.
Revised: M in along the ray direction by rendering a depth map, taking into account the perspective projection.

Original: This allows us to sample points along the ray r ( t )
Revised: This allows us to sample points along the ray r ( t )

Original: =
Revised: =

Original: o + t d within a set of uniformly spaced intervalsaround the surface intersection point of the ray and the input mesh:
Revised: o + t d within a set of uniformly spaced intervalsaround the surface intersection point of the ray and the input mesh:

Original: [Equation]
Revised: [Equation]

Original: 5) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of thesampling region.
Revised: ) where N surf is the number of sampled points, and δ is a parameter controlling the “thickness” of thesampling region.

Original: We gradually shrink this margin from 0 .
Revised: We gradually shrink this margin from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 05 of the volume range.
Revised: 05 of the volume range.

Original: Furthermore, we suppress the volume density in the space far away from the given mesh using adistance-aware volume density regularizer.
Revised: Furthermore, we suppress the volume density in the space far away from the given mesh using adistance-aware volume density regularizer.

Original: Specifically, we additionally sample a sparse set of pointsalong each ray, and denote the absolute distances from these sampling points to the mesh surface asd i =
Revised: Specifically, we additionally sample a sparse set of pointsalong each ray, and denote the absolute distances from these sampling points to the mesh surface asd i [Equation]

Original: | t i − t m | , i = 1 , . . .
Revised: i [Equation] t m | , i = [Equation]

Original:  , N vol .
Revised: . , N vol .

Original: The distance-aware volume density regularizer R d is defined as:
Revised: The distance-aware volume density regularizer R d is defined as:

Original: [Equation]
Revised: [Equation]

Original: ) where σ i is the volume density evaluated at those sample points, and α is an inverse temperatureparameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .
Revised: where σ i is the volume density evaluated at those sample points, and α is an inverse temperatureparameter (set to 20 ), which controls the strength of the penalty response to increased distance ∆ d i .

Original: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .
Revised: Note that this penalty only applies to sample points beyond the surface sampling range δ 2 .

Original: Themesh-guided volume sampler together with the volume density regularizer effectively constrains thegenerated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.
Revised: Themesh-guided volume sampler together with the volume density regularizer effectively constrains thegenerated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab. 2.

Original: 3.4 Auxiliary Losses for Fine-grained 3D Control
Revised: 3.4 Auxiliary Losses for Fine-grained 3D Control

Original: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMMmesh.
Revised: So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMMmesh.

Original: However, since 3DMM is only a low-dimensional PCA model of human faces withoutcapturing the fine-grained shape and texture details of real faces, enforcing the generated 3D facesto fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does notguarantee precise control over the geometric details, especially under large facial expression changes.
Revised: However, since 3DMM is only a low-dimensional PCA model of human faces withoutcapturing the fine-grained shape and texture details of real faces, enforcing the generated 3D facesto fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does notguarantee precise control over the geometric details, especially under large facial expression changes.

Original: To enable precise geometric control, we further introduce two auxiliary loss terms.3D Landmark Loss encourages the semantically important facial landmarks to closely follow theinput condition.
Revised: To enable precise geometric control, we further introduce two auxiliary loss terms.3D Landmark Loss encourages the semantically important facial landmarks to closely follow theinput condition.

Original: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68facial landmarks [40, 10].
Revised: Specifically, we identify a set of 3DMM vertices that correspond to N k = 68 faciallandmarks [41, 9].

Original: Let { l k } N k k
Revised: Let { l k } N k k

Original: =1 denote the 3D locations of these landmarks on the input3DMM mesh.
Revised: =1 denote the 3D locations of these landmarks on the input 3DMMmesh.

Original: In order to find the same set of landmarks on the generated image, we extract 3Dlandmarks { ˆ l k } N k k
Revised: In order to find the same set of landmarks on the generated image, we extract 3D landmarks{ ˆ l k } N k k

Original: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain theprojections.
Revised: =1 from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2Dprojections.

Original: We then render the depth values of NeRF at these 2D locations using volume renderingand back-project them to 3D to obtain the 3D landmark locations of the synthesis volume { ˆ l ′ k } .
Revised: We then render the depth values of NeRF at these 2D locations using volume renderingand back-project them to 3D to obtain the 3D landmark locations of the generated volume { ˆ l ′ k } .

Original: Toavoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) whencalculating loss.
Revised: Toavoid the impact of occlusion, we skip the facial contour landmarks (numbered from 1 to 17) whencalculating loss.

Original: The landmark loss is defined among these three sets of 3D landmarks:
Revised: The landmark loss is defined among these three sets of 3D landmarks:

Original: [Equation]
Revised: [Equation]

Original: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the inputmesh at these semantically important landmark locations, which is particularly helpful in achievingprecise expression control, as validated in Tab. 2.
Revised: Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the inputmesh at these semantically important landmark locations, which is particularly helpful in achievingprecise expression control, as validated in Tab. 2.

Original: Volume Warping Loss enforces that the NeRF volume generated with a different expression codeshould be consistent with the original volume warped by a warping field induced from the two 3DMMmeshes.
Revised: Volume Warping Loss enforces that the NeRF volume generated with a different expression codeshould be consistent with the original volume warped by a warping field induced from the two 3DMMmeshes.

Original: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled duringtraining, M ′ =
Revised: More concretely, let M = Λ( z shape , z exp ) be a 3DMM mesh randomly sampled duringtraining, M ′ =

Original: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp ,and V and V ′ be their vertices respectively.
Revised: Λ( z shape , z ′ exp ) be a mesh of the same face but with a different expression code z ′ exp ,and V and V ′ be their vertices respectively.

Original: We first compute the vertex displacements denoted by∆ V = V ′ − V , and render them into a 2D displacement map
Revised: We first compute the vertex displacements denoted by∆ V = V ′ − V , and render them into a 2D displacement map

Original: F ∆ V ∈ R H
Revised: F ∆ V ∈ R H

Original: × W × 3 , where each pixel
Revised: × W × 3 , where each pixel

Original: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .
Revised: F ( u,v ) ∆ V defines the 3D displacement of its intersection point on the mesh M .

Original: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities andradiance colors ( σ i , c i ) =
Revised: For each pixel ( u, v ) in the image I g generated from the first NeRF, we obtain the densities andradiance colors ( σ i , c i ) =

Original: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.
Revised: f ( x i , d , z ) of the N surf mesh-guided sample points x i described above.

Original: The superscript ( u, v ) is dropped for simplicity.
Revised: The superscript ( u, v ) is dropped for simplicity.

Original: We warp all the 3D sample points along each rayusing the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D pointsx ′
Revised: We warp all the 3D sample points along each rayusing the same 3D displacement value rendered at the pixel F ∆ V to obtain the warped 3D pointsx ′

Original: i =
Revised: i =

Original: x i + F ∆ V (assuming that all the points are already close to the mesh surface).
Revised: x i + F ∆ V (assuming that all the points are already close to the mesh surface).

Original: We then query thesecond NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densitiesand radiance colors ( σ ′
Revised: We then query thesecond NeRF generated with z ′ exp at these warped 3D locations x ′ i to obtain another set of densitiesand radiance colors ( σ ′

Original: i , c ′ i ) =
Revised: i , c ′ i ) =

Original: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.
Revised: f ( x ′ i , d ′ , z ′ ) , and encourage them to stay close to the original ones.

Original: In addition, we also enforce the images rendered from the two radiance fields to be consistent.
Revised: In addition, we also enforce the images rendered from the two radiance fields to be consistent.

Original: Weintegrate the radiance colors c
Revised: Weintegrate the radiance colors c

Original: i of the warped points with their densities σ ′
Revised: i of the warped points with their densities σ ′

Original: i using Eq.
Revised: i using Eq.

Original: (1), and obtainan image
Revised: (1), and obtainan image

Original: ˆ
Revised: ˆ

Original: I g (with all pixels).
Revised: I g (with all pixels).

Original: This image ˆ I g rendered from the second radiance field is encouraged tobe identical to the original image
Revised: This image ˆ I g rendered from the second radiance field is encouraged tobe identical to the original image

Original: I g .
Revised: I g .

Original: The final warping loss is thus composed of three terms:
Revised: The final warping loss is thus composed of three terms:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where β d , β c and β I are the balancing weights.
Revised: where β d , β c and β I are the balancing weights.

Original: 3.5 Neural Renderer and Overall Training Objective
Revised: 3.5 Neural Renderer and Overall Training Objective

Original: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.
Revised: Due to the computation- and memory-expensive volume rendering process, the quality of the generated images is bound to a small resolution ( 64 × 64 ) in training.

Original: To improve the image quality withoutintroducing too much computational overhead, we leverage an auxiliary neural renderer introducedin [50] as a standalone module, which takes in the rendered coarse 2D image and produces a higherresolution version of it ( 512 × 512 ).
Revised: To improve the image qualitywithout introducing too much computational overhead, we leverage an auxiliary neural rendererintroduced in [52] as a standalone module, which takes in the rendered coarse 2D image and producesa higher resolution version of it ( 512 × 512 ).

Original: We take their pretrained model on the FFHQ dataset and finetuneit on the Celeba-HQ dataset.
Revised: We take their pre-trained model on the FFHQ datasetand finetune it on the CelebA-HQ dataset.

Original: This neural renderer significantly improves the quality of the generatedimages without compromising much 3D controllability, as shown in Tab. 1.
Revised: This neural renderer significantly improves the quality ofthe generated images without compromising much 3D controllability, as shown in Tab. 1.

Original: [Table]
Revised: [Table]

Original: Similar to RegNeRF [33], we further impose a smoothness regularizer R smooth on the generated NeRFvolume by penalizing the L2 differences of neighboring pixels in both the depth maps and normalmaps (referred to as R depthsmooth and R normsmooth below).
Revised: Similar to RegNeRF [32], we further impose a smoothness regularizer R smooth on the generated NeRFvolume by penalizing the L2 differences of neighboring pixels in both the depth maps and normalmaps (referred to as R depthsmooth and R normsmooth below).

Original: The final training objective is thus given by
Revised: The final training objective is thus given by

Original: [Equation]
Revised: [Equation]

Original:  where the λ ’s specify the weights for each term.
Revised: (9) where the λ ’s specify the weights for each term.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We train our model on the CelebA [25] dataset, which consists of 200 k celebrity faceimages with various poses, expressions and lighting conditions.
Revised: We train our model on the CelebA [24] dataset, which consists of 200 k celebrity faceimages with various poses, expressions, and lighting conditions.

Original: We use the cropped images anddo not make use of their facial attribute annotations .
Revised: We use the cropped images anddo not make use of their facial attribute annotations .

Original: The neural renderer module was originallytrained on FFHQ dataset, which uses a slightly different cropping scheme.
Revised: The neural renderer module was originallytrained on FFHQ dataset, which uses a slightly different cropping scheme.

Original: Therefore, we finetunetheir pretrained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.
Revised: Therefore, we finetunetheir pre-trained model on the CelebA-HQ subset of CelebA containing 30 k high-res images.

Original: Implementation Details.
Revised: Implementation Details.

Original: We provide implementation details in the supplementary material, includingnetwork architectures and hyper-parameters.
Revised: We provide implementation details in the supplementary material, includingnetwork architectures and hyper-parameters.

Original: Here, we highlight a few important ones.
Revised: Here, we highlight a few important ones.

Original: We build ourmodel on top of the implementation of pi-GAN [5], which uses a SIREN -based NeRF representation.
Revised: We build ourmodel on top of the implementation of pi-GAN [4], which uses a SIREN -based NeRF representation.

Original: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for thedensity regularizer.
Revised: We sample N surf = 12 points around the 3DMM input mesh and N vol = 12 coarse points for thedensity regularizer.

Original: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.
Revised: The final model is trained for 72 hours on 8 GeForce GTX TITAN X GPUs.

Original: Metrics.
Revised: Metrics.

Original: (1) Disentanglement Score (DS) .
Revised: (1) Disentanglement Score (DS) .

Original: Following [8], we evaluate 3D controllability using a3DMM Disentanglement Score.
Revised: Following [7], we evaluate 3D controllability using a3DMM Disentanglement Score.

Original: To measure the DS on one disentangled property z i ( e.g . shape,expression or pose), we randomly sample a set of z i from a normal distribution while keeping othercodes fixed { z j } , j ̸ = i , and render images.
Revised: To measure the DS on one disentangled property z i ( e.g . shape,expression or pose), we randomly sample a set of z i from a normal distribution while keepingother codes fixed { z j } , j ̸ = i , and render images.

Original: We re-estimate the 3DMM parameters
Revised: We re-estimate the 3DMM parameters

Original: ˆ z i and { ˆ z j } fromthese rendered images and compute DS i = (cid:81) j,j ̸
Revised: ˆ z i and{ ˆ z j } from these rendered images and compute DS i = (cid:81) j,j ̸

Original: = i σ ˆ z
Revised: = i σ ˆ z

Original: i /σ
Revised: i /σ

Original: ˆ z j , where σ ˆ z i and σ ˆ z j denote variance ofthe reconstructed parameters of the measured property
Revised: ˆ z j , where σ ˆ z i and σ ˆ z j denotevariance of the reconstructed parameters of the measured property

Original: ˆ z
Revised: ˆ z

Original: i and the rest ˆ z j , respectively.
Revised: i and the rest ˆ z j , respectively.

Original: A higher DSscore indicates that the model achieves more precise controllability on that property while keeping theothers unchanged.
Revised: Ahigher DS score indicates that the model achieves more precise controllability on that property whilekeeping the others unchanged.

Original: (2) Chamfer Distance  .
Revised: (2) Chamfer Distance (CD) .

Original: We report the bi-diretional Chamfer Distance between thegenerated face surface and the input 3DMM to measure how closely the generated 3D face follows thecondition.
Revised: We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely thegenerated 3D face follows the condition.

Original: (3) Landmark Error .
Revised: (3) Landmark Distance (LD).

Original: To evaluate the fine-grained controllability of our model, we detectthe 3D landmarks in the generated images using [10] and compute the Euclidean distance between thedetected landmarks and the ones extracted from the input 3DMM vertices.
Revised: To evaluate the fine-grainedcontrollability of our model, we detect the 3D landmarks in the generated images using [9] andcompute the Euclidean distance between the detected landmarks and the ones extracted from the input3DMM vertices.

Original: (4) Landmark Correlation .
Revised: (4) Landmark Correlation (LC).

Original: We measure the precision of expression control, by computing the 3D landmark displacements of twofaces generated with a pair of random expression codes, and reporting the correlation score betweenthese displacements and those obtained from the two 3DMM meshes.
Revised: We measure the precision of expression control, bycomputing the 3D landmark displacements of two faces generated with a pair of random expressioncodes, and reporting the correlation score between these displacements and those obtained from thetwo 3DMM meshes.

Original: (5) Frechet Inception Distance(FID) .
Revised: (5) Frechet Inception Distance (FID) .

Original: The quality of the generated face images is measured using Frechet Inception Distance [16]with an ImageNet-pretrained Inception-V3 [46] feature extractor.
Revised: The quality of the generated face imagesis measured using Frechet Inception Distance [15] with an ImageNet-pretrained Inception-V3 [48]feature extractor.

Revised: We align images with 68 landmarks [41, 46, 39] to avoid domain shift.

Original:  Qualitative Evaluation
Revised: 4.1 Qualitative Evaluation

Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 4 shows some example images generated by our proposed cGOF model.
Revised: 4 shows some example images generated by our proposed cGOF model.

Original: In particular, wevisualize the generated faces by varying only one of the controlled properties at a time, includingfacial expression, head pose, and identity–combining shape and texture as in [8].
Revised: In particular, wevisualize the generated faces by varying only one of the controlled properties at a time, includingfacial expression, head pose, and identity–combining shape and texture as in [7].

Original: Our model is ableto generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner,even in the presence of large expression and pose variations.
Revised: Our model is ableto generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner,even in the presence of large expression and pose variations.

Original: In contrast, existing methods tend to
Revised: In contrast, existing methods tend to

Original: [Figure]
Revised: [Figure]

Original: produce inconsistent face images as shown in Fig.
Revised: produce inconsistent face images as shown in Fig.

Original: Please refer to the supplementary materials formore qualitative results and side-by-side comparisons against state-of-the-art methods.
Revised: Please refer to the supplementary materials formore qualitative results and side-by-side comparisons against state-of-the-art methods.

Original: 4.2 Quantitative Evaluation
Revised: 4.2 Quantitative Evaluation

Original: We compare our method to several state-of-the-art methods.
Revised: We compare our method to several state-of-the-art methods.

Original: E4E [49] and GAN-Control [43] arepurely 2D controllable face image synthesis methods, which are based on facial attribute annotations.
Revised: E4E [51] and GAN-Control [44] arepurely 2D controllable face image synthesis methods, which are based on facial attribute annotations.

Original: DiscoFaceGAN [8], PIE [47] and StyleRig [48] leverage 3DMM priors but still use a 2D image-based
Revised: DiscoFaceGAN [7], PIE [49] and StyleRig [50] leverage 3DMM priors but still use a 2D image-based

Original: StyleGAN generator.
Revised: StyleGAN generator.

Original: HeadNeRF [17] uses a 3D NeRF representation as well as 3DMM but is trainedwith a reconstruction loss using annotated multi-view datasets, whereas our method is a 3D generativemodel trained on unannotated single-view images only.
Revised: HeadNeRF [16] uses a 3D NeRF representation as well as 3DMM but is trainedwith a reconstruction loss using annotated multi-view datasets, whereas our method is a 3D generativemodel trained on unannotated single-view images only.

Original: We compare the controllability accuracy using the Disentanglement Score (see  Tab. 1).
Revised: We compare the controllability accuracy using the Disentanglement Score (see Col 3 to 5 of Tab. 1).

Original: Our methodachieves significantly better disentanglement and more precise control in terms of shapes, expressions,
Revised: Our method achieves significantly better disentanglement and more precise control in terms of shapes,expressions, and poses, compared to existing methods.

Revised: Note that since PIE and StyleRig do not haveopen-sourced code, we compute the score on the 167 generated images provided on the PIE project

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ( +ℒ (%)*+ )

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: + MgS +ℛ ( +ℛ ,-’’*+#’$- +ℒ .(-/ +ℒ 0"$) +ℛ ,-’’*+(%)*+ gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM. Refer to Sec. 4.3 for more details. ∗ Note that we do not use the depth loss in the final model, which compares the depths rendered from NeRF and 3DMM.

Original: [Table]
Revised: page 1 .

Original: [Equation]
Revised: We also compare the FID scores in Col 6 of Tab. 1. Our model achieves plausible imagequality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: We conduct ablation studies to validate the effectiveness of each component in our model by addingthem one by one, and evaluate the generated images with the four metrics specified above.
Revised: We conduct ablation studies to validate the effectiveness of each component in our model by addingthem one by one, and evaluating the generated images with the four metrics specified above.

Original: Tab. 2 and Fig.
Revised: Tab. 2and Fig.

Original: 5 illustrate the studies and show that all the components improve the precision of 3D controllableimage generation.
Revised: 5 illustrate the studies and show that all the components improve the precision of 3Dcontrollable image generation.

Original: In particular, the mesh-guided volume sampler (MgS) significantly improves theeffectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).
Revised: In particular, the mesh-guided volume sampler (MgS) significantlyimproves the effectiveness of the 3DMM condition (denoted as ‘+ MgS’ in Tab. 2).

Original: The 3D landmark loss L ldmk(denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) further enhancethe control over fine-grained geometric details.
Revised: The 3D landmarkloss L ldmk (denoted as ‘+ L ldmk ’) and the volume warping loss L warp (denoted as ‘+ L warp ’) furtherenhance the control over fine-grained geometric details.

Revised: We also report FIDs of the outputs of thecGOF with the image resolution of 128 × 128 .

Original: Overall, although the geometric regularizers lead toslightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability bylarge margins (indicated by the improvement in other metrics).
Revised: Overall, although the geometric regularizers lead toslightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability bylarge margins (indicated by the improvement in other metrics).

Original:  Conclusions
Revised:  Conclusions

Original: We have presented a controllable 3D face synthesis method that can generate high-fidelity face imagesbased on a conditional 3DMM mesh.
Revised: We have presented a controllable 3D face synthesis method that can generate high-fidelity face imagesbased on a conditional 3DMM mesh.

Original: We propose a novel conditional Generative Occupancy Fieldrepresentation and a set of 3D losses that effectively impose 3D conditions directly on the generated
Revised: We propose a novel conditional Generative Occupancy Fieldrepresentation and a set of 3D losses that effectively impose 3D conditions directly on the generated

Original: NeRF volume, which enable much more precise 3D controllability over 3D shape, expression, poseof the generated faces than state-of-the-art 2D counterparts.
Revised: NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, andpose of the generated faces than state-of-the-art 2D counterparts.

Original: Limitations.
Revised: Limitations.

Original: The proposed method stilllacks precise texture and illumination control over the generated face images.
Revised: The proposed methodstill lacks precise texture and illumination control over the generated face images.

Original: It also relies heavilyon the prior 3DMM, and does not generate high-fidelity shape details, such as wrinkles and hair.
Revised: It also relies heavilyon the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: The proposed work might be used to generate fake facial images for imposture orto create synthesized facial images to hide the true identities of cyber criminals.
Revised: The proposed work might be used to generate fake facial images for imposture orto create synthesized facial images to hide the true identities of cybercriminals.

Original: Defensive algorithmson recognizing such synthesized facial images should be accordingly developed.Checklist
Revised: Defensive algorithmsfor recognizing such synthesized facial images should be accordingly developed.

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See ?? .•

Original: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Sec. 5(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Sec. 5(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sec.

Original: 3(b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Sec.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We plan to

Original: submit the code, data, and instructions in the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] See Sec.

Original: 4 and the supplementary materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Sec.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [Yes] The license of the assets will be

Original: released together with our code.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [No] We are using only public, standard benchmark

Original: datasets.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: We did not use crowdsourcing or conduct research with human

Original: subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]