Original: Exploiting Multiple Guidance From 3DMM For Face Reenactment
Revised: Exploiting Multiple Guidance From 3DMM For Face Reenactment

Original: Abstract
Revised: Abstract

Original: Face reenactment aims to transfer the expression and pose from a driving face to a source face.
Revised: Face reenactment aims to transfer the expression and pose from a driving face to a source face.

Original: Great progress has been made on this task with the recent success of deep generative models.
Revised: Great progress has been made on this task with the recent success of deep generative models.

Original: However, it remains challenging when the two faces hold a significant pose discrepancy.
Revised: However, it remains challenging when the two faces hold a significant pose discrepancy.

Original: Identity change and im- age distortion arise as pose discrepancy increases.
Revised: Identity change and im- age distortion arise as pose discrepancy increases.

Original: To tackle these problems, we propose to exploit multiple guidance de- rived from the 3D morphable face model (3DMM).
Revised: To tackle these problems, we propose to exploit multiple guidance de- rived from the 3D morphable face model (3DMM).

Original: Firstly, a precomputed optical flow is utilized to guide the estimation of motion fields.
Revised: Firstly, a precomputed optical flow is utilized to guide the estimation of motion fields.

Original: Secondly, a precomputed occlusion map is utilized to guide the perception of occluded areas.
Revised: Secondly, a precomputed occlusion map is utilized to guide the perception of occluded areas.

Original: Finally, a rendered image is utilized to guide the restoration of missing contents.
Revised: Finally, a rendered image is utilized to guide the restoration of missing contents.

Original: We present a new reenactment framework to inte- grate the above guidance and generate high-quality results.
Revised: We present a new reenactment framework to inte- grate the above guidance and generate high-quality results.

Original: Extensive experiments show the superior performance of our framework compared with several state-of-the-art methods.
Revised: Extensive experiments show the superior performance of our framework compared with several state-of-the-art methods.

Original: Ablation studies demonstrate the effectiveness of exploiting multiple guidance from the 3DMM.
Revised: Ablation studies demonstrate the effectiveness of exploiting multiple guidance from the 3DMM.

Original: Introduction
Revised: Introduction

Original: Given a source face and a driving face face reenactment refers to the task of generating a reenacted face with expression and pose from the driving face and identity from the source face.
Revised: Given a source face and a driving face face reenactment refers to the task of generating a reenacted face with expression and pose from the driving face and identity from the source face.

Original: This task has a wide range of practical applications such as film production, next-generation communication, and role-playing video games
Revised: This task has a wide range of practical applications such as film production, next-generation communication, and role-playing video games

Original: Traditional solutions (Thies et al. 2015, 2016) to face reenactment mainly come from the graphics community.
Revised: Traditional solutions (Thies et al. 2015, 2016) to face reenactment mainly come from the graphics community.

Original: With advances in generative adversarial networks (GANs) (Goodfellow et al. 2014), learning-based methods have demonstrated impressive results.
Revised: With advances in generative adversarial networks (GANs) (Goodfellow et al. 2014), learning-based methods have demonstrated impressive results.

Original: The commonly used strategy is to decouple appearance and motion information from input images and obtain motion descriptors such as facial landmarks.
Revised: The commonly used strategy is to decouple appearance and motion information from input images and obtain motion descriptors such as facial landmarks.

Original: Based on this strategy, some methods (Wu et al.
Revised: Based on this strategy, some methods (Wu et al.

Original: ; Zakharov et al.
Revised: ; Zakharov et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Ha et al. 2020)
Revised: Ha et al. 2020)

Original: generate reenacted results by translating motion descriptors into images.
Revised: generate reenacted results by translating motion descriptors into images.

Original: Another kind of method (Wiles,
Revised: Another kind of method (Wiles,

Original: Koepke, and Zisserman 2018; Siarohin et al.
Revised: Koepke, and Zisserman 2018; Siarohin et al.

Original: 2019a,b; Zhao and Zhang 2022) predicts dense motion fields to deform source images in the pixel or feature domain.
Revised: 2019a,b; Zhao and Zhang 2022) predicts dense motion fields to deform source images in the pixel or feature domain.

Original: Nevertheless,
Revised: Nevertheless,

Original: All
Revised: All

Original: [Figure]
Revised: [Figure]

Original: it remains challenging for these methods to generate realistic results when there exists a significant pose discrepancy between the source face and the driving face.
Revised: it remains challenging for these methods to generate realistic results when there exists a significant pose discrepancy between the source face and the driving face.

Original: The large pose discrepancy not only makes it difficult to capture longdistance motion patterns, but the caused occlusions place high demands on restoring missing contents.
Revised: The large pose discrepancy not only makes it difficult to capture longdistance motion patterns, but the caused occlusions place high demands on restoring missing contents.

Original: To deal with pose discrepancy, 3D information is critically important since the head motions happen in the 3D physical world.
Revised: To deal with pose discrepancy, 3D information is critically important since the head motions happen in the 3D physical world.

Original: Compared with 2D photographic depictions, 3D representations contain richer spatial information that is beneficial for modeling large motions.
Revised: Compared with 2D photographic depictions, 3D representations contain richer spatial information that is beneficial for modeling large motions.

Original: Recently, several methods (Yao et al. 2020; Doukas, Zafeiriou, and Sharmanska 2021; Ren et al. 2021;
Revised: Recently, several methods (Yao et al. 2020; Doukas, Zafeiriou, and Sharmanska 2021; Ren et al. 2021;

Original: Hong et al. 2022) have been proposed to utilize 3D information for more accurate motion estimation.
Revised: Hong et al. 2022) have been proposed to utilize 3D information for more accurate motion estimation.

Original: However, proper handling of occlusions with 3D information is also important while rarely considered.
Revised: However, proper handling of occlusions with 3D information is also important while rarely considered.

Original: In this paper, we propose to exploit multiple guidance derived from the 3D morphable face model (3DMM) (Blanz and Vetter 1999) for both accurate motion estimation and proper handling of occlusions.
Revised: In this paper, we propose to exploit multiple guidance derived from the 3D morphable face model (3DMM) (Blanz and Vetter 1999) for both accurate motion estimation and proper handling of occlusions.

Original: Based on the 3D face representation, we first compute an optical flow for the inner face, which serves as a good starting point to guide the motion estimation for the entire image.
Revised: Based on the 3D face representation, we first compute an optical flow for the inner face, which serves as a good starting point to guide the motion estimation for the entire image.

Original: To handle occlusions, we then compute an occlusion map for the inner face to indicate the occluded areas and guide the perception of occlu- sions.
Revised: To handle occlusions, we then compute an occlusion map for the inner face to indicate the occluded areas and guide the perception of occlu- sions.

Original: Finally, to enhance the restoration of missing contents caused by occlusions, we render the target 3D face into a 2D image to guide the generation of faithful facial structure.
Revised: Finally, to enhance the restoration of missing contents caused by occlusions, we render the target 3D face into a 2D image to guide the generation of faithful facial structure.

Original: A new reenactment framework is presented to utilize the above guidance to generate high-quality reenacted results.
Revised: A new reenactment framework is presented to utilize the above guidance to generate high-quality reenacted results.

Original: We evaluate our framework qualitatively and quantitatively on two public datasets: VoxCeleb (Nagrani, Chung, and Zisserman 2017) and CelebV (Wu et al. 2018).
Revised: We evaluate our framework qualitatively and quantitatively on two public datasets: VoxCeleb (Nagrani, Chung, and Zisserman 2017) and CelebV (Wu et al. 2018).

Original: Experimental results show the proposed multiple guidance can facilitate the generation of accurate facial motions and photo-realistic results, even under a significant pose discrepancy.
Revised: Experimental results show the proposed multiple guidance can facilitate the generation of accurate facial motions and photo-realistic results, even under a significant pose discrepancy.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • We propose to utilize multiple guidance provided by the 3DMM for face reenactment, including computed optical flow, computed occlusion map, and rendered mesh.
Revised: • We propose to utilize multiple guidance provided by the 3DMM for face reenactment, including computed optical flow, computed occlusion map, and rendered mesh.

Original: • We present a new end-to-end reenactment framework, which effectively incorporates the multiple guidance to estimate accurate motion fields, properly handle occlusions, and generate photo-realistic results.
Revised: • We present a new end-to-end reenactment framework, which effectively incorporates the multiple guidance to estimate accurate motion fields, properly handle occlusions, and generate photo-realistic results.

Original: • Experiments demonstrate that our framework can alleviate the problems caused by a significant pose discrepancy and achieve superior performance compared with several state-of-the-art methods.
Revised: • Experiments demonstrate that our framework can alleviate the problems caused by a significant pose discrepancy and achieve superior performance compared with several state-of-the-art methods.

Original: Related Work 3D Morphable Face Model
Revised: Related Work 3D Morphable Face Model

Original: 3D Morphable Face Model (3DMM) (Blanz and Vetter 1999) is a parametric face model that represents 3D facial shapes and textures with a set of parameters.
Revised: 3D Morphable Face Model (3DMM) (Blanz and Vetter 1999) is a parametric face model that represents 3D facial shapes and textures with a set of parameters.

Original: To achieve better facial modeling, several variants (Paysan et al. 2009;
Revised: To achieve better facial modeling, several variants (Paysan et al. 2009;

Original: Cao et al. 2013) have been proposed to represent faces with fine-grained semantic parameters of identity, expression, pose, etc.
Revised: Cao et al. 2013) have been proposed to represent faces with fine-grained semantic parameters of identity, expression, pose, etc.

Original: These models produce 3D face representations from multiple face scans using principal component analysis (PCA).
Revised: These models produce 3D face representations from multiple face scans using principal component analysis (PCA).

Original: We use Basel Face Model (BFM) (Paysan et al. 2009) in our framework.
Revised: We use Basel Face Model (BFM) (Paysan et al. 2009) in our framework.

Original: Face Reenactment
Revised: Face Reenactment

Original: 2D-based methods 2D-based methods directly decouple appearance and motion information from input 2D images, which can be roughly classified into translation-based methods and warping-based methods.
Revised: 2D-based methods 2D-based methods directly decouple appearance and motion information from input 2D images, which can be roughly classified into translation-based methods and warping-based methods.

Original: Translation-based methods model the task of face reenactment as an image translation problem (Isola et al. 2017;
Revised: Translation-based methods model the task of face reenactment as an image translation problem (Isola et al. 2017;

Original: Zhu et al. 2017).
Revised: Zhu et al. 2017).

Original: For instance, ReenactGAN (Wu et al.
Revised: For instance, ReenactGAN (Wu et al.

Original: 2018) translates the transformed facial landmarks of a specific source face into reenacted images with an encoderdecoder model.
Revised: 2018) translates the transformed facial landmarks of a specific source face into reenacted images with an encoderdecoder model.

Original: To generalize to unseen persons, Zakharov et al.
Revised: To generalize to unseen persons, Zakharov et al.

Original: (2019) propose to synthesize a neural talking head with several images of that person.
Revised: (2019) propose to synthesize a neural talking head with several images of that person.

Original: Landmarks and source appearance are fused in the generator to produce reenacted faces.
Revised: Landmarks and source appearance are fused in the generator to produce reenacted faces.

Original: Nevertheless, fine-tuning is still required for unseen identities due to the use of meta-learning mechanism.
Revised: Nevertheless, fine-tuning is still required for unseen identities due to the use of meta-learning mechanism.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) introduce a one-shot framework trained in an unsupervised manner, which only needs a single source image.
Revised: (2019) introduce a one-shot framework trained in an unsupervised manner, which only needs a single source image.

Original: Reenacted images are generated by fusing the source appearance with the target face parsing maps.
Revised: Reenacted images are generated by fusing the source appearance with the target face parsing maps.

Original: However, the results are distorted when reenacting a different identity.
Revised: However, the results are distorted when reenacting a different identity.

Original: Burkov et al.
Revised: Burkov et al.

Original: (2020) utilize pose augmentation to boost the performance of cross-identity reenactment.
Revised: (2020) utilize pose augmentation to boost the performance of cross-identity reenactment.

Original: Bi-layer (Zakharov et al. 2020) generates two components from perspectives of low frequency and high frequency separately and fuses them to obtain final results.
Revised: Bi-layer (Zakharov et al. 2020) generates two components from perspectives of low frequency and high frequency separately and fuses them to obtain final results.

Original: Since the used motion descriptors such as landmarks and parsing maps are subject-specific, the identity preservation problem arises when the driving face holds a clearly different facial shape from the source face.
Revised: Since the used motion descriptors such as landmarks and parsing maps are subject-specific, the identity preservation problem arises when the driving face holds a clearly different facial shape from the source face.

Original: Ha et al.
Revised: Ha et al.

Original: (2020) propose a landmark transformer to alleviate this problem, while the fidelity of reenacted images is still not high enough.
Revised: (2020) propose a landmark transformer to alleviate this problem, while the fidelity of reenacted images is still not high enough.

Original: Warping-based methods model the relative motion between the source face and the driving face with dense motion fields such as optical flow, which are used to deform source images into reenacted ones.
Revised: Warping-based methods model the relative motion between the source face and the driving face with dense motion fields such as optical flow, which are used to deform source images into reenacted ones.

Original: X2Face (Wiles, Koepke, and Zisserman 2018) predicts optical flow from driving images, pose vectors or audio.
Revised: X2Face (Wiles, Koepke, and Zisserman 2018) predicts optical flow from driving images, pose vectors or audio.

Original: The predictions are then used to warp source face images.
Revised: The predictions are then used to warp source face images.

Original: However, it lacks the ability to generate contents that do not exist in the source images.
Revised: However, it lacks the ability to generate contents that do not exist in the source images.

Original: Monkey-Net (Siarohin et al. 2019a) applies the estimated optical flow to deform the source face in the feature domain and achieves better results.
Revised: Monkey-Net (Siarohin et al. 2019a) applies the estimated optical flow to deform the source face in the feature domain and achieves better results.

Original: FOMM (Siarohin et al. 2019b) models local motions around keypoints with affine transformations and obtains more accurate motion fields.
Revised: FOMM (Siarohin et al. 2019b) models local motions around keypoints with affine transformations and obtains more accurate motion fields.

Original: To boost the performance of motion estimation, One-shot Talking
Revised: To boost the performance of motion estimation, One-shot Talking

Original: Head (Wang, Mallya, and Liu 2021) extends the dimensionality of keypoints and predicts several flow fields.
Revised: Head (Wang, Mallya, and Liu 2021) extends the dimensionality of keypoints and predicts several flow fields.

Original: MRAA (Siarohin et al. 2021) introduces a region-based motion field modeling approach.
Revised: MRAA (Siarohin et al. 2021) introduces a region-based motion field modeling approach.

Original: Zhao and Zhang (2022) propose to use more flexible TPS transformation to replace affine transformation.
Revised: Zhao and Zhang (2022) propose to use more flexible TPS transformation to replace affine transformation.

Original: DaGAN (Hong et al. 2022) predicts a face depth map to guide motion estimation and image generation.
Revised: DaGAN (Hong et al. 2022) predicts a face depth map to guide motion estimation and image generation.

Original: Generally, warping-based methods achieve higher fidelity than translation-based methods since the high-frequency information can be better preserved with spatial deformation.
Revised: Generally, warping-based methods achieve higher fidelity than translation-based methods since the high-frequency information can be better preserved with spatial deformation.

Original: Nevertheless, the motion fields are commonly learned from scratch in a self-supervised manner due to the lack of labels.
Revised: Nevertheless, the motion fields are commonly learned from scratch in a self-supervised manner due to the lack of labels.

Original: The performance will drop significantly when large motions are observed.
Revised: The performance will drop significantly when large motions are observed.

Original: Our method is also warping-based, while the motion field is first computed and then completed instead of learned from scratch.
Revised: Our method is also warping-based, while the motion field is first computed and then completed instead of learned from scratch.

Original: 3D-based methods 3D-based methods are built upon the prior knowledge of 3D face models.
Revised: 3D-based methods 3D-based methods are built upon the prior knowledge of 3D face models.

Original: Kim et al.
Revised: Kim et al.

Original: (2018) render the reenacted meshes of a 3D face model and translate them into photo-realistic images.
Revised: (2018) render the reenacted meshes of a 3D face model and translate them into photo-realistic images.

Original: This method requires training on a large number of images of a specific person and therefore has to be retrained for new-coming identities.
Revised: This method requires training on a large number of images of a specific person and therefore has to be retrained for new-coming identities.

Original: Yao et al.
Revised: Yao et al.

Original: (2020) employ a graph neural network to learn optical flow from meshes of source and driving faces, while the areas that cannot be modeled by 3D models are less considered.
Revised: (2020) employ a graph neural network to learn optical flow from meshes of source and driving faces, while the areas that cannot be modeled by 3D models are less considered.

Original: HeadGAN (Doukas, Zafeiriou, and Sharmanska 2021) extracts PNCC representations from reconstructed face meshes to perform conditional synthesis and takes audio features as complementary information to boost performance.
Revised: HeadGAN (Doukas, Zafeiriou, and Sharmanska 2021) extracts PNCC representations from reconstructed face meshes to perform conditional synthesis and takes audio features as complementary information to boost performance.

Original: SAFA (Wang, Zhang, and Li 2021) combines FOMM with 3D models to estimate flexible expressional motions, while the subject-specific facial keypoints are still involved and lead
Revised: SAFA (Wang, Zhang, and Li 2021) combines FOMM with 3D models to estimate flexible expressional motions, while the subject-specific facial keypoints are still involved and lead

Original: [Figure]
Revised: [Figure]

Original: 2020b) uses 3DMM parameters modulate the latent of pre-trained StyleGAN (Karras, Laine, and Aila 2019) to control the expression and pose of generated face images
Revised: 2020b) uses 3DMM parameters modulate the latent of pre-trained StyleGAN (Karras, Laine, and Aila 2019) to control the expression and pose of generated face images

Original: Based on this, Tewari et al.
Revised: Based on this, Tewari et al.

Original: (2020a) propose an optimizationbased method to compute embeddings of existing images to perform reenactment on real-world faces.
Revised: (2020a) propose an optimizationbased method to compute embeddings of existing images to perform reenactment on real-world faces.

Original: More recent work
Revised: More recent work

Original: PIRenderer (Ren et al. 2021) uses 3DMM parameters to predict an optical flow and control the generation process.
Revised: PIRenderer (Ren et al. 2021) uses 3DMM parameters to predict an optical flow and control the generation process.

Original: However, the results become unrealistic when a significant identity or pose discrepancy occurs.
Revised: However, the results become unrealistic when a significant identity or pose discrepancy occurs.

Original: Different from these methods, we extract multiple guidance from the 3DMM and apply them for both accurate motion estimation and proper handling of occlusions.
Revised: Different from these methods, we extract multiple guidance from the 3DMM and apply them for both accurate motion estimation and proper handling of occlusions.

Original: Methodology
Revised: Methodology

Original: Given a source image I s and a driving image I d , we aim to generate an image I g with the expression and pose of I d and other attributes of I s such as identity, illumination, and background.
Revised: Given a source image I s and a driving image I d , we aim to generate an image I g with the expression and pose of I d and other attributes of I s such as identity, illumination, and background.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 2, we first perform 3D reconstruction on ( I s , I d ) and extract multiple guidance including computed optical flow F c , computed occlusion map O c , and rendered image I r .
Revised: 2, we first perform 3D reconstruction on ( I s , I d ) and extract multiple guidance including computed optical flow F c , computed occlusion map O c , and rendered image I r .

Original: Subsequently, we employ a completion module to estimate motions for the entire image based on the computed results ( F c , O c ) .
Revised: Subsequently, we employ a completion module to estimate motions for the entire image based on the computed results ( F c , O c ) .

Original: Finally, we apply the complete optical flow F and occlusion map O to transform I s into I g in our reenactment module, which is also guided by the rendered image I r .
Revised: Finally, we apply the complete optical flow F and occlusion map O to transform I s into I g in our reenactment module, which is also guided by the rendered image I r .

Original: Extraction of Multiple Guidance
Revised: Extraction of Multiple Guidance

Original: In this part, we perform 3D reconstruction on input images and obtain multiple guidance from reconstructed 3D faces, as shown in Fig.
Revised: In this part, we perform 3D reconstruction on input images and obtain multiple guidance from reconstructed 3D faces, as shown in Fig.

Original: A pre-trained face reconstruction model provided by (Deng et al. 2019) is used to extract 3DMM parameters from face images.
Revised: A pre-trained face reconstruction model provided by (Deng et al. 2019) is used to extract 3DMM parameters from face images.

Original: Given a face image, the reconstruction model regresses a vector v = ( α , β , δ , γ , p )
Revised: Given a face image, the reconstruction model regresses a vector v = ( α , β , δ , γ , p )

Original: ∈ R, where α ∈ R 80 , β ∈ R 64 , and δ ∈ R 80 represent the identity, expression, and texture parameters, respectively.
Revised: ∈ R, where α ∈ R 80 , β ∈ R 64 , and δ ∈ R 80 represent the identity, expression, and texture parameters, respectively.

Original: γ ∈ R 27 and p ∈ R 6 refer to the illumination and face pose, respectively.
Revised: γ ∈ R 27 and p ∈ R 6 refer to the illumination and face pose, respectively.

Original: The mesh topology we adopt contains v = 35709 vertices and f = 70789 triangles.
Revised: The mesh topology we adopt contains v = 35709 vertices and f = 70789 triangles.

Original: The 3D coordinates V ∈ R v × 3 of mesh vertices can be computed with the regressed parameters.
Revised: The 3D coordinates V ∈ R v × 3 of mesh vertices can be computed with the regressed parameters.

Original: [Equation]
Revised: [Equation]

Original: where V mean and V i
Revised: where V mean and V i

Original: d are the PCA bases of BFM (Paysan et al. 2009), and V exp is built from (Cao et al. 2013).
Revised: d are the PCA bases of BFM (Paysan et al. 2009), and V exp is built from (Cao et al. 2013).

Original: R ∈ SO(3) and t ∈ R 3 are the rotation and translation derived from p , respectively.
Revised: R ∈ SO(3) and t ∈ R 3 are the rotation and translation derived from p , respectively.

Original: With attributes A ∈ R v × d assigned to the vertices, a 2D image can be rendered through rasterization.
Revised: With attributes A ∈ R v × d assigned to the vertices, a 2D image can be rendered through rasterization.

Original: [Equation]
Revised: [Equation]

Original: where I ∈ R h × w ×
Revised: where I ∈ R h × w ×

Original: d is the rendered image.
Revised: d is the rendered image.

Original: h and w denote the spatial size while d denotes the attribute dimension.
Revised: h and w denote the spatial size while d denotes the attribute dimension.

Original: R is the rendering function that rasterizes 3D vertex attributes into the 2D image plane.
Revised: R is the rendering function that rasterizes 3D vertex attributes into the 2D image plane.

Original: C is the perspective camera model.
Revised: C is the perspective camera model.

Original: We use PyTorch3D (Ravi et al. 2020) to implement the rendering process.
Revised: We use PyTorch3D (Ravi et al. 2020) to implement the rendering process.

Original: Given I s
Revised: Given I s

Original: and I d , we extract their 3DMM parameters v s and v d , which are then recombined to obtain target parameters v t
Revised: and I d , we extract their 3DMM parameters v s and v d , which are then recombined to obtain target parameters v t

Original: =
Revised: =

Original: ( α s , β d , δ s , γ s , p d ) .
Revised: ( α s , β d , δ s , γ s , p d ) .

Original: The target mesh can be constructed with v t and then rendered into an image I r with computed color according to the texture and illumination of the source face.
Revised: The target mesh can be constructed with v t and then rendered into an image I r with computed color according to the texture and illumination of the source face.

Original: I r serves as the guidance of facial structure to facilitate the restoration of missing contents caused by occlusions.
Revised: I r serves as the guidance of facial structure to facilitate the restoration of missing contents caused by occlusions.

Original: The optical flow for the inner face can be computed with the constructed vertices as follows:
Revised: The optical flow for the inner face can be computed with the constructed vertices as follows:

Original: [Equation]
Revised: [Equation]

Original: where P is the function that projects 3D vertices into the 2D coordinate space.
Revised: where P is the function that projects 3D vertices into the 2D coordinate space.

Original: G is used to generate a 2D coordinate grid of shape R h × w × 2 .
Revised: G is used to generate a 2D coordinate grid of shape R h × w × 2 .

Original: We additionally compute an occlusion map for the inner face to indicate the occluded areas and guide the perception of occlusions.
Revised: We additionally compute an occlusion map for the inner face to indicate the occluded areas and guide the perception of occlusions.

Original: [Equation]
Revised: [Equation]

Original: where O c ∈ R h × w × 1 is the computed occlusion map.
Revised: where O c ∈ R h × w × 1 is the computed occlusion map.

Original: i and j denote the indices of row and column.
Revised: i and j denote the indices of row and column.

Original: P t and P s are the
Revised: P t and P s are the

Original: [Figure]
Revised: [Figure]

Original: nearest triangles at each pixel of the inner facial region in
Revised: nearest triangles at each pixel of the inner facial region in

Original: I g
Revised: I g

Original: and I s , respectively.
Revised: and I s , respectively.

Original: The indicator function 1 identifies such a rule that a target pixel is occluded if its corresponding triangle cannot be found in the set of source triangles.
Revised: The indicator function 1 identifies such a rule that a target pixel is occluded if its corresponding triangle cannot be found in the set of source triangles.

Original: Flow-and-Occlusion Completion
Revised: Flow-and-Occlusion Completion

Original: After obtaining the optical flow and occlusion map for the inner face, we employ a completion network F c to estimate motions for the entire image based on the computed results, as shown in Fig.
Revised: After obtaining the optical flow and occlusion map for the inner face, we employ a completion network F c to estimate motions for the entire image based on the computed results, as shown in Fig.

Original: The precomputed optical flow and occlusion map serve as good starting points to estimate final motion fields.
Revised: The precomputed optical flow and occlusion map serve as good starting points to estimate final motion fields.

Original: F c takes the source image I s and a subset of driving parameters ˜ v d = ( β d , p d ) as inputs and produces an optical flow, an occlusion map, and corresponding masks.
Revised: F c takes the source image I s and a subset of driving parameters ˜ v d = ( β d , p d ) as inputs and produces an optical flow, an occlusion map, and corresponding masks.

Original: [Equation]
Revised: [Equation]

Original: where F p and O p represent the predicted optical flow and occlusion map, respectively.
Revised: where F p and O p represent the predicted optical flow and occlusion map, respectively.

Original: M f and M o are the predicted masks for fusing the predicted and computed results.
Revised: M f and M o are the predicted masks for fusing the predicted and computed results.

Original: We adopt the adaptive instance normalization (AdaIN) (Huang and Belongie 2017) to inject ˜ v d into F c .
Revised: We adopt the adaptive instance normalization (AdaIN) (Huang and Belongie 2017) to inject ˜ v d into F c .

Original: The final optical flow and occlusion map can be obtained as follows:
Revised: The final optical flow and occlusion map can be obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is the element-wise multiplication.
Revised: where ⊙ is the element-wise multiplication.

Original: With the guidance of the computed results, the completion network can pay more attention to the motion estimation of non-facial regions, which cannot be modeled by the 3DMM.
Revised: With the guidance of the computed results, the completion network can pay more attention to the motion estimation of non-facial regions, which cannot be modeled by the 3DMM.

Original: The final results take advantage of the computed and predicted results to make a consistent and accurate estimation for both the facial and non-facial regions.
Revised: The final results take advantage of the computed and predicted results to make a consistent and accurate estimation for both the facial and non-facial regions.

Original: [Figure]
Revised: [Figure]

Original: module.
Revised: module.

Original: Reenacting
Revised: Reenacting

Original: In this part, we employ a reenactment network F r to transform the source image I s into the reenacted image I g , as shown in Fig.
Revised: In this part, we employ a reenactment network F r to transform the source image I s into the reenacted image I g , as shown in Fig.

Original: F r is designed with an encoder-decoder architecture.
Revised: F r is designed with an encoder-decoder architecture.

Original: To preserve source textures, we fuse features at multiple scales via skip connections.
Revised: To preserve source textures, we fuse features at multiple scales via skip connections.

Original: The encoder features are first warped and then masked before being ported to the corresponding decoder layers.
Revised: The encoder features are first warped and then masked before being ported to the corresponding decoder layers.

Original: The optical flow F is used to warp the source features to the desired regions, while the occlusion map O is used to mask out the occluded areas that should be inpainted.
Revised: The optical flow F is used to warp the source features to the desired regions, while the occlusion map O is used to mask out the occluded areas that should be inpainted.

Original: To guide the restoration of missing contents caused by occlusions and produce a faithful facial structure, we inject the rendered image I r into the decoder part of F r with spatially-adaptive normalization (SPADE)
Revised: To guide the restoration of missing contents caused by occlusions and produce a faithful facial structure, we inject the rendered image I r into the decoder part of F r with spatially-adaptive normalization (SPADE)

Original: (Park et al. 2019).
Revised: (Park et al. 2019).

Original: Training
Revised: Training

Original: We train our framework end-to-end in a self-supervised manner.
Revised: We train our framework end-to-end in a self-supervised manner.

Original: In the training stage, a pair of images ( I s , I d ) are randomly chosen from each video to perform selfreenactment.
Revised: In the training stage, a pair of images ( I s , I d ) are randomly chosen from each video to perform selfreenactment.

Original: While the identities of I s and I d can be different in the inference stage.
Revised: While the identities of I s and I d can be different in the inference stage.

Original: The framework is trained with a reconstruction loss L c and a style loss L s , which are based on the perceptual loss of (Johnson, Alahi, and Fei-Fei 2016).
Revised: The framework is trained with a reconstruction loss L c and a style loss L s , which are based on the perceptual loss of (Johnson, Alahi, and Fei-Fei 2016).

Original: Similar to (Siarohin et al. 2019b), we downsample I g
Revised: Similar to (Siarohin et al. 2019b), we downsample I g

Original: and I d several times and calculate L c at multiple scales.
Revised: and I d several times and calculate L c at multiple scales.

Original: L c and L s calculate ℓ 1 distance and statistic error between the activation maps of generated image I g and ground-truth I d .
Revised: L c and L s calculate ℓ 1 distance and statistic error between the activation maps of generated image I g and ground-truth I d .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: where ϕ i is the activation map of the i -th layer in the pretrained VGG-19 network, and j denotes downsampling j times.
Revised: where ϕ i is the activation map of the i -th layer in the pretrained VGG-19 network, and j denotes downsampling j times.

Original: G ϕi is the Gram matrix constructed from ϕ i .
Revised: G ϕi is the Gram matrix constructed from ϕ i .

Original: The final loss used for training our framework is a weighted summation of the above losses.
Revised: The final loss used for training our framework is a weighted summation of the above losses.

Original: [Equation]
Revised: [Equation]

Original: where we set λ s = 250 in the experiments.
Revised: where we set λ s = 250 in the experiments.

Original: Experiments Experimental Setup
Revised: Experiments Experimental Setup

Original: Datasets We conduct experiments on two public datasets:
Revised: Datasets We conduct experiments on two public datasets:

Original: VoxCeleb (Nagrani, Chung, and Zisserman 2017) and
Revised: VoxCeleb (Nagrani, Chung, and Zisserman 2017) and

Original: CelebV (Wu et al. 2018).
Revised: CelebV (Wu et al. 2018).

Original: The VoxCeleb dataset contains 22496 talking-head videos extracted from YouTube.
Revised: The VoxCeleb dataset contains 22496 talking-head videos extracted from YouTube.

Original: Following the same pre-processing method described in (Siarohin et al. 2019b), we crop valid faces from the original videos.
Revised: Following the same pre-processing method described in (Siarohin et al. 2019b), we crop valid faces from the original videos.

Original: A total of 17913 and 514 videos with lengths varying from 64 to 1024 frames are obtained for the train and test splits, respectively.
Revised: A total of 17913 and 514 videos with lengths varying from 64 to 1024 frames are obtained for the train and test splits, respectively.

Original: We use a similar test set sampling strategy of (Ha et al.
Revised: We use a similar test set sampling strategy of (Ha et al.

Original: and collect 10280 image pairs for both selfreenactment and cross-reenactment.
Revised: and collect 10280 image pairs for both selfreenactment and cross-reenactment.

Original: For self-reenactment, 20 image pairs are randomly sampled from each video of the test split, while the two images in each pair are from different identities under the cross-reenactment setup.
Revised: For self-reenactment, 20 image pairs are randomly sampled from each video of the test split, while the two images in each pair are from different identities under the cross-reenactment setup.

Original: To further evaluate the performance of reenacting unseen identities, similar to the in-the-wild scenario, we construct another test set using the CelebV dataset that includes videos of five different celebrities.
Revised: To further evaluate the performance of reenacting unseen identities, similar to the in-the-wild scenario, we construct another test set using the CelebV dataset that includes videos of five different celebrities.

Original: We randomly sample 1000 image pairs for each identity and obtain 5000 pairs in total.
Revised: We randomly sample 1000 image pairs for each identity and obtain 5000 pairs in total.

Original: Metrics We use multiple metrics to evaluate the quality of results.
Revised: Metrics We use multiple metrics to evaluate the quality of results.

Original: To measure the realism of generated images,
Revised: To measure the realism of generated images,

Original: Fr´echet Inception Distance ( FID ) (Heusel et al. 2017) is utilized to estimate the difference between the distributions of the generated and real images.
Revised: Fr´echet Inception Distance ( FID ) (Heusel et al. 2017) is utilized to estimate the difference between the distributions of the generated and real images.

Original: Besides, the quality of identity preservation is evaluated with the cosine similarity ( CSIM ) of embedding vectors from pretrained face recognition model.
Revised: Besides, the quality of identity preservation is evaluated with the cosine similarity ( CSIM ) of embedding vectors from pretrained face recognition model.

Original: For the motion accuracy, following the previous work of Ren et al.
Revised: For the motion accuracy, following the previous work of Ren et al.

Original: (2021), we use the Average Expression
Revised: (2021), we use the Average Expression

Original: Distance ( AED ) and Average Pose Distance ( APD ) to evaluate the quality of expression and pose transfer, respectively.
Revised: Distance ( AED ) and Average Pose Distance ( APD ) to evaluate the quality of expression and pose transfer, respectively.

Original: Training Details
Revised: Training Details

Original: We train our framework on the VoxCeleb dataset forepochs using four TITAN V GPUs.
Revised: We train our framework on the VoxCeleb dataset forepochs using four TITAN V GPUs.

Original: The ADAM optimizer
Revised: The ADAM optimizer

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and Ba 2014) is adopted with an initial learning rate of 1 × 10 − 4 .
Revised: and Ba 2014) is adopted with an initial learning rate of 1 × 10 − 4 .

Original: The learning rate is decreased to 1 × 10 −after 60 epochs.
Revised: The learning rate is decreased to 1 × 10 −after 60 epochs.

Original: The batch size is set to 8 in our experiments.
Revised: The batch size is set to 8 in our experiments.

Original: Comparisons
Revised: Comparisons

Original: We evaluate the performance of our framework under two setups: self-reenactment and cross-reenactment.
Revised: We evaluate the performance of our framework under two setups: self-reenactment and cross-reenactment.

Original: The evaluation results are compared with the following methods:
Revised: The evaluation results are compared with the following methods:

Original: X2Face (Wiles, Koepke, and Zisserman 2018), FOMM (Siarohin et al. 2019b), PIRenderer (Ren et al. 2021) and
Revised: X2Face (Wiles, Koepke, and Zisserman 2018), FOMM (Siarohin et al. 2019b), PIRenderer (Ren et al. 2021) and

Original: DaGAN (Hong et al. 2022).
Revised: DaGAN (Hong et al. 2022).

Original: The absolute motions are used for all methods.
Revised: The absolute motions are used for all methods.

Original: Self-reenactment We first compare the generated results under the self-reenactment setup, where the source image and the driving image are from the same person.
Revised: Self-reenactment We first compare the generated results under the self-reenactment setup, where the source image and the driving image are from the same person.

Original: The quantitative results are reported in Table 1.
Revised: The quantitative results are reported in Table 1.

Original: Our framework achieves the best results on AED and FID and competitive results on CSIM and APD.
Revised: Our framework achieves the best results on AED and FID and competitive results on CSIM and APD.

Original: The improvement in AED and
Revised: The improvement in AED and

Original: FID shows that our framework can capture more accurate facial motions and generate more realistic results.
Revised: FID shows that our framework can capture more accurate facial motions and generate more realistic results.

Original: Cross-reenactment We also perform comparisons under the cross-reenactment setup, where the source and driving faces are from different identities.
Revised: Cross-reenactment We also perform comparisons under the cross-reenactment setup, where the source and driving faces are from different identities.

Original: Compared with selfreenactment, cross-reenactment is a more meaningful setup, which is required for the vast majority of practical applications.
Revised: Compared with selfreenactment, cross-reenactment is a more meaningful setup, which is required for the vast majority of practical applications.

Original: The quantitative results are provided in Tableand Table 2.
Revised: The quantitative results are provided in Tableand Table 2.

Original: Our framework achieves the best results on
Revised: Our framework achieves the best results on

Original: CSIM, which demonstrates that the identity information of the source face is well-preserved.
Revised: CSIM, which demonstrates that the identity information of the source face is well-preserved.

Original: The lower AED and APD indicate that our framework is able to capture facial motions more accurately with multiple guidance from the 3DMM.
Revised: The lower AED and APD indicate that our framework is able to capture facial motions more accurately with multiple guidance from the 3DMM.

Original: In addition, our framework produces more realistic results,
Revised: In addition, our framework produces more realistic results,

Original: [Figure]
Revised: [Figure]

Original: reaching a better FID score.
Revised: reaching a better FID score.

Original: To investigate the performance under different pose discrepancies between the source and driving faces, we analyze the quantitative results of crossreenactment on the VoxCeleb dataset, as shown in Fig.
Revised: To investigate the performance under different pose discrepancies between the source and driving faces, we analyze the quantitative results of crossreenactment on the VoxCeleb dataset, as shown in Fig.

Original: It can be seen that our framework can preserve source identities better and generate more realistic results under both small and large pose discrepancies.
Revised: It can be seen that our framework can preserve source identities better and generate more realistic results under both small and large pose discrepancies.

Original: The qualitative results are shown in Fig.
Revised: The qualitative results are shown in Fig.

Original: It can be seen that X2Face produces unrealistic distorted results.
Revised: It can be seen that X2Face produces unrealistic distorted results.

Original: This is because it uses the predicted optical flow to deform the source image directly in the pixel domain and lacks the ability to generate reliable contents for the occluded areas.
Revised: This is because it uses the predicted optical flow to deform the source image directly in the pixel domain and lacks the ability to generate reliable contents for the occluded areas.

Original: It is difficult to estimate accurate optical flow directly from the input images, especially under a large motion.
Revised: It is difficult to estimate accurate optical flow directly from the input images, especially under a large motion.

Original: By deforming the source image with the estimated optical flow in the feature domain, FOMM can produce reasonable results.
Revised: By deforming the source image with the estimated optical flow in the feature domain, FOMM can produce reasonable results.

Original: However, since it uses subject-specific sparse keypoints to estimate facial motions, the facial structure is distorted when the source face and the driving face hold significantly different facial shapes.
Revised: However, since it uses subject-specific sparse keypoints to estimate facial motions, the facial structure is distorted when the source face and the driving face hold significantly different facial shapes.

Original: The facial expressions are not well-reenacted either.
Revised: The facial expressions are not well-reenacted either.

Original: When the identities of the source face and the driving face are quite different (e.g. different genders), PIRenderer fails to faithfully preserve the source identity.
Revised: When the identities of the source face and the driving face are quite different (e.g. different genders), PIRenderer fails to faithfully preserve the source identity.

Original: There appear some leakages of identity-specific features (e.g. mustache in the second row of Fig. 5) from the driving face to the reenacted face.
Revised: There appear some leakages of identity-specific features (e.g. mustache in the second row of Fig. 5) from the driving face to the reenacted face.

Original: One possible explanation is that PIRenderer maps the driving parameters into a latent vector and injects it into the generation process.
Revised: One possible explanation is that PIRenderer maps the driving parameters into a latent vector and injects it into the generation process.

Original: The model may overfit some specific identities during the training process, leading to the leakage of identity information.
Revised: The model may overfit some specific identities during the training process, leading to the leakage of identity information.

Original: DaGAN generates more accurate facial expressions than FOMM by incorporating the learned depth map.
Revised: DaGAN generates more accurate facial expressions than FOMM by incorporating the learned depth map.

Original: However, the facial shapes are still inaccurate due to the use of subject-specific sparse keypoints.
Revised: However, the facial shapes are still inaccurate due to the use of subject-specific sparse keypoints.

Original: With the help of multiple guidance derived from the 3DMM, our method produces more realistic results with
Revised: With the help of multiple guidance derived from the 3DMM, our method produces more realistic results with

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: FG
Revised: FG

Original: [Equation]
Revised: [Equation]

Original: OG
Revised: OG

Original: [Figure]
Revised: [Figure]

Original: SG
Revised: SG

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: more accurate facial motions compared with the other methods, even under a large identity or pose discrepancy between the source face and the driving face.
Revised: more accurate facial motions compared with the other methods, even under a large identity or pose discrepancy between the source face and the driving face.

Original: Notably, compared with
Revised: Notably, compared with

Original: PIRenderer and DaGAN, which also incorporate 3D information, our framework exploits 3D priors in a more comprehensive way.
Revised: PIRenderer and DaGAN, which also incorporate 3D information, our framework exploits 3D priors in a more comprehensive way.

Original: We extract multiple guidance from the 3D face meshes for both accurate motion estimation and proper handling of occlusions.
Revised: We extract multiple guidance from the 3D face meshes for both accurate motion estimation and proper handling of occlusions.

Original: By utilizing the rich priors contained in the 3DMM, our framework generates more realistic results and preserves source identities better.
Revised: By utilizing the rich priors contained in the 3DMM, our framework generates more realistic results and preserves source identities better.

Original: Ablation Study
Revised: Ablation Study

Original: We perform ablation experiments on the CelebV dataset to investigate the effectiveness of the proposed multiple guidance.
Revised: We perform ablation experiments on the CelebV dataset to investigate the effectiveness of the proposed multiple guidance.

Original: The optical flow and occlusion map used in the base model are directly predicted by the completion network, without guidance of the computed results.
Revised: The optical flow and occlusion map used in the base model are directly predicted by the completion network, without guidance of the computed results.

Original: Besides, no rendered image is injected into the reenactment network.
Revised: Besides, no rendered image is injected into the reenactment network.

Original: We add the proposed multiple guidance to the base model in turn and report the quantitative results in Table 3.
Revised: We add the proposed multiple guidance to the base model in turn and report the quantitative results in Table 3.

Original: The flow guidance (FG), occlusion guidance (OG), and synthesis guidance (SG) represent the computed optical flow, computed occlusion map, and rendered image, respectively.
Revised: The flow guidance (FG), occlusion guidance (OG), and synthesis guidance (SG) represent the computed optical flow, computed occlusion map, and rendered image, respectively.

Original: It can be seen that the flow guidance improves CSIM and AED, which means that it can facilitate more accurate motion estimation and benefit identity preservation.
Revised: It can be seen that the flow guidance improves CSIM and AED, which means that it can facilitate more accurate motion estimation and benefit identity preservation.

Original: Comparing the second and third rows of Table 3, the occlusion guidance brings a clear improvement to FID, which indicates that it can facilitate the generation of realistic results.
Revised: Comparing the second and third rows of Table 3, the occlusion guidance brings a clear improvement to FID, which indicates that it can facilitate the generation of realistic results.

Original: The synthesis guidance improves all four metrics by providing explicit clues for properly handling occlusions.
Revised: The synthesis guidance improves all four metrics by providing explicit clues for properly handling occlusions.

Original: P
Revised: P

Original: I R e nd e r e r
Revised: I R e nd e r e r

Original: O u r s
Revised: O u r s

Original: Source
Revised: Source

Original: Driving
Revised: Driving

Original: Expression
Revised: Expression

Original: Pose
Revised: Pose

Original: Expression+Pose
Revised: Expression+Pose

Original: P
Revised: P

Original: I R e nd e r e r
Revised: I R e nd e r e r

Original: O u r s
Revised: O u r s

Original: The qualitative results are shown in Fig.
Revised: The qualitative results are shown in Fig.

Original: It can be seen that the base model suffers from distortions of facial shapes due to predicted inaccurate motion fields.
Revised: It can be seen that the base model suffers from distortions of facial shapes due to predicted inaccurate motion fields.

Original: By incorporating the flow guidance as a good starting point for motion estimation, the distortions of facial shapes are reduced.
Revised: By incorporating the flow guidance as a good starting point for motion estimation, the distortions of facial shapes are reduced.

Original: With explicit guidance for the perception of occlusions, the visual quality improves to a certain extent.
Revised: With explicit guidance for the perception of occlusions, the visual quality improves to a certain extent.

Original: The synthesis guidance provides semantic information such as facial structure and texture, further helping generate more accurate facial motions and realistic results.
Revised: The synthesis guidance provides semantic information such as facial structure and texture, further helping generate more accurate facial motions and realistic results.

Original: Disentangled Reenactment
Revised: Disentangled Reenactment

Original: Since the 3DMM parameters are fully disentangled, it is possible to reenact one of the expression and pose alone while keeping the other one unchanged.
Revised: Since the 3DMM parameters are fully disentangled, it is possible to reenact one of the expression and pose alone while keeping the other one unchanged.

Original: The reenacted results are shown in Fig.
Revised: The reenacted results are shown in Fig.

Original: It can be seen that our framework can reenact expression or pose independently and generate more realistic images than PIRenderer (Ren et al. 2021).
Revised: It can be seen that our framework can reenact expression or pose independently and generate more realistic images than PIRenderer (Ren et al. 2021).

Original: Conclusionand Discussion
Revised: Conclusion

Original: In this paper, we propose to exploit multiple guidance derived from the 3DMM and present a new face reenactment framework.
Revised: In this paper, we propose to exploit multiple guidance derived from the 3DMM and present a new face reenactment framework.

Original: The multiple guidance not only helps with more accurate motion estimation but also facilitates proper handling of occlusions.
Revised: The multiple guidance not only helps with more accurate motion estimation but also facilitates proper handling of occlusions.

Original: Our framework can generate realistic results with accurate facial motions, even under a significant head pose discrepancy between the source face and the driving face.
Revised: Our framework can generate realistic results with accurate facial motions, even under a significant head pose discrepancy between the source face and the driving face.

Original: Experimental results show the superior performance of our framework compared with several state-of-theart methods.
Revised: Experimental results show the superior performance of our framework compared with several state-of-theart methods.

Original: Ablation studies demonstrate the effectiveness of the proposed multiple guidance.
Revised: Ablation studies demonstrate the effectiveness of the proposed multiple guidance.

Original: The proposed method enables modifying the expression and pose of existing faces and may be used to make fake content for spoofing.

Original: The generated high-quality results bring challenges for forgery detection techniques.

Original: We call on the development of robust approaches to detect such forgery.

Original: On the other hand, our method can serve as a new benchmark for research on face forgery detection.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work is supported by National Natural Science Foundation of China (No. 62172021) and Shenzhen Fundamental Research Program (GXWD2020123116580700720200806163656003).
Revised: This work is supported by National Natural Science Foundation of China (No. 62172021) and Shenzhen Fundamental Research Program (GXWD2020123116580700720200806163656003).

Original: We thank all reviewers for their valuable feedback and helpful comments.
Revised: We thank all reviewers for their valuable feedback and helpful comments.