FNeVR: Neural Volume Rendering for Face Animation
Equal contributions
Abstract
Face animation, one of the hottest topics in computer vision, has achieved a promising performance with the help of generative models.
However, it remains a critical challenge to generate identity preserving and photo-realistic images due to the sophisticated motion deformation and complex facial detail modeling.
To address these problems, we propose a Face Neural Volume Rendering (FNeVR) network to fully explore the potential of 2D motion warping and 3D volume rendering in a unified framework.
In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering.
Specifically, we first extract 3D information with a well-designed architecture, and then introduce an orthogonal adaptive ray-sampling module for efficient rendering.
We also design a lightweight pose editor, enabling FNeVR to edit the facial pose in a simple yet effective way.
Extensive experiments show that our FNeVR obtains the best overall quality and performance on widely used talking-head benchmarks.
Our code is available 1 .
Introduction
Aiming at animating a still head, face animation has far-reaching applications, such as photography, social media, movie production and virtual assistance, and has become a popular research topic in computer vision and computer graphics [14; 50].
It synthesizes a realistic talking-head video by combining the appearance and identity from one source face image with poses and motions extracted from a given driving video, possibly of a different person’s identity from the source.
With the progress of generative models, especially Generative Adversarial Networks (GANs) and Variational
Auto-Encoders (VAEs), recent methods have achieved a promising performance in face animation [65; 52; 59; 16].
However, it is still a challenging task because of the difficulty of guaranteeing the motion transform validity and generating authentic images simultaneously.
Face animation methods can be mainly divided into three categories: model-free, landmark-based, and 3D structure-based.
Model-free methods [63; 51; 52; 60; 9] realize compelling reenactment by modeling the relative motion field from the source to the driving faces.
Notably, First Order Motion Model (FOMM)
[52] significantly improves the performance of image animation by using self-supervised keypoints with 2D motion warping.
Our studies show that 2D warping is effective in generating
[Figure]
high-quality motion transfer.
However, it has a limited ability to produce sufficiently realistic images (see Fig. 1(c)).
Landmark-based methods [2; 19; 64; 57; 65] introduce 2D facial landmarks and obtain considerable progress in face animation, but they are intrinsically disadvantageous in identity preservation.
Motivated by the advanced 3D Morphable Models (3DMMs) [4; 46; 20; 5; 6; 7; 36], researchers tend to integrate the 3D geometric prior to condition synthesis and achieve impressive performance to generate realistic images [35; 16].
Nonetheless, these 3D structure-based methods often focus on introducing 3D facial information in the motion model to improve the authenticity of the results, such as Face vid2vid [59], neglecting the essential advantages of 2D warping in motion transfer.
As shown in Fig. 1(d), Face vid2vid produces a clear head profile but with poor facial details.
In this paper, we propose a Face Neural Volume Rendering (FNeVR) network for more realistic face animation, by taking the merits of 2D motion warping [52] on facial expression transformation and 3D volume rendering on high-quality image synthesis in a unified framework.
Inspired by the remarkable properties of neural rendering [39; 26] in establishing high fidelity representations, we design a novel 3D Face Volume Rendering (FVR) module for face animation, which enables our FNeVR to learn more realistic facial details efficiently and thus achieve photo-realistic results.
Specifically, the proposed FVR module first adopts two convolutional neural networks (CNNs), which we design to extract the 3D color and shape information from the 2D warped feature with the aid of the 3D reconstruction results.
We only apply the 3D reconstruction module to supervise the 3D
CNNs in the training process without any overhead for inference, which is different from existing 3D structure-based methods [35; 16; 59].
After that, we introduce an orthogonal adaptive ray-sampling module to efficiently calculate the sampled color field and the voxel probability for subsequent image rendering.
Instead of using hierarchical sampling as in [39], our orthogonal adaptive ray-sampling only needs one MLP to process the 3D information directly.
Additionally, we design a perceptual loss to guarantee the quality of rendering feature maps.
All in all, our FNeVR can not only generate more realistic images than 2D-based methods, but also obtain more accurate motion transfer than 3D-based methods (see Fig. 1(e)).
Moreover, we present a Lightweight Pose Editing (LPE) module, which is simple yet effective and only takes Euler angle as the input to efficiently realize facial pose editing.
We further design a new loss to supervise the face editing model.
The contributions of this paper include:
• A Face Neural Volume Rendering (FNeVR) network is presented to take the merits of 2D motion warping on face expression transformation and 3D volume rendering on high-quality image synthesis in a unified framework for realistic face animation.
• A Face Volume Rendering (FVR) module with orthogonal adaptive ray-sampling is proposed to capture facial details effectively and improve animation performance.
Additionally, a simple yet effective Lightweight Pose Editing (LPE) module is presented only based on the Euler angle.
• Extensive experiments are conducted to compare FNeVR with state-of-the-art methods.
The results show that our FNeVR obtains the best overall quality and performance on widely used talking-head benchmarks.
Related Work
Face Animation.
Model-free methods [63; 51; 52; 60; 9; 59] complete the deformation of face images by establishing the motion field without extracting prior facial information in advance.
Early models [63] can extract identity and pose information, but they may fail to capture the appropriate deformations in some problematic cases.
MonkeyNet [51] handles motion transfer by acquiring sparse keypoints.
Subsequently, the First Order Motion Model (FOMM)
[52] significantly improves the performance of face animation with a rigorous first-order mathematical model.
While reporting promising synthesized images, most 2D model-free methods have limited ability to model out-ofplane rotations and expressional motions.
Face vid2vid [59] further improves FOMM by introducing 3D representations to model a 3D motion field for face generation.
Nevertheless, it has a considerable computational cost and still lacks the capability of expression transformation.
Landmark-based methods [2; 19; 64; 23; 57; 65] utilize 2D facial landmarks as conditions for reenactment.
However, since 2D facial landmarks cause the injection of identity-related information from the driving face to the generated one, these methods often cannot handle identity preservation well during the generation process.
In order to address the above issues, many works [33; 32; 35; 16; 47; 56] focus more on 3D structurebased methods by means of the geometric prior of 3D faces and produce impressive results on subject-agnostic face synthesis.
HeadGAN [16] employs the rendered 3D mesh as input to synthesize deformed images, but it does not work well in facial expression transformation.
Conditioned on the parameters of the 3D Morphable Model (3DMM) [4], StyleRig [55] and GIF [21] respectively employ pre-trained StyleGAN [30] and StyleGAN2 [31] to warp face images.
PIRenderer [47] controls the face motions and predicts a flow field for deformation.
Nonetheless, these methods are intrinsically limited to modeling the specific head components such as hair and teeth.
3D Morphable Face Models.
The 3D Morphable Model (3DMM) [4] is a statistical model that provides 3D face representation for rendering face images by parameter modulation.
Recent methods [46; 20; 5; 6; 7] extend 3DMM to more precise modeling of shape, expression, or pose on the basis of the original synthesis.
Among these models, FLAME [36] additionally models the head pose, joint pose, and geometric expression information, offering more realistic and expressive 3D faces.
Benefitting from 3DMM, the face reconstruction methods [15; 18; 1; 62] achieve significant progress.
The representative method DECA [17] introduces an encoder to extract the parameters of the face for
FLAME and employs another encoder to attain detailed reconstruction.
In this paper, we exploit a pre-trained encoder of DECA to extract the parameters required by FLAME and then utilize FLAME to reconstruct a reliable 3D face representation from 2D images.
3D Volume Rendering.
3D volume rendering is used for rendering 2D projections of 3D objects and scenes.
NeRF [39] is a breakthrough in view synthesis which can produce impressive rendered results based on the input positional information.
Its core idea lies in optimizing a continuous volumetric scene function using a sparse set of input views.
Thanks to its excellent performance, many works [49; 42; 43; 44; 10; 22; 67; 26] integrate NeRF with GANs to directly control the pose of synthesized result and generate impressive multi-view images.
GRAF [49] introduces conditional GANs on the basis of NeRF essentially, which improves the quality and 3D consistency of the rendering results.
GIRAFFE [44] can complete the rendering of a real-world scene and freely add multiple objects to the same scene.
Differently, we introduce a lightweight and efficient adaptive volume rendering method, which only requires a single view to enhance the quality of the rendered 2D image.
Method
We formulate the face animation task as a talking-head generation with an editable pose.
For a given source image S and a driving video { D 1 ,D 2 , · · · ,D N } , where D i denotes the i -th frame in the video and N is the total frame number, the objective of our FNeVR is to generate a photo-realistic talking-head video with the same face identity as S and the motion and facial expressions derived from { D 1 ,D 2 , ··· ,D N } .
As shown in Fig.
2, the whole framework mainly consists of four modules to achieve this goal: (1) 2D motion estimation, (2) 3D face reconstruction, (3) FVR, and (4) LPE.
With
S and D i as input, we first utilize the 2D motion estimation module to estimate 2D facial keypoints and generate a motion field to transform the features.
We also employ the 3D reconstruction module to build 3D face representations to provide the rendering network with precise and reliable 3D shape information.
Then a novel FVR module is introduced to fully explore the capacity of the 2D motion module on the challenging expression transformation and generate images with high fidelity.
In addition, the LPE module is added to extend our model to edit the facial pose efficiently.
3.1 2D Motion Estimation
Despite the trend that most existing methods [59; 25; 16] focus on studying various estimation methods of motion fields, extensive experimental results in [52; 56] show that the strategy of warping
[Figure]
f θ 𝝅 𝒗𝒐𝒍
We first extract the warped feature and 3D face reconstruction representation, on which FVR is then performed to obtain the 3D shape and color information for face rendering.
The 3D face reconstruction result supervises LPE and FVR by providing pose angles and 3D face vertices.
2D features achieves a consistent motion transfer of facial details.
Based on this observation, we utilize the 2D motion estimation module proposed in FOMM [52] to complete the warping process.
As shown in Fig. 3(a), we use an Hourglass Network [41] as the keypoint detector to extract K =keypoints { p S,k , p D,k ∈ R 2 } and their Jacobians { J S,k , J D,k ∈ R 2 × 2 } of both source and driving images.
For each keypoint, we utilize the first-order Taylor expansion to build an affine approximation motion filed according to [52]:
[Equation]
1) where z ∈ R 2 indicates a location in the image.
Apart from K sparse motion fields, an extra motion field is further introduced to preserve the static background information.
To generate the final dense motion field, we aggregates these K + 1 motion fields with related weight masks { M 0 ,M 1 , · · · ,M K } estimated by a U-Net used in FOMM.
Furthermore, an additional occlusion map O indicating regions to be inpainted is also predicted.
The process of obtaining the dense motion field and the warped feature F w is:
[Equation]
where f w ( · , · ) is the warping function, F S denotes the source feature and ⊙ is the Hadamard product.
3.2 3D Face Reconstruction
As shown in Fig. 3(b), we exploit the capability of a pre-trained encoder [17] to extract the parameters required by FLAME [36] and use the prior face knowledge of FLAME to reconstruct a 3D face representation from 2D images.
The reconstruction module is a typical encoder-decoder structure, where FLAME serves as the decoder, using linear blend skinning (LBS) and additional expression blend shapes to generate a detailed mesh v ∈ R 3 N with N = 5023 vertices.
Mathematically, given the parameters of shape β ∈ R | β | , pose θ ∈ R 3( K +1)
(a global rotation vector and K joints’ rotation vectors), and expression ψ ∈ R | ψ | , FLAME is represented by a blend skinning function W as:
[Equation]
where T p ∈ R 3 N depicts a head mesh in zero pose, J ∈ R 3 K denotes the locations of the K joints, and W ∈ R K × N acts as blend weights.
The camera parameter c is further estimated to scale and translate the reconstructed vertices to the camera view.
Meanwhile, the shape parameter β of the source image is assigned to the driving image as the reconstruction target.
[Figure]
Estimation &
Pose Editor
[Figure]
3.3 Face Volume Rendering (FVR)
We design a new neural rendering framework, Face Volume Rendering (FVR), for face animation as illustrated in the red box in Fig.
In this framework, we innovatively employ our designed CNNs to effectively extract the 3D color and shape features from the warped feature.
We also introduce a new loss function to optimize the 3D shape feature with the 3D reconstruction result, which is only utilized during the training process without any overhead for inference.
Moreover, we develop a new orthogonal adaptive ray-sampling module to estimate the sampled color field and voxel probability for image rendering in an efficient way.
With these novel designs, our FVR module is capable of capturing more realistic facial details than 2D rendering.
We also provide more details about our method in the supplementary materials.
3D Feature Extraction.
Based on the warped feature F w , we first need to extract the high dimensional 3D shape feature F σ and 3D color feature F color for subsequent face volume rendering.
To this end, we respectively design two CNNs P σ and P color (see the supplementary materials), by which richer 3D shape and color features can be achieved than [39].
On the one hand, P σ calculates F σ = P σ (
F w ) ∈
R H
× W × D
×
N σ , where D represents the spatial depth information and N σ is the number of channels representing the 3D shape information, which is set to 16 in this paper.
Observing that the spatial mesh v can be well reconstructed by the 3D face reconstruction module, it is thus reliable to utilize the 3D mesh feature F m derived from v to supervise P σ to achieve a reasonable result.
We maximize the inner product between vectorized F m and F σ , which leads to a matching loss function L σ :
[Equation]
where α 1 and α 2 are parameters to control the shape of the loss function.
In practice, we empirically set α 1 = 10 and α
2 = 0 .
Here, we elaborate on computing F m below.
Firstly, we conduct a down-sampling operation on the 3D vertices v to get N down sampled vertices v down , which helps to reduce the calculation cost and eliminate redundant vertex information.
Additionally, we leverage the
Gaussian function to process the i -th sampled vertex v down,i , forming a Gaussian heatmap F m,i with the size of H × W × D × 1 :
[Equation]
where σ is the variance and we set σ = 0 .
Afterward, the 3D mesh feature of the reconstruction result F m with the size of H × W × D × 1 is calculated by a weighted sum of all the Gaussian heatmaps F m,i as:
[Equation]
where w i is the weight for F m,i .
Notably, w i is predicted adaptively by an additional CNN.
It is worth noting that the 3D reconstruction vertices are not needed during inference, and we can therefore complete the image synthesis in an efficient way.
On the other hand, we need to extract the 3D color feature from the 2D warped feature F w .
To this end, we use P color to estimate the 3D color feature F color with the size of H ×
W × D × N color :
F color =
P color ( F w ) ∈
R H
× W × D
×
N color , (7) where N color is the number of channels representing the color information.
Orthogonal Adaptive Ray-Sampling.
With the extracted 3D features, we further calculate the sampled color field and the voxel probability, which are required for rendering.
Different from conventional 3D volume rendering tasks [22; 49; 44], face animation can be considered as a singleview task, which casts rays from the camera into the scene in the direction orthogonal to the image plane.
Accordingly, we design an orthogonal ray-sampling module to adaptively estimate the sampled color field p color and the voxel probability p σ , which is achieved by a single MLP-based one-stage process f θ as follows:
[Equation]
) where M color is the number of channels representing the color information of p color , and the interval sampling number D is the same as the previous spatial channel number D .
Note that p σ is the integral of the volume density with an adaptive interval size along the orthogonal ray.
In particular, the selection of the interval size is involved in estimating the voxel probability p σ and adaptively adjusted by the MLP.
It is worth noting that unlike the hierarchical volume sampling as in [39], which requires a two-stage network to process multiple views according to the rays of different directions, our orthogonal adaptive ray-sampling only needs one MLP to deal with the assumed ray at the angle orthogonal to the image plane.
Furthermore, instead of decoupling the facial attributes as geometric-prior guided sampling methods [26], we directly process the 3D related features to predict the sampled color field and the voxel probability more efficiently.
In short, by simplifying the ray-sampling with an adaptive approach, our one-stage method can significantly increase the rendering speed compared with the common two-stage process.
More details about the sampling module can refer to the supplementary materials.
Image Rendering.
Specifically, we project the sampled color field p color and the estimated voxel probability p σ onto 2D to generate the rendering feature map F r .
Our rendering method is a modified version of that in [38] according to orthogonal adaptive ray-sampling.
Also let i be a pixel point in the feature map, and j be a sample interval in each pixel i .
The rendering method calculates the final color result F r,i for each pixel i according to the corresponding sampled color field p color,i,j and voxel probability p σ,i,j as:
[Equation]
where p σ,i,j and p color,i,j represent the voxel probability and the sampled color field of the j -th interval of the i -th pixel respectively, and τ j = exp( (cid:80)
j − 1 k =1 − p σ,i,k ) is the transmittance.
Moreover, to ensure the effectiveness of the rendering feature F r , we introduce a two-layer CNN P t and the VGG perceptual loss [28; 58] in the training process.
Specifically, we directly input F r into the CNN to generate an intermediate image I m , and then compute the perceptual loss L R between
I m and the driving frame D i as:
[Equation]
Afterward, we concatenate the volume rendering feature map F r and the warped feature F w , and feed them into our designed rendering decoder which introduces the SPADE layer [45] to obtain the final generated result.
This decoder (see the supplementary materials) can enhance the rendering quality while preserving facial details of the input.
3.4 Lightweight Pose Editing (LPE)
Existing face animation methods, which adopt a 2D warping motion network [52; 56; 25], work well in the feature warping but fail on pose editing.
As illustrated in Fig. 3(a), we introduce a lightweight network (LPE) to add a new pose editing function for our framework.
Taking the rotation
[Table]
angle information { yaw ∈ R 1 , pitch ∈ R 1 , roll ∈ R 1 } provided by the 3D face reconstruction module and the keypoint information of the source image as input, LPE ( f editor ) estimates the target keypoints δ value and their Jacobians δ Jacobian .
In practice, we first formulate yaw , pitch and roll as a 3D rotation matrix r rotate ∈ R 3 × 3 , and then introduce an MLP as f editor to compute δ value and δ Jacobian under a specific rotation angle θ rotate :
[Equation]
Moreover, we design a loss L editor to supervise f editor by aligning δ value and δ Jacobian with the keypoints and the Jacobians of the driving frame, respectively:
[Equation]
where λ 1 and λ 2 are balancing parameters, and L 1 is the l 1 norm loss.
Empirically we set λ 1 =
and λ 2 = 0 .
3.5 Training
Our framework is implemented based on self-supervised learning or self-reenactment, where a pair of the source image S and the driving image D i from the same speaker in a video is used as the input and D i also acts as the ground truth for supervision.
We train our FNeVR by minimizing the following loss L total :
[Equation]
where x generated is the output of the whole network FNeVR.
L total consists of four parts: (1) L P and L G are used to maintain the quality of the generated image; (2) L E is used to adjust the 2D motion estimation module to predict reasonable motion fields; (3) L R and L σ are used to optimize the 3D volume rendering module to generate better 3D rendering features; (4) L editor is used to optimize the face pose editor.
For (1) and (2), we employ the same loss functions as in [52], including the perceptual loss L P , GAN loss L G and equivalence constraint loss L E .
For (3) and (4), the losses mainly provide the quality guarantee of the 3D rendering feature F r .
Experiments
In addition to the extensive experiments described in this section, we also provide more results in the supplementary materials.
4.1 Implementation Details
Datasets.
Our evaluation is performed on VoxCeleb [40], which contains more than 100,000 videos covering 1,251 speakers of different identities, and VoxCeleb2 [12], which contains about 1M videos of different celebrities.
We adopt the same preprocessing strategy of cropping faces from the videos and resizing them to 256×256 as in [52].
Training Details.
We train FNeVR with self reconstruction on the training set, in which a pair of source and driving images from the same speaker in a video are used as input, and the driving image also serves as ground truth for supervision.
FNeVR is trained for 100 epochs, repeating the video
[Figure]
[Table] [Table]
image set 75 times per epoch.
We adopt Adam [34] optimizer with learning rate η = 2 × 10 − 4 , β 1 = 0 .
5 and β 2 = 0 .
9 for each module.
Since our model is lightweight, we only use 2 24GB
NVIDIA 3090 GPUs during training.
Evaluation Metrics.
The metrics include (1) reconstruction faithfulness using L 1 , PSNR, SSIM [61], and LPIPS [66], (2) output visual quality using FID [24], (3) Average Keypoint Distance (AKD) [52], (4) Average Euclidean Distance (AED) [52], and (5) identity preservation cosine similarity CSIM [48; 27].
4.2 Comparison with State-of-the-Art Methods
Methods.
We compare our FNeVR with five state-of-the-art (SOTA) models: FOMM [52], Face vid2vid [59], Face vid2vid-S [59], Bilayer [64], DaGAN [25], and PIRenderer [47].
We use the official pre-trained models for Bilayer, FOMM, DaGAN and PIRenderer, and a widely recognized unofficial model for Face vid2vid due to the absence of the official code.
Face vid2vid-S is obtained by replacing the decoder of Face vid2vid with a SPADE [45] layer-based decoder.
All of these models are trained on VoxCeleb.
[Figure]
Configurations
[Table]
Same-Identity Reconstruction.
We conduct a quantitative comparison with SOTA methods on the testing dataset of VoxCeleb when the source image and the driving image belong to the same person.
As shown in Table
[Table]
it is evident that our FNeVR outperforms other SOTA methods on all metrics.
Especially, FNeVR achieves pronounced improvements on several metrics, such as image quality (PNSR), semantic consistency (AKD)
[Table]
and image authenticity (FID).
Fig. 4(a) shows that FNeVR can generate the most realistic images that are most similar to the driving faces.
Cross-Identity Reenactment.
We compare the performance of our FNeVR with SOTA methods on the VoxCeleb and VoxCeleb2 testing datasets when the source image and driving video come from different persons.
Concretely, we randomly select 10 source images and 14 driving videos from the testing datasets.
Table 2 illustrates that FNeVR produces the best overall performance.
Although
Face vid2vid-S has a better result in terms of CSIM, it consumes several times more computation and memory cost (FLOPs and Parameters) than FNeVR, as indicated in Table 3, demonstrating FNeVR’s outstanding efficiency in compact computation.
Moreover, Fig. 4(b) visualizes some synthesis results from different methods of reenactment and shows that FNeVR can generate more realistic faces whose poses and expressions are closest to the driving faces.
4.3 Ablation Study
We conduct comprehensive experiments on the same-identity reconstruction task to further demonstrate the effectiveness of each module in the proposed FNeVR.
Specifically, we employ FOMM [52] as the baseline, and construct four comparison models: (1) replacing FOMM’s decoder with our designed decoder, (2) inserting FVR into FOMM, (3) FNeVR without F w inputted into the decoder, and (4) FNeVR without L σ .
The results are shown in Table 4.
Both FVR and our designed decoder have significant effectiveness.
In particular, FOMM with FVR inserted produces much better results than the baseline.
Furthermore, FNeVR without F w performs worse than the full FNeVR, indicating that not inputting F w into the decoder will affect the model’s ability to transfer the facial poses and expressions, and lead to the loss of facial details in the generated results.
Furthermore, the results show that if L σ is not introduced, all metrics are worse than the full FNeVR, indicating that it is necessary to introduce reliable 3D information.
Meanwhile, even without the 3D information, FNeVR still performs well, demonstrating the great effectiveness of the proposed framework.
Note that our full model produces the best overall results and surpasses other models a lot in several metrics, fully verifying the efficacy of FNeVR.
4.4 Lightweight Pose Editing
With the pose parameters, we conduct a qualitative experiment with pose editing visualization in
Fig.
5 to further evaluate the effectiveness of the proposed LPE.
Specifically, according to the Euler angle ( yaw , pitch , and roll ) provided by the 3D face reconstruction result, we separately adjust the rotation effect of the generated results.
Fig.
5 shows that LPE can generate reliable face rotations according to a given Euler angle.
Conclusion
In this paper we propose a Face Neural Volume Rendering (FNeVR) network for face animation, which unifies the 2D motion warping and 3D volume rendering in one framework.
We follow the line of 2D motion warping and introduce the 3D face reconstruction module to provide reliable 3D shape information for the rendering process.
We innovatively develop a Face Volume Rendering (FVR) module to enhance the facial details of the warped feature and generate high-quality faces.
Moreover, we design a Lightweight Pose Editing (LPE) module, which can directly implement pose editing with rotation angles.
Extensive experiments illustrate that our FNeVR achieves state-of-the-art performance
Acknowledgements
This work was supported by “the Fundamental Research Funds for the Central Universities” and the National Natural Science Foundation of China under Grant 62076016, Beijing Natural Science
Foundation-Xiaomi Innovation Joint Fund L223024.
Besides, we gratefully acknowledge the support of MindSpore 2 , CANN (Compute Architecture for Neural Networks) and Ascend AI processor used for this research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] , please refer to the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] , please refer to the supplemental material.
(b) Did you include complete proofs of all theoretical results?
[Yes] , please refer to the supplemental material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Derivation of the Volume Rendering Formula
This section provides the derivation of our 3D volume rendering formula Eq. 9 in the paper.
Previous methods adopt a two-stage architecture to realize multi-view synthesis discretely, which is very time-consuming with redundant information.
Hence we propose an orthogonal adaptive raysampling module to adaptively estimate the attributes required for image rendering of single-view face animation.
A.1 Classical Discrete Volume Rendering Formula
Generally speaking, the volume rendering formula needs to be transformed from continuous integration to discrete summation, for which [39] designs a two-stage sampling method to realize the multi-view synthesis of a single object.
Specifically, the classical volume rendering [29] calculates the pixel color along the camera ray as follows:
[Equation]
where σ represents the volume density, c is the emitted color, [ t s , t e ] is the integral interval, and τ is the transmittance.
To make it implementable, the continuous integration is transformed to discrete summation with D intervals.
With some manipulations, [38] shows:
[Equation]
where δ i =
t i +1 − t
i is the i -th interval size, σ i
and c i are constants to approximate the volume density and color values in [ t i , t i +1 ], respectively, and τ i = exp ( − (cid:80) i − 1 j =1 σ j δ j ) .
A.2 Volume Rendering of FNeVR
We introduce an orthogonal ray adaptive sampling module for single-view face animation.
Following [39], we perform ray casting in the normalized device coordinate (NDC) space.
For each image, we define N = H × W rays, where W and H are the width and height of the input image, respectively.
Each ray is orthogonal to the near clipping plane, and the ray path from the pixel centre is calculated.
According to Eq.
15 above, conventional neural rendering first uses a "coarse" network to roughly estimate the computational contribution of each uniform sampling location, and then further performs non-uniform discretization for a more informed sampling of points, which are inputted to another "fine" network for accurate estimation.
Instead, our FVR only introduces one MLP network to directly estimate the voxel probability p σ of each voxel, which is the integral of the volume density σ within a suitable interval size δ .
Therefore, for our model, the selection of the interval size δ is actually involved in the estimation of the voxel probability p σ and adaptively adjusted by the MLP.
Finally, we rewrite Eq.
15 as the weighted sum of all sampled colors p color for each ray as:
[Equation]
which is Eq.
9 in the main paper.
It is worth noting that our FVR can be applied to more generation tasks as long as appropriate 3D shape and color information is provided.
For example, if we introduce reliable 3D body information to FVR, FVR can be applied to generate photo-realistic images of the body.
[Figure]
For the basic blocks UpBlock2D and DownBlock2D, please refer to Fig. 8(c) and Fig
8(d)
[Figure]
and SPADE, please refer to Fig. 8(e) and Fig. 8(b), respectively.
B Implementation Details B.1 Architecture Details
Keypoint Detector.
We employ the keypoint detector proposed by [52], which takes one image as input and adopts a UNet-based encoder-decoder architecture to extract the feature for keypoints prediction.
Since we adopt a 2D warping strategy, only 2D CNN is required.
Its detailed structure is shown in Fig. 6(a).
Dense Motion Network.
We use the dense motion network proposed by [52] to predict an occlusion map indicating the regions to be inpainted and a group of weight masks related to the K sparse motion fields, which are calculated by each pair of keypoints { p S,k , p D,k } with their Jacobians.
Then we can aggregate the sparse motions by a weighted sum to obtain the final dense motion field.
The detailed structure of the dense motion network is shown in Fig. 6(b).
In particular, the input to the network is the concatenation of K heatmaps and K + 1 deformed source images, where the heatmaps are calculated by the keypoints, and the deformed images are transformed by K + 1 motion fields.
Instance Normalization 3 × 3-Conv-C ReLU 3 × 3-Conv3 × 3-Conv-C 3 × 3-Conv-C β γ Norm ( 1 + 𝛾 ) * Norm + β
Modulation Input
ReLU 3 × 3-Conv-C SPADE Layer
SPADE Layer (b) SPADE-C 3 × 3-Conv-C, BN, ReLU, DownsampleUpsample-2, 3 × 3-Conv-C, BN, ReLU
BN, ReLU, 3 × 3 × 3-Conv-C
BN, ReLU, 3 × 3 × 3-Conv-C (c) UpBlock2D-C (d) DownBlock2D-C (e) ResBlock3D-C (a) SPADE Layer
[Figure]
Modulation
[Figure]
Output
[Figure]
Output
3D Feature Extractor.
Since we need to extract the 3D information of the warped feature for subsequent image rendering, we design a color extractor and a shape extractor, both of which first refine the 2D feature with the size of 256 × 64 × 64 into 3D features with the size of 16 × 8 × 64 × 64 .
The detailed structure of the 3D feature extractors is shown in Fig. 7(a).
N σ and N color are the final dimensions of the outputs of the color extractor and the shape extractor, respectively, and they are both set to 16.
Orthogonal Adaptive Ray-Sampler.
Based on the extracted 3D features above, we can further calculate the sampled color field and the voxel probability, which are required for volume rendering.
We adopt an MLP-based network to estimate the sampled color field with the size of 64 × 64 × 8 ×and the voxel probability with the size of 64 × 64 × 8 × 1 .
The detailed structure of the orthogonal adaptive ray-sampler is shown in Fig. 7(b).
Rendering Decoder.
We design a rendering decoder based on the SPADE layer [45].
We adopt a series of the SPADE layers and upsampling layers to transform the warped feature and the volume rendering results into a photo-realistic image.
The detailed structure of this decoder is shown in Fig.
7(c).
B.2 Details of the Loss Functions
Perceptual Loss.
During the training process, we use the VGG perceptual loss [28] to align the high-level semantic feature of the generated image with that of the ground truth.
Specifically, we use the pre-trained VGG-19 [53] model f V GG to extract L features of the generated image x generated and the driving frame D i and use the L 1 norm loss for similarity evaluation.
Following [52], we first adopt the multi-scale pyramid by downsampling the generated image and ground truth to obtain four different resolution results, and then we aggregate the losses of the four resolutions to obtain the final result:
[Equation]
[Equation]
GAN Loss.
We use a discriminator D GAN with N feature extractors to distinguish the generated results from the ground truth in M scales.
To be specific, we use the same generator-discriminator
GAN loss as [52], including the least square loss [37] and the feature matching loss [58] to train an adversarial generative network:
[Equation]
L G,generator,matching =
E
M (cid:88) m =
N (cid:88) n =
L 1 ( D ( n ) GAN,m ( D i,m ) , D ( n ) GAN,m ( x generated,m )) , (19)
L G,discriminator =
M (cid:88) m =
E [(1 − D GAN,m ( D i,m )) 2 ] + E
[( D GAN,m ( x generated,m )) 2 ] , (20) where we denote the m -th scale discriminator as D GAN,m and its n -th layer feature extractor as
D ( n ) GAN,m .
With the GAN loss, we further improve the authenticity of the generated images.
Equivariance Loss.
To ensure the validity and consistency of the unsupervised keypoint detector, we use a known geometric transformation T to provide the equivariance constraints as stated in [52].
Concretely, for the keypoints { p T ( D ) ,k } , which are extracted from the transformed image T ( D ) , and the keypoints { p D,k } of the original image D , we have:
[Equation]
21) where T − 1 is the inverse transformation of T .
Furthermore, we optimize the Jacobians { J D,k } by:
[Equation]
(22) where 1 denotes the 2 × 2 identity matrix, J T − 1 is the Jacobian of the inverse geometric transformation
T − 1 , and { J T ( D ) ,k } are the Jacobians detected from the transformed image T ( D ) .
C Experiment Details C.1 Evaluation Metrics
We quantitatively compare our FNeRV with other state-of-the-art methods using the following metrics:
L 1 Distance.
We compute the averaged L 1 distance between the generated and ground-truth images
Learned Perceptual Image Patch Similarity (LPIPS).
LPIPS [66] estimates the perceived distance of the generated image from the ground truth image by computing the cosine distances between the VGG network [53] features of the two images layer by layer and averaging them.
Peak Signal-to-Noise Ratio (PSNR).
Using the ratio of the maximum possible power of the ground truth image to the mean square error between the reconstructed image and the ground truth to measure the image reconstruction quality.
Structure Similarity Index Measure (SSIM).
From the perspective of image distortion modelling discussed in [61], the structural similarity of two images is measured by considering three different factors, brightness, contrast, and structure, where the mean is used as an estimate of brightness, the standard deviation as an estimate of contrast, and the covariance as a measure of structural similarity.
Average Keypoint Distance (AKD).
We use a facial landmark detector [8] to detect facial landmarks from the reconstructed image and the ground-truth image and calculate the average distance between the two groups of landmarks.
AKD can measure the accuracy of the facial posture of the generated frame and driving frame.
Average Euclidean Distance (AED).
We use the face recognition network [3] to calculate the feature representations of the ground-truth image and the generated video frame and calculate their averaged Euclidean distance.
[Figure]
[Table]
[Figure]
[Table] [Table]
Frechet Inception Distance score (FID).
FID [24] is a metric to evaluate the image quality created by the generated model by imitating human perception of image similarity.
It uses inception V3 [54] to compare the distribution of the generated image with the distribution of the real image used to train the generator.
Cosine Similarity (CSIM).
The cosine similarity measures the difference between two individuals by using the cosine value of the angle between two vectors in a vector space.
We use the pre-trained face recognition model [27] to generate embedded vectors to compute the cosine similarity to assess the quality of identity preservation.
In the same-identity case, we perform an experiment on all test videos from the Voxceleb [40] dataset.
We treat the first frame as the source portrait and all the other frames as the driving video.
Hence, we can get the synthesized frames with corresponding ground truth frames.
We can directly calculate the evaluation metrics of L 1 , LPIPS, PSNR, SSIM, AKD, AED and FID between the real images and the synthetic images.
For the cross-identity case, we randomly select 10 source images anddriving videos from each testing set of the Voxceleb and Voxceleb2 [12].
In this case, we do not have ground-truth images, so we can only calculate FID between the driving images and the synthetic images, and CSIM between the source images and the synthetic images.
C.2 Reenactment Analysis
In the reenactment experiment, our FNeVR outperforms other SOTA methods on FID but has a certain gap with Face vid2vid [59] on CSIM.
CSIM verifies the identity preservation according to the feature distance between the generated image and the source image, and is adopted in a number of studies [16; 59; 25].
According to our analysis, there are two reasons accounting for why CSIM of
Face vid2vid is better than our FNeVR.
On the one hand, 3D warping indeed provides a relatively stable face pose transformation with a high computational cost.
On the other hand, we argue that it is not entirely reasonable to use CSIM to examine the preservation of face identity information, since 3D warping often leads to poor facial expression transformation, whereas it still produces good
CSIM.
We think that expression and pose variation have a significant impact on CSIM, which can be proved by a simple experiment.
Specially, we use three feature extraction networks (MoCo v2 [11],
Arcface[13], and Curricularface [27]) to calculate the CSIM values between one source image and five sample images with the same person identity as the source image but different poses or expressions.
As shown in Fig.
9, compared with the source image, Sample 1 and Sample 2 are slightly different in expressions, while Sample 3 has a great difference in pose.
Meanwhile, Sample 4 and Sample 5 are disparate from the source image in both poses and expressions.
The comparison of CSIM is shown in
Table 5.
Note that only Arc CSIM and Curricular CSIM focus on facial feature extraction.
Obviously, despite the same identity, the variations in both poses and expressions have a great impact on CSIM,
[Figure] [Figure]
especially for detectors focusing on face features.
Especially, there are pronounced decreases in
CSIM for Sample 3, Sample 4 and Sample 5, indicating the considerable influence of facial poses.
In other words, rich variations in facial poses and expressions make it difficult to maintain relatively high CSIM.
Consequently, it is understandable that our model obtains lower CSIM than Face vid2vid.
Finding more reasonable metrics to verify the preservation of identity information is part of our future research.
D Additional Experimental Results D.1 Pose Editing
We design an MLP-based lightweight pose editor (LPE), and its detailed structure is shown in Fig.
In the training process, we input the deviations of the Euler angles yaw , pitch , and roll between the source image and the driving frame provided by the 3D reconstruction network into LPE, together with the keypoint information of the source image.
In the inference process, we can directly control the face pose with specific yaw , pitch , and roll angles.
D.2 Additional Comparisons with State-of-the-Art Methods
Reconstruction.
In Fig.
11, we show three animation examples from the VoxCeleb [40] dataset, in which there is significant expression variation between each pair of two faces.
From the generated
[Figure]
results, we see that FOMM [52] can generate promising expression transformed images but with blurred head profiles, while Face vid2vid [59] and DaGAN [25] generate the images with clear head profiles but poor expression transformation.
For instance, Face vid2vid and DaGAN cannot transfer the visual directions or the corners of the mouth very well.
Our FNeVR can combine their merits and generate promising expression transformed images with clear head profiles.
Reenactment.
To enrich the reenactment experimental results shown in the main paper, we conduct additional experiments and show more samples in Fig.
Notably, in the cross-identity reenactment experiment of the main paper, all FOMM-based models [52; 59; 25] do not employ the relative motion transfer approach [52], in which the frame D best in the driving video whose pose is most similar to the source image S is selected as the basis, and the motion fields of other driving images
D i are calculated by the motion differences from D best .
Utilizing the relative motion transfer, we can maximize the preservation of identity information.
Therefore, we conduct additional reenactment experiments based on this approach on VoxCeleb [40], and the results are shown in Fig.
It can be seen that Bilayer [64] has a greater advantage in maintaining identity information but cannot handle the image background.
FOMM [52] has a desirable performance in expression transfer, but the generated results are distorted in some areas.
Face vid2vid [59] works best on identity information due to the use of 3D representation.
However, the facial details are not well captured, such as the inconsistency of the irises between the driving images and the source images.
DaGAN [25] has a significant deficiency in maintaining identity information, and the contours of the faces in the generated images are completely deformed.
Both Fig.
12 and Fig.
13 show that our FNeVR achieves the best results by maximizing the transfer of facial expressions without losing the facial features.
E Social Impact
This work does not have a direct negative social impact.
However, we should be aware of the power of realistic face animation and prevent it from being abused for malicious purposes.