Paper: DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
While the style encoder offers more customizability to the generated face motions, it seems to be limited to the reference videos received during training. Is this understanding correct, or can the style encoder generalize to novel styles during inference?

Author response:
The style encoder can generalize to novel styles during inference. In fact, all the experiments and examples in this paper used novel videos as the reference videos.

Reviewer comment (Evaluation):
What is the latency of the end-to-end generation pipeline during inference? What is the latency of the stylization component?

Author response:
Although our method can inference at 30 fps, for live streaming scenarios, it introduces a 7.33-second delay. This delay is primarily due to our windowing strategy, which involves an initial 4 seconds wait to fill the first window and an additional 3.33 seconds for processing. However, once this initial processing is complete, our inference does not introduce any further latency. Regarding the style encoder, inference on a reference segment takes just 2ms, and the style feature can actually be extracted beforehand. We have added these discussions to Sec. A.3 of the revised appendix.

Reviewer comment (Theory):
For Eqn. 2, is there any specific reason to operate on the vertex space and on the parameter space? The vertex space is much higher dimensional and less constrained, which could make the training more unstable.

Author response:
Actually, Eqn. 2 (the simple loss), calculates the MSE of the predicted results and the ground truth in the parameter space instead of in the vertex space. We only calculate the geometric losses in the vertex space because the vertex space can yield better loss guidance for the position, velocity, and acceleration of face vertices than the parameter space.

Reviewer comment (Evaluation):
Limitations are not elaborated.

Author response:
The following discussions on limitations have been added to the revised appendix.
“Although our method is able to generate high-quality stylistic 3D talking face animation with vivid head poses, there are still some limitations within our framework that could be addressed in follow-up works. Firstly, the computational cost of inference is relatively high due to the sequential nature of the denoising process. To mitigate this, future research can explore more advanced denoisers such as DPM-solver++. Secondly, like existing SOTAs, our approach focuses on animating the face shape while ignoring the inner mouth (including teeth and tongue). Exploring representation and animation of the inner mouth can lead to more realistic results.  Lastly, a promising direction for future research would be to collect real-world 3D talking data that encompasses a broader range of identities and styles, which would further enhance the effectiveness of our approach and contributes to the research community.”

Reviewer comment (Theory):
In Section 4.1 it is motivated the use of synthesis data as a source for training the proposed architecture. It is also reported the 3D face reconstruction method in Filntisis et al., 2022) based on a 3DMM is used for accurate reconstruction of lip movements. This reconstruction than learning of a set of 3DMM parameters with related lips movements based on these 2D datasets. However, this is not much convincing to me. Lips movements associated to speech are quite subtle and with specificity from each individual. Reconstructing such subtle movements with a 3DMM from a 2D video sequence seems not capable of capturing the reality of movement. In my opinion, authors should convince the readers more about this point which is also fundamental for the proposed approach.

Author response:
Thank you for the valuable comments. 

We understand that your main concern is regarding our use of "3D data reconstructed from 2D datasets" rather than "scanned real 3D data" in our work. We would like to respond to this concern first, and respond to other detailed questions in the next comment.

Firstly, we emphasize that our contribution is to propose a novel and **general** diffusion-based method for generating speech-to-3D animation. Specifically, our method is able to control the speaker's style using any reference style and support the prediction of head movements. All reviewers agree that this application scenario is valuable. We also note that **existing scanned real 3D datasets cannot be used to build algorithms for this new scenario** due to their limited scale and coverage of identities, styles, and head movements. Therefore, there are two possible strategies: one is to collect new scanned 3D real data, and the other is to reconstruct from existing 2D data. The method proposed in this paper belongs to the latter. We are also very much looking forward to someone coming up with methods of the former in the future, and comparing them with ours. 

Secondly, **the strategy of using synthetic data from 3D reconstruction is well-established in various SOTA methods**. Due to the limited scale and coverage in existing 3D real datasets, three recent SOTAs [1-3], including two on emotional 3D facial animations generation [2, 3], used reconstructed data from 2D. In addition, [3] also adopted the reconstruction method we used. Additionally, many successful 2D talking head videos methods that use 3DMM as an intermediate face representation, such as [4-7], also utilize reconstructed data to train their speech-to-3D animation generation networks. Thus, **the practice of using reconstructed data is reasonable and well-established in this field**.

We understand your concern about the quality of reconstructed lips. However, this is not a key issue in our method because existing SOTA reconstruction methods can reconstruct reasonable lip shapes most of the time, especially since **we used a method optimized for lip shape reconstruction (Filntisis et al., 2022)**. Quantitative results, qualitative experiments, and user studies in this paper demonstrated that our method achieved satisfactory generation and outperformed baseline methods. Moreover, **our method does not rely on how 3DMM coefficients are obtained**. In fact, the regression-based reconstruction method we currently use achieves satisfactory results. Of course, if an advanced optimization-based reconstruction algorithm is used or an even stronger regression-based reconstruction algorithm that can provide accurate 3DMM coefficients becomes available in the future, our method can readily use it. 

Lastly, we would like to point out that our method’s usage of 3DMM coefficients presents several advantages over the methods utilizing scanned 3D meshes.  First, 2D video data is much easier to collect and reconstructed 3DMM data can **cover a broader range**, leading to models with **stronger generalization capabilities**. Second, different from 3D meshes, the lower dimensionality of 3DMM coefficients **reduces computational load and enhances inference speed**, which is crucial for diffusion models. This reduced dimensionality, combined with blendshapes as a prior, **makes the** **learning process easier and improves generalization** [2]. Third, using 3DMM coefficients **facilitates integration with downstream applications** such as driving rig-based gaming avatars or 3DMM-based expression editing [8, 9].

- [1] C. Zhang et al., “3D Talking Face With Personalized Pose Dynamics” , TVCG, 2023.
- [2] Z. Peng et al., “EmoTalk: Speech-driven emotional disentanglement for 3D face animation”, ICCV, 2023.
- [3] R. Daněček, K. Chhatre, S. Tripathi, Y. Wen, M. J. Black, and T. Bolkart, “Emotional Speech-Driven Animation with Content-Emotion Disentanglement”, SIGGRAPH Asia Conference, 2023.
- [4] J. Thies, M. Elgharib, A. Tewari, C. Theobalt, and M. Nießner, “Neural Voice Puppetry: Audio-driven Facial Reenactment”, ECCV, 2020.
- [5] Z. Zhang, L. Li, Y. Ding, and C. Fan, “Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset”, CVPR, 2021.
- [6] W. Zhang et al., “SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation”, CVPR, 2023.
- [7] C. Xu et al., “High-Fidelity Generalized Emotional Talking Face Generation With Multi-Modal Emotion Space Learning”, CVPR, 2023.
- [8] Z. Geng, C. Cao, and S. Tulyakov, “3D Guided Fine-Grained Face Manipulation”, CVPR, 2019.
- [9] Z. Sun et al., “Continuously Controllable Facial Expression Editing in Talking Face Videos,” arXiv, 2022.

Reviewer comment (Evaluation):
The supplemental video material is not fully convincing. For some speech the lips movement and the audio track are not well synchronized. The lips movement is also quite synthetic with the lips that are not able to close. The movement is also on the lips only without facial expression shown during face lips animation.

Author response:
Q2: Supplemental video: for some parts, the lip movements and the audio track are not well synchronized, and movement is on the lips only without facial expression shown during face lips animation.

A2: First, we clarify that **our method can automatically generate upper face motions**, such as eye blinks (e.g., in demo video 2:07-2:20), that are weakly-related or unrelated to speech, as stated in “Qualitative Evaluation” in Sec. 4.3. In contrast, the other methods show less upper face motions. Our FDD metrics also surpass other methods by a wide margin, **which indicates that our upper face motions are closer to the ground truth**. This improvement is largely because we adapted the diffusion model for facial animation generation. The probabilistic approach can capture the data distribution of speech-unrelated movements more effectively than regression-based methods. Second, our method **shows significant improvements in lip synchronization, style control, and naturalness over previous SOTA methods**, as demonstrated by our quantitative results, qualitative experiments, and user studies.

Reviewer comment (Writing):
There is not much discussion on the proposed method. In particular, limitations of the proposed approach are not evidenced.

Author response:
Q3: More limitation discussions on the proposed method.

A3: The following discussions on limitations have been added to the revised appendix.
“Although our method is able to generate high-quality stylistic 3D talking face animation with vivid head poses, there are still some limitations within our framework that could be addressed in follow-up works. Firstly, the computational cost of inference is relatively high due to the sequential nature of the denoising process. To mitigate this, future research can explore more advanced denoisers such as DPM-solver++. Secondly, like existing SOTAs, our approach focuses on animating the face shape while ignoring the inner mouth (including teeth and tongue). Exploring representation and animation of the inner mouth can lead to more realistic results.  Lastly, a promising direction for future research would be to collect real-world 3D talking data that encompasses a broader range of identities and styles, which would further enhance the effectiveness of our approach and contributes to the research community.”

Reviewer comment (Experiments):
An experiments using real data could have been reported (e.g., VOCASET) for a better comparison with other methods.

Author response:
Q5: Report an experiment using real data (e.g., VOCASET) for a better comparison.

A5: We have compared our method with FaceFormer and CodeTalker on the VOCASET. We list the quantitative results here. More details about the experiment can be found in Sec. A.6 of the revised appendix, and qualitative results can be found in the updated demo video (5:12-5:31).

| Method     | LVE (mm) $\downarrow$ | FDD ($\times 10^{-5}$m $\downarrow$) |
| ---------- | ----------- | ----------- |
| FaceFormer | 4.43    | 13.61    |
| CodeTalker | 4.55    | 10.34    |
| Ours       | **4.35** | **8.85** |

Reviewer comment (Experiments):
While the paper claims that they can meaningfully extract "style" from the reference videos, in the examples shown in the video (2:42), they don't show the reference styles so it's hard to understand how faithfully the reference style was matched, further only a a single example for the same is shown.

Author response:
Q2: Show more examples and the reference styles in video for better understanding.

A2: We have added more examples with both the style reference videos and the generated videos in the updated demo video (2:34-3:53).

Reviewer comment (Evaluation):
In the user-study the authors claim that they have superior scores, but none of the user study samples are shared, so it's hard to understand what the users were really scoring.

Author response:
Q3: Share user study samples for better understanding. 

A3: We have added screenshots and more detailed description of our user study in Sec. A.5 of the revised appendix.

Reviewer comment (Evaluation):
What metric will you use to measure the disentanglement between style and content? Can you show a couple of example of style or content muted performances (where the original video is assumed to have both)?

Author response:
Q5: Metric to measure the disentanglement between style and content, and show more examples of style or content muted performances

A5: Similar to the challenges in quantitatively evaluating the disentanglement of emotion and content in EmoTalk (Peng et al., 2023), quantitatively measuring the disentanglement of style and content presents similar difficulties. However, it is possible to evaluate this qualitatively. To illustrate, we did a qualitative experiment: we randomly selected 10 speaking styles and 20 audio clips to generate 10*20 animations. The speaking styles from these animations were then extracted using our style encoder. We employed t-SNE for visualizing these extracted speaking styles. The results demonstrated that animations with identical reference styles cluster together, regardless of content differences. For more details, please refer to Sec. A.4 in the revised appendix . In addition, we also show examples of style or content muted performances in the demo video (4:28-4:46).