Paper: GAIA: Zero-shot Talking Avatar Generation (ICLR-2024)

Reviewer comment (Experiments):
The authors did not provide a large number of generation examples, and it would be desirable to see more visual results.

Author response:
A: Thank you for your interest. We provide more samples on the anonymous **[demo page](https://gaiavatar.github.io/gaia/)** for convenience. More specifically, 1) an example of "long video generation" is provided in **[this video](https://gaiavatar.github.io/gaia/video/more/long_video.mp4)**. Since we focus on demonstrating the effectiveness of our proposed method, the long video generation is not yet well explored in the original manuscript. According to your comments, we present a simple yet effective strategy to generate infinite long videos -- by fixing the first several frames of each generation step during the reverse diffusion process. In this way, we obtain a smooth transition between each sub-video. 2) more examples of "different reference frame but the same driving video" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_reference_same_driving.mp4)**. 3) more examples of "different driving video but the same reference frame" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_driving_same_reference.mp4)**.

We hope the above videos could make our results more clear.

Reviewer comment (Experiments):
Long videos for the generated talking avatars should be provided as examples.

Author response:
A: Thank you for your interest. We provide more samples on the anonymous **[demo page](https://gaiavatar.github.io/gaia/)** for convenience. More specifically, 1) an example of "long video generation" is provided in **[this video](https://gaiavatar.github.io/gaia/video/more/long_video.mp4)**. Since we focus on demonstrating the effectiveness of our proposed method, the long video generation is not yet well explored in the original manuscript. According to your comments, we present a simple yet effective strategy to generate infinite long videos -- by fixing the first several frames of each generation step during the reverse diffusion process. In this way, we obtain a smooth transition between each sub-video. 2) more examples of "different reference frame but the same driving video" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_reference_same_driving.mp4)**. 3) more examples of "different driving video but the same reference frame" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_driving_same_reference.mp4)**.

We hope the above videos could make our results more clear.

Reviewer comment (Presentation):
Examples showing different reference video/frame but the same driving video should be included.

Author response:
A: Thank you for your interest. We provide more samples on the anonymous **[demo page](https://gaiavatar.github.io/gaia/)** for convenience. More specifically, 1) an example of "long video generation" is provided in **[this video](https://gaiavatar.github.io/gaia/video/more/long_video.mp4)**. Since we focus on demonstrating the effectiveness of our proposed method, the long video generation is not yet well explored in the original manuscript. According to your comments, we present a simple yet effective strategy to generate infinite long videos -- by fixing the first several frames of each generation step during the reverse diffusion process. In this way, we obtain a smooth transition between each sub-video. 2) more examples of "different reference frame but the same driving video" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_reference_same_driving.mp4)**. 3) more examples of "different driving video but the same reference frame" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_driving_same_reference.mp4)**.

We hope the above videos could make our results more clear.

Reviewer comment (Experiments):
Examples showing different driving video but the same reference video/frame should be included.

Author response:
A: Thank you for your interest. We provide more samples on the anonymous **[demo page](https://gaiavatar.github.io/gaia/)** for convenience. More specifically, 1) an example of "long video generation" is provided in **[this video](https://gaiavatar.github.io/gaia/video/more/long_video.mp4)**. Since we focus on demonstrating the effectiveness of our proposed method, the long video generation is not yet well explored in the original manuscript. According to your comments, we present a simple yet effective strategy to generate infinite long videos -- by fixing the first several frames of each generation step during the reverse diffusion process. In this way, we obtain a smooth transition between each sub-video. 2) more examples of "different reference frame but the same driving video" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_reference_same_driving.mp4)**. 3) more examples of "different driving video but the same reference frame" are provided in **[this video](https://gaiavatar.github.io/gaia/video/more/different_driving_same_reference.mp4)**.

We hope the above videos could make our results more clear.

Reviewer comment (Reproducibility):
Will the proposed method and the established dataset be publically available?

Author response:
A: We promise that the code will be released upon publication. For the dataset and pre-trained models, since privacy information is involved (i.e., real human faces), we are going through an internal compliance review and will try our best to mitigate the privacy risks and release them to facilitate future research.

Reviewer comment (Experiments):
A comparison with https://arxiv.org/pdf/2012.08261.pdf for video driven is critically missing as a recently proposed SOTA method. In their paper they show improvements compared to face-vid2vid and FOMM which are used as baselines here and they provide a pretrained checkpoint.

Author response:
A: Thanks for reminding us of this important related work! We evaluate this model with the released checkpoint on our test set. To ensure fair comparisons, in addition to the model trained on our dataset, we also train one smaller GAIA model with 80M parameters on the VoxCeleb2 dataset to keep consistent with HeadGAN. We test models on the video-driven self-reconstruction and cross-reenactment settings, and the results are shown below.

Table 5. Quantitative comparisons of self-reconstruction with HeadGAN.

|                       |    FID$\downarrow$ |   LPIPS$\downarrow$   |   PSNR$\uparrow$ | AKD$\downarrow$  |   MSI$\uparrow$ |
| --------------------- | ----- | ----- | ----- | ----- | ----- | 
| HeadGAN          | 21.499 | 0.278  | 18.555 | 2.990  | 0.835 |
| GAIA-small w/ VoxCeleb2 | 16.099 | 0.173  | 22.896 | 1.434  | 1.083 |
| GAIA | 15.730 | 0.167  | 23.942 | 1.442 | 0.856 |

Table 6. Quantitative comparisons of cross-reenactment with HeadGAN.

|                       |  FID$\downarrow$ | AKD$\downarrow$   |  MSI$\uparrow$  |
| --------------------- | ----- | ----- | ----- |
| HeadGAN    | 90.746 | 5.964  | 0.788 |
| GAIA-small w/ VoxCeleb2 | 27.643 | 2.968  | 1.035 |
| GAIA | 15.200 | 2.003  | 1.102

From the results, GAIA achieves consistent improvements over HeadGAN across different metrics and settings. We have added the comparison with HeadGAN to Table 2 and 7 in the manuscript.

Reviewer comment (Reproducibility):
Will the dataset be shared as part of this submission?

Author response:
A: We promise that the code will be released upon publication. For the dataset and pre-trained models, since privacy information is involved (i.e., real human faces), we are going through an internal compliance review and will try our best to mitigate the privacy risks and release them to facilitate future research.

Reviewer comment (Experiments):
It would be interesting to see an ablation on training data size to assess whether there are benefits from scaling data further.

Author response:
A: We have investigated the scaling of training data in Tables 4 \& 5 of the manuscript, where \#Hours indicates the total training size. It can be observed that, with the same model size (80M VAE and 180M diffusion), our method obtains better results on more training data (e.g., 17.486 vs. 18.353 FID score, 8.913 vs. 9.145 Sync-D score). It is also interesting to see what will happen on larger data size (i.e., more than 1K hours). We leave it as future work.

Reviewer comment (Evaluation):
How does randomness from the diffusion model affect generations?

Author response:
A: Thank you for your interest! To show the effect of randomness, we conduct three kinds of visualizations in **[this video](https://gaiavatar.github.io/gaia/video/more/randomness.mp4)**: 1) we use different random seeds (i.e., seed 42 \& 82) for the generation, from which we observe that different seeds make slight differences for the generated videos; 2) we evaluate the models from different training epochs (i.e., epoch 1599 \& 1799), which shows the large difference on the generated motion; 3) we make generation with different reference images, which demonstrates that the reference image affects the appearance of the generated video since the missing information should be predicted by the model. 

We welcome further discussion and are willing to answer any further questions.

Reviewer comment (Experiments):
The experiments are limited, especially comparison with end-to-end trainable models not provided. I suggest enriching the benchmarking with other existing models such as PC-AVS and PD-FGC as they are also end-to-end trainable models.

Author response:
>**Q: The experiments are limited, especially comparison with end-to-end trainable models not provided. I suggest enriching the benchmarking with other existing models such as PC-AVS and PD-FGC as they are also end-to-end trainable models.**

A: Thanks for your suggestion! Following your advice, we have evaluated PC-AVS [1] and PD-FGC [2] with their released models on the same test set. The results are listed in the table below. The visual comparison is provided in **[this video](https://gaiavatar.github.io/gaia/video/more/comparison_with_PC_PD.mp4)** and the full comparison is presented in **[this video](https://gaiavatar.github.io/gaia/video/more/1_speech-driven_add.mp4)**.

Table 4. Quantitative comparisons of speech-driven generation with previous end-to-end trainable models.

|                       |  Nat.$\uparrow$   |   Lip.$\uparrow$   |  Jit.$\uparrow$  |   Vis.$\uparrow$  |   Mot.$\uparrow$   |  Sync-D$\downarrow$   |  MSI$\uparrow$   |   FID$\downarrow$ |
| --------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| PC-AVS      |    2.797   |  3.843  |  3.546  |  3.452  |  2.091  |  8.341  |    0.677  |  59.464 |
| PD-FGC*     |    3.283   |  3.893  |  1.905  |  3.417   | 4.512 |   8.573  |    0.478 |   58.943 |
| GAIA        |    4.362   |  4.332  |  4.345  |  4.320  |  4.243  |  8.528   |   1.181  |  22.924 |

In this table, \* indicates that PD-FGC depends on extra driving videos to provide pose, expression, and eye motions. We use the real (ground-truth) video as its driving video (therefore the score of motion diversity is high). We also note that the Sync-D score for real video is 8.548, which is close to ours. Overall, our method beats the previous end-to-end trainable models in terms of overall naturalness (Nat.), lip-sync quality (Lip.), motion jittering (Jit.), visual quality (Vis.), and motion diversity (Mot.). We have added the comparison in Table 8 in the revised version.

[1] Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. CVPR 2021.

[2] Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis. CVPR 2023.

Reviewer comment (Presentation):
Although the writing quality is decent, it is hard to follow as it refers to other sections frequently and other issues (see Questions 1 and 2).

Author response:
>**Q: Although the writing quality is decent, it is hard to follow as it refers to other sections frequently and other issues (see Questions 1 and 2).**

A: Thanks for the suggestion! We have removed the unnecessary referrals to make the manuscript easy to follow. For example, we remove the referral in Section 5.3.1 (Ablation Studies on Scaling) and Section 5.5 (Discussion). In the revised version, apart from the referrals to the tables or figures, most referrals are referred to the Appendix for detailed illustrations.

Reviewer comment (Experiments):
In section 3, what does 'we collect High Definition Talking Face Dataset (HDTF) (Zhang et al., 2021) and Casual Conversation datasets v1&v2 (CC v1&v2) (Hazirbas et al., 2021; Porgali et al., 2023)' and 'we also collect a large-scale internal talking avatar dataset named AVENA' mean? Does it mean you collect those datasets you use their sample in your dataset or you use their samples in your training but they are not in your dataset? From the supplementary material, my understanding is you collect AVENA and use samples from other datasets (HDTF, CC v1, and v2) in the training of your model. Could you please elaborate and make it clear?

Author response:
>**Q: About the collection of HDTF, CC v1\&v2 and AVENA datasets.**

A: Thanks for your careful reading of our paper and supplementary materials, and sorry for the confusion. Your understanding is correct. AVENA is an internal dataset collected from our institution, and we utilize the union of AVENA, HDTF, CC v1 and v2 as the training set of our model. We have revised the description in Section 3 (Data Collection and Filtering) and Section 5.1 (Experimental Setups) for better understanding and marked it in blue.

Reviewer comment (Presentation):
Why do you have a discussion section at the end of Section 4 Model? I think it makes more sense in/after experiments. So, you can consider reorganizing the manuscript to have a better flow and complete discussion.

Author response:
>**Q: About the discussion section.**

A: Thanks for the suggestion! We have reorganized the flow in the revised manuscript and marked the revisions in blue. For example, we moved the improved discussion to the end of the experiments (Section 5.5).

Reviewer comment (Presentation):
Ethical consideration is left in Appendix F. However, for this study ethical consideration is important. So, you might consider putting it into the main manuscript to give necessary importance if possible.

Author response:
>**Q: About the location of Ethical consideration.**

A: Thanks for the suggestion, and we agree that ethical considerations are crucial for the completeness of the paper. We have revised the content of the manuscript and put the ethical considerations in Section 6 (Conclusion).

We hope our rebuttal and paper revision can address your concerns. We welcome further discussion and are willing to answer any further questions.

Reviewer comment (Experiments):
Missing evaluation of disentanglement between appearance and pose latent codes, i.e., cross-reenactment with the motion codes extracted from the image of a different identity.

Author response:
>**Q: Missing evaluation of disentanglement between appearance and pose latent codes, i.e., cross-reenactment.**

A: We have evaluated the cross-reenactment performance of GAIA and baselines in the right half of Table 2 of the manuscript, denoted as "Cross-Reconstruction" in the original version, where GAIA achieves significant improvements over baselines. Sorry for the confusion about the notations we utilized, we have changed it to "Cross-Reenactment" in this version to keep consistent with baselines.

Reviewer comment (Novelty):
Missing discussion of the related works, such as [1, 2], that explored the concept of pose-identity disentanglement for talking head synthesis before this work.

Author response:
>**Q: Missing discussion of some related works.**

A: Thank you for pointing out the related works! We make discussions as follows. We have also added the discussions to Section 2 (Related Works) and the revisions are marked in blue.

Latent Pose Descriptors [1] achieved pose-identity disentanglement, where the identity embedding is averaged across multiple frames and the pose embedding is obtained with augmented input. However, the model needs additional fine-tuning for the unseen identities while we can directly generalize to the unseen identities with only one reference portrait image.

MegaPortraits [2] adopted warping-based transformation, which is similar to face-vid2vid [3], but uses latent descriptors to represent expression instead of keypoints like face-vid2vid [3]. However, as we have validated in the subjective comparison and the experiments of the original manuscript, the usage of the heuristics hinders direct learning from data distribution, leading to unnatural results and limited diversity.

In addition, instead of supporting video-driven avatar generation only [1,2], GAIA contains both motion \& appearance disentanglement and speech-to-motion modules that support speech-driven, video-driven, pose-controllable, full-controllable and text-instructed avatar generation in a unified framework.

[1] Burkov et al., "Neural Head Reenactment with Latent Pose Descriptors", CVPR 2020

[2] Drobyshev et al., "MegaPortraits: One-shot Megapixel Neural Head Avatars", ACMMM 2022

[3] Wang et al., "One-shot free-view neural talking-head synthesis for video conferencing", CVPR 2021

Reviewer comment (Experiments):
As far as I can tell, the proposed method and the baselines were trained on different datasets. The resulting comparison evaluates the proposed framework _and_ the dataset at the same time. A comparison should include the experiments where base methods are trained on the same dataset, and the proposed method is trained on unprocessed datasets used in previous works.

Author response:
>**Q: The proposed method and the baselines were trained on different datasets.**

A: The reason we choose different datasets is that we find commonly utilized datasets such as VoxCeleb2 suffer a lot from the jittering content and thus have relatively low quality. However, we agree with the point that training on the same dataset provides fair comparisons. Therefore, we train our model on VoxCeleb2, the training dataset utilized by face-vid2vid, while keeping the test set unchanged. Due to the time limitation, we train a smaller model with 80M parameters and will train a larger one in the next. The results of video-driven self-reconstruction and cross-reenactment are listed as follows.

Table 1. Quantitative comparisons of self-reconstruction when trained on VoxCeleb2 dataset.

|                       |  FID$\downarrow$  | LPIPS$\downarrow$  | PSNR$\uparrow$ | AKD$\downarrow$   |  MSI$\uparrow$  |
| --------------------- | ----- | ----- | ----- | ----- | ----- |
| face-vid2vid    | 18.604 | 0.184  |  23.681 | 2.195  | 0.813 |
| GAIA-small w/ VoxCeleb2 | 16.099 | 0.173  | 22.896 | 1.434  | 1.083 |


Table 2. Quantitative comparisons of cross-reenactment when trained on VoxCeleb2 dataset.

|                       |  FID$\downarrow$ | AKD$\downarrow$   |  MSI$\uparrow$  |
| --------------------- | ----- | ----- | ----- |
| face-vid2vid    | 28.093 | 3.630  | 0.853 |
| GAIA-small w/ VoxCeleb2 | 27.643 | 2.968  | 1.035 |

From the results, we can find that when trained on the same dataset, GAIA still outperforms face-vid2vid on most metrics except PSNR (though it is the smallest model), demonstrating the power of the proposed method. We have added the results in Table 7 and more discussions in Section C.1 (More Video-driven Results) of the manuscript.