Paper: LEO: Generative Latent Image Animator for Human Video Synthesis (ICLR-2024)

Reviewer comment (Experiments):
The author only includes pickup methods for comparison, STOA methods are not included for comparison. Recent methods, such as MoStGAN-V, VDM, Video-LDM, VideoFactory, and Make-A-Video, should be included for comparison.

Author response:
**A1.** We have included listed works in Related Works in the revised manuscript.

For MostGAN-V,  we have updated Tab. 1 in revised manuscript and Supplementary Material website on both FaceForensics and CelebV datasets. Results show that our method outperforms MoStGAN-V both qualitatively and quantitatively.

For VDM, Video-LDM, VideoFactory, and Make-A-Video, **none of the listed methods is officially open-sourced, as well as reported any evaluation results on FaceForensics, CelebV and Taichi datasets.** Qualitativley and quantitatively comaprison with those methods would be difficult. 

We note that our work differs from listed methods in the tasks to solve. We focus on human-centric video generation, while the listed methods mainly target to solve general unconditional video or text-to-video generation problems.  Technically, the listed methods aim to generate very short video clips (e.g., 16 frames before interpolation) to align with text input. They do not propose any techniques for appearance and motion disentanglement, nor for long video generation, which, however, are the main targets in our work.  We think our proposed techniques could be a complementery component in general video generation system.

Reviewer comment (Experiments):
The author should include experiments on more challenging datasets, such as MSR-VTT and UCF101.

Author response:
**A2.** From our point of view, articulated human video generation is a very challenging task, even large-scale diffusion models can not handle it very well. It requires the model to simultaneously learn the human structure and motion in both spatial and temporal dimensions.

We have provided quantitative evaluation on UCF101 for unconditonal generation in Appendix A.2 Tab. 5 in revised manuscript. Results show that our method outperforms state-of-the-art w.r.t. FVD. However, from the results, we observe that it is still challenging for LEO in general video generation. We have addressed this limitation in the revised manuscript in Appendix A.4 and will leave it for our future work.

We argue that MSR-VTT (text-video dataset) and UCF101 (label-video dataset) are well-known for general conditional video generation, especially for label-/text-to-video generation. We think they are not proper datasets to evaluate the contributions of our method as LEO is designed for human-centric data rather than general video data. We focus on appearance and motion disentanglment, as well as long video generation. We have quantitatively and qualitatively evaluated LEO on three widely used human-centric datasets.

Reviewer comment (Evaluation):
Limitations and failure cases: these aspects are not presented in the papers and supplementary. Proper discussions on what LEO cannot do well can help the readers to better assess the contribution of the work, and also open up possible future directions.

Author response:
**Q2. What are the limitations of LEO? What are the failure cases? It would be great if the paper could show and discuss these topics.**  

**A2.** In the revised manuscript, we have addressed the limitations of our approach in Appendix A.4. Additionally, we have provided updated failure cases in the Supplementary Material website for further analysis. It is worth noting that, due to the autoregressive nature of our method for long video generation, there are instances where the model may inadvertently repeat the same action in videos of considerable length. To mitigate this issue, we propose a potential solution in the form of introducing random tiny perturbations to the motion code. By incorporating these subtle variations, we aim to enhance diversity and avoid repetitive patterns in the generated videos.

Reviewer comment (Experiments):
It would be nice to see some human-specific baselines, especially since the focus of the paper is on humans, e.g., utilizing skeleton/3DMM guidance.

Author response:
**A1.**  We compared our method with a 3DMM-based talking head generation method [1], as well as two diffusion-based conditional generation methods using skeleton as guidance [2, 3]. We showed several results on the Taichi and FaceForensics datasets in the updated Supplementary Material website.

Upon analysis, we observed that the results of the diffusion-based methods [2, 3] were significantly influenced by the quality of the extracted poses and skeletons. In contrast, our proposed method exhibited superior visual quality and temporal coherency. Furthermore, when compared to the 3DMM-based method [1], our approach achieved notable improvements in terms of facial details and head motion.

[1] Min et al., StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles, AAAI 2023

[2] Zhang et al, ControlVideo: Training-free Controllable Text-to-Video Generation, arXiv:2305.13077

[3] Khachatryan et al., Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, ICCV 2023

Reviewer comment (Experiments):
I believe a comparison (or at least discussion) to video-ldm [1] would be beneficial.

Author response:
**A2.** Video-LDM is a large-scale video generation framework, which extends a pretrained Stable Diffusion with temporal modules in both VAE and 2D UNet. Video features are first extracted by VAE-Encoder,  and then used in spatio-temporal UNet to learn the latent video distribution. In Video-LDM, appearance and motion are not disentangled, and spatio-temporal distribution is jointly learned in 3D UNet. In addition, Video-LDM is only able to generate 16-frame short video clips.

In our work, we pre-trained an image animator on human face and body. LIA is a flow-based image animator which enables motion transfer from driving video to target image. An important property of proposed framework is that it supports disentagnlment of motion and appearance. Appearance is represented by the input image, and motion is represented by the generated flow maps which are controlled by the motion code in latent space. Our framework is able to produce the same input image performing diverse motion.  Since motion is only a per-frame 1D vector, one major advantage of our work compared to Video-LDM is the low cost of autoressively generating long videos (e.g., ~1 min).

Reviewer comment (Presentation):
I am missing a section on the limitations and ethical considerations.

Author response:
**A3.** We have included the discussion of limitations in Appendix A.4, as well as the Section Ethics Statement in the revised manuscript.