Paper: Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR-2024)

Reviewer comment (Experiments):
Experiment comparison setting: According to the supplementary video, CDM’s generation has considerably slow motion or low motion amplitude. However, LVDM and ModelScope can produce diverse and large motions. Therefore, the FVD comparison setting is not fair. We need other experiments to show CDM is better. For example, LVDM is trained on WebVid-10M with frame stride 8 (i.e. sample 16 keyframes in a 128-frame video clip), please re-train CDM to match the setting of LVDM.

Author response:
As mentioned in Appendix B, we followed the setup of VideoLDM [1] to show our method can generate videos with smooth motions. Nevertheless, we agree that adjusting FPS with LVDM and ModelScope will provide a more fair comparison, and we re-trained CMD with an adjusted frame sampling rate that you suggested and reported the FVD score in the Table below. As shown in this table, our method still shows a better FVD than baselines. 

| Method       | FVD   |
|--------------|-------|
| LVDM         | 455.5 |
| ModelScope   | 414.1 |
| VideoFactory | 292.4 |
| CMD (Ours)   | **238.3** |

[1] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, CVPR 2023

Reviewer comment (Evaluation):
Evaluation of autoencoder: Please conduct a quantitative evaluation of the performance of the CDM autoencoder. You can use a series of quantitative metrics. For example, SSIM, PSNR, etc. It would be great if you also showed how good stable-diffusion KLVAE can do on video reconstruction.

Author response:
Thanks for your constructive comments. Besides the quantitative evaluation of our autoencoder using reconstruction FVD in Table 4(b), we additionally evaluate our autoencoder with the metrics you suggested, SSIM and PSNR, and report them in the Table below. While our autoencoder (CMD) has a 9.14$\times$ smaller dimension in the latent space of encoding videos, it still shows reasonable reconstruction quality. 

| Method           | Dimension | R-FVD | SSIM  | PSNR  |
|------------------|-----------|-------|-------|-------|
| CMD              | 57344     | 56.80 | 0.863 | 28.71 |
| Stable diffusion | 524288    | 24.26 | 0.914 | 33.58 |

Reviewer comment (Experiments):
Training cost: To train CDM, one needs to train an autoencoder, a content diffusion, and a motion diffusion model. As a result, the training cost of CDM is very close to that of training ModelScope or LVDM. According to the supplementary material, training the autoencoder consumes 8 A100 for a week. After that, content diffusion and motion diffusion models can be trained. And they use 96 A100 for days to train on WebVid-10M. The training cost of CDM is not considerably lower than LVDM and ModelScope.

Author response:
There are two factors that make a direct comparison in terms of the total training GPU hours difficult. First, to our best knowledge, we could not find the exact total GPU hours for training LVDM and ModelScope. If the reviewer can provide a reference on that, we will be happy to perform a comparison and add the comparison and incorporate that in the manuscript. Second, the training cost of CMD in Appendix is measured with video clips of resolution 512$\times$1024 with a length 16, which is significantly higher-dimensional than the data preprocessed in LVDM and ModelScope (which is 256$\times$256 with a length 16). Hence, if the reported training cost of CMD is similar to (or slightly lower than) LVDM or ModelScope, it implies CMD can be trained much more efficiently than baselines. 

Given the above reasons, we instead measured the training cost of our method and baselines in the same normalized setup (i.e., batch size 1 and a single forward pass with the same resolution). As shown in Figure 5, CMD has much lower training cost than other baselines in our evaluation setting.

Reviewer comment (Evaluation):
Visual definition problem: CDM produces 512x1024 resolution videos. However, the supplementary is a 9M mp4 file, and is very hard to tell the visual definition of the samples. It would be great if the author provided evaluations on the visual definition of CDM.

Author response:
To address your concern, we visualized each video with a single text prompt and saved them to different video files (in total 28 videos). We attached them to the supplementary material.

Reviewer comment (Novelty):
In the abstract, the author says CDM is the first that can directly utilize a pretrained image diffusion model. But, to the best of my knowledge, Video LDM, LVDM, and ModelScope are also utilizing pretrained text-to-image models.

Author response:
As you mentioned, several works use pretrained text-to-image for text-to-video generation. However, they need to add some temporal layers inside the text-to-image models to deal with a sequence of image-wise latent vectors, which is both memory and computation inefficient. Our approach is different: CMD directly fine-tunes the text-to-image model without adding any additional parameters inside the text-to-image model, enabling efficient training and inference. Nevertheless, we also agree that the current abstract might mislead readers and thus we revised it accordingly.

Reviewer comment (Presentation):
Please number your equations. It will be very helpful for readers to understand your math formulas.

Author response:
Thanks for your constructive comments. We now added equation numbers in the revised manuscript.

Reviewer comment (Reproducibility):
The training time for the content frame diffusion generator is not shown in the supplementary material.

Author response:
Thank you for pointing this out. We now added training time for content frame diffusion in Appendix B.3.

Reviewer comment (Experiments):
Since the focus is on relatively short videos (16 frames), it would be interesting to establish whether CMD’s downstream performance is still favorable as the number of frames increases.

Author response:
**[W1] Performance of CMD when the number of frames increases?**

We fully agree that it would be an exciting future direction to extend our framework for longer video generation, and we strongly believe CMD has a strong potential for such a goal. First, we provided the performance of an autoencoder with a longer length of 24 in Table 4. It still shows a reasonable reconstruction performance, and thus, we think our CMD can be extended for longer video clips. For extremely long videos (e.g., a length of thousands), our method can be extended as well by following the strategy in PVDM [1] that divides a long video into multiple short video clips, encodes each video clip into succinct latents, and then perform clip-by-clip generation conditioned on a previous clip. 

[1] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

Reviewer comment (Evaluation):
The presented results can be even stronger if confidence intervals are provided.

Author response:
**[W2, Q1] Confidence interval of the method.**

Thank you for your constructive comments. We added a standard deviation of CMD in Table 1 in the revised manuscript. We also added a standard deviation of baselines if it is reported from references. For Table 2 and 3, it is difficult to add confidence intervals of baselines, because most baselines have not reported them in their quantitative evaluation. Nevertheless, we fully agree that at least providing confidence intervals of CMD in Table 2 and 3 will help future readers and research, and we will add them to the final manuscript. Please understand it was difficult to add in the revision due to the limited resources we have.

Reviewer comment (Evaluation):
Would it be feasible to include confidence intervals for results in Tables 1-3?

Author response:
**[W2, Q1] Confidence interval of the method.**

Thank you for your constructive comments. We added a standard deviation of CMD in Table 1 in the revised manuscript. We also added a standard deviation of baselines if it is reported from references. For Table 2 and 3, it is difficult to add confidence intervals of baselines, because most baselines have not reported them in their quantitative evaluation. Nevertheless, we fully agree that at least providing confidence intervals of CMD in Table 2 and 3 will help future readers and research, and we will add them to the final manuscript. Please understand it was difficult to add in the revision due to the limited resources we have.

Reviewer comment (Presentation):
In Figure 5, wouldn’t the comparison between different models’ computational & memory consumption be represented more accurately if there is a single bar for the CMD model with three stacked parts (autoencoder, content frame DM, motion latent DM)? For example, in the Memory panel, it would show that CMD has higher memory consumption than LVDM.

Author response:
**[Q3] Single bar for the CMD with three stacked parts for a fair efficiency comparison (Figure 5)?**

We did not report the stacked values for Figure 5, since training of three components do not need to be done simultaneously. Specifically, we do not need to allocate memory for three models at once since each model can be trained separately, so we just provided the component-wise memory and computation consumption. 

Nevertheless, we agree that for computation efficiency (FLOPs and sec/step) in training, stacking all components of CMD can help readers understand the computation efficiency of entire pipelines in handling a single video batch. Our method has 1.25 TFLOPs and 0.360 sec/step, which is still much better than 6.28 TFLOPs and 0.456 sec/step of LVDM.

Reviewer comment (Experiments):
Additionally, how were the baselines (ModelScope, LVDM) chosen in Figure 5?

Author response:
**[Q4] How were the baselines chosen in Figure 5?**

As mentioned in Section 4.3, we chose text-to-video diffusion models as baselines if their implementation and checkpoints are publicly available.

Reviewer comment (Reproducibility):
Are the authors planning to release an implementation of their proposed framework?

Author response:
**[Q5] Plan for releasing an implementation.**

We are planning to release the implementation.

Reviewer comment (Writing):
Abstract: “can directly utilizes a pretrained” -> “can directly utilize a pretrained”.

Author response:
**[C1] Editorial comment.**

Thanks for your careful reading. We fixed the typo in the revised manuscript.

Reviewer comment (Experiments):
Constrained by computational resources, the auto-encoder can only handle a limited number of frames at one time, which means that the auto-encoder can only encode motions within a limited interval. The reduced computational cost does not help us to explore the generation of longer videos.

Author response:
As you mentioned, there exists a limit length to an autoencoder that can encode videos as succinct latent vectors at once. Nevertheless, we believe that it does help in exploring generations of longer videos. Specifically, for longer videos, one can consider dividing it into multiple video clips of a fixed length, encode each clip as latent vectors using our autoencoder, and then consider modeling these multiple latent vectors. Such a strategy has also been widely adopted in latent video generative modeling literature [1,2]. While we did not try this due to the lack of resources, we strongly believe our method can be similarly extended for longer videos.

[1] Villegas et al., Phenaki: Variable Length Video Generation From Open Domain Textual Description, ICLR 2022.  
[2] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

Reviewer comment (Theory):
The content code is represented by the weighted sum of original frames, which may limit the representing ability of the content code. For example, the key components of a video may appear in different frames, (e.g. one person in frame 1 and another person in frame 2), in which case, it is difficult to maintain the information all key components in the content code.

Author response:
As you stated, content frames alone as a weighted sum of the original frame might limit representation capability. However, at the same time, this also constrains the space of content code to easily exploit pretrained image diffusion models. Also, recall that we also included "motion latents" in the autoencoder design, which can compensate for the representing ability of content frames. Because of this, our encoding scheme with this content code design can encode video with a high quality even if the scene change is dramatic, as empirically shown in Table 4a and 4b. Exploring a better design of content code and motion latent vectors should be an interesting future work.

Reviewer comment (Novelty):
The idea of decomposing the video generation into two factors has been proposed in previous methods [a, b, c].

[a] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

[b] Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, Ziwei Liu. Text2Performer: Text-Driven Human Video Generation

[c] Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, Martin Renqiang Min. Conditional Image-to-Video Generation with Latent Flow Diffusion Models.

Author response:
**[W1] Idea of decomposing video generation into two factors has been proposed in previous methods**

We agree that previous works have shared a similar idea of decomposing videos into two factors. However, the key difference between CMD and them is in the purpose of decomposition. In this work, we explored a new video decomposition that (a) exploits pretrained image diffusion models for video generation (b) in computation- and memory-efficient manner. To achieve this, we introduced an encoding scheme in which one of the factors (content frame) lies in the image space (for (a)), and the remaining part (motion latent vectors) becomes a low-dimensional vector, and thus we train an additional lightweight diffusion model to predict these motion latents vectors (for (b)).

Unlike our method, existing approaches have not considered (a) and (b) simultaneously. Specifically, [1] exploits pretrained image diffusion models, but it deals with videos as high-dimensional cubic tensors directly. Moreover, [2, 3] do not use pretrained image diffusion models, and [3] uses flow for motion encoding, which is a high-dimensional cubic tensor as well. Nevertheless, we think adding a discussion with relevant work [1, 2, 3] that you mentioned will give better insight to future readers, and we added a related discussion in Appendix A in the revised manuscript.  

[1] Guo et al.,. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning  
[2] Jiang et al., Text2Performer: Text-Driven Human Video Generation  
[3] Ni et al., Conditional Image-to-Video Generation with Latent Flow Diffusion Models.

Reviewer comment (Experiments):
In the training of autoencoder, how to ensure the model learns meaningful motion representation, rather than directly copying the information containing content information from the inputs?

Author response:
**[W2] How to ensure the model learns meaningful motion representation?**

This is a good point. In our video autoencoder, the content frame is a static image without the time dimension, so without meaningful motion representation learned by the motion latent, it cannot reconstruct the video. In other words, it is the video reconstruction and content-motion decomposition in our video autoencoder that ensures the motion latent captures the meaningful motion representation. Empirically, we also found our motion latent vectors indeed contain meaningful motion information. For instance, when generating time-lapse videos of the sky, our model can generate videos with diverse cloud-moving directions at a given fixed content frame. We included this result in Appendix F in the revised manuscript.

---

**[W2]** We use an extremely low-dimensional motion vector as the constraint. For instance, in our video experiments to encode 512$\times$1024 resolution RGB frames with a length of 16, the dimensionality of the motion latent vector is 24,576, 0.1% of the original dimensionality). Such a low dimensionality gives a strong bottleneck to encode the entire video without using content frames; hence, the autoencoder should utilize rich spatial information included in the content frame to achieve high-quality encoding. Also, in Appendix F, we have shown that fixing the content frame maintains the overall contents of the generated videos even if it is combined with different motion latent vectors. This result also empirically verifies that our autoencoder does use the information in the content frame, not just solely relying on the motion latent vector to encode the videos.

Reviewer comment (Evaluation):
The content frame is an average of the latent representations of multiple frames, and this would lead to the generated frames being blurry, especially for the first frame. From the results of Fig. 10 and Fig. 11, the first frames are kindly of blurry.

Author response:
**[W3] Generated frames being blurry due to the design of the content frame**

Thanks for pointing this out. In principle, since our design includes not only a content frame but also a “motion latent vector”, the learned motion latent will compensate for the representing power of the content frame, and reconstruct each frame with a high quality. This has been empirically supported by the high-quality encoding performance in Table 4a and 4b. In practice, we think the appearance of blurriness in generated videos may be attributed to a relatively not large dataset and a relatively small model size. Hence, we believe visual quality can be much better if we use a larger text-video dataset, as validated in the concurrent work [4]. Moreover, recall that our model includes a “single” diffusion model that has a relatively small model size (1.7B including the autoencoder) in total without any cascaded modules, unlike the existing video diffusion models; we also believe training larger models will greatly improve the frame quality. 

[4] Wang et al., VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation, arXiv 2023

---

**[W3]** To further address your concern, we provide some qualitative results in Appendix H and the supplementary material (left are real videos, right are reconstructions) to support our hypothesis further. The results demonstrate the high quality reconstructions from our autoencoder without blurry artifacts. Moreover, we think low reconstruction FVD (lower is better) that we provided in Table 4a and 4b implies high-quality video encoding hardly suffer from blurry artifact, because with such artifact, it is unlikely to achieve FVDs at our level asd due to the high distribution shift. Note that this metric has been widely used in previous latent video generation works [1, 2] to measure the reconstruction quality of videos. 

[1] Yu et al., MAGVIT: Masked Generative Video Transformer, CVPR 2023.  
[2] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.