Paper: VersVideo: Leveraging Enhanced Temporal Diffusion Models for Versatile Video Generation (ICLR-2024)

Reviewer comment (Novelty):
The proposed method directly combines existing techniques, such as excitation network and MoE, to enhance spatiotemporal feature representation, which makes the novelty a bit weak. The proposed Multi-Excitation Convolution applies excitation convolution multiple times. Original excitation convolution should be cited.

Author response:
>Q1: The proposed method directly combines existing techniques, such as excitation network, MoE to enhance spatiotemporal feature representation, which makes the novelty a bit weak. The proposed Multi-Excitation Convolution applies excitation convolution multiple times. Original excitation convolution should be cited.

Thank you for your expertise and the time you dedicated to reviewing our paper.


In this paper, we tackle three major challenges related to diffusion models for video generation. 
- Firstly, we focus on the denoising UNet, which is the core component responsible for modeling spatial-temporal dynamics. 
- Secondly, we address the VAE's limitations, which lead to flicker artifacts and inter-frame inconsistency caused by information loss during video encoding into latents and reconstruction into pixel space. 
- Finally, we aim to achieve controllable generation, allowing for video generation under a range of conditions.


Our contributions strive to address these challenges:
- We enhance the spatial-temporal capability of the denoising UNet with Multi-Excitation Convolution (MEC) and Mixture of Expert (MoE) spatial-temporal transformers, as shown in Table 1 of the paper.
- We build the TCM-VAE for video decoding to reduce information loss and increase inter-frame consistency, as demonstrated in Table 3 and visually showcased in [video generation demo](https://docs.google.com/document/d/e/2PACX-1vQyICnc6WH8CgCCRuX0avsOI7MoO4gyPRQEro-v1UGu0nI60CTCiDksLUXOV5y_A597X41rL5mQdyER/pub) and [video reconstruction demo](https://anonymous-pages.github.io/video_demos/). 
- We propose a unified controllable video generation strategy for text, visual, image, and style conditions, as demonstrated in Fig. 6 and visually showcased in the [demo](https://anonymous-pages.github.io/video_demos/). 


**Multi-Excitatin Convolution (MEC) motivation.** In contrast to existing video diffusion models, which often restrict spatial-temporal performance due to the over-simplification of standard 3D operations, we propose multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes. 

The proposed MEC is constructed by assembling four paths in parallel, allowing for the **activation of multi-type information in videos**. Our design is fundamentally inspired by the squeeze-and-excitation (SE) block [ref-1; ref-2], as it explicitly models channel/temporal interdependencies. Drawing inspiration from these two previous works, we design the excitation paths:
1. Factorized 2D spatial + 1D temporal convolution. This path is a **simplified adaptation of 3D convolution** and is the only temporal module widely used in most existing video diffusion models [ref-3; ref-4; ref-5]. 
2. Spatial-temporal excitation. This path pools the 5-D tensor $(B, C, T, H, W)$ along channel dimension, resulting in the **spatial-temporal activation** mask $(B, 1, T, H, W)$ which is later processed with 3D convolution. 
3. Channel excitation. This path get global spatial information $(B, C, T, 1, 1)$ of the input feature $(B, C, T, H, W)$ by spatial average pooling.
4. Motion excitation. The **motion information** is modeled by adjacent frames using the same squeeze and unsqueeze strategy. Related works have explored this, aiming to model motion information based on the feature level instead of the pixel level [ref-2; ref-6].



*Reference*:

[ref-1] Hu, J., Shen, L. and Sun, G., 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).

[ref-2] Li, Y., Ji, B., Shi, X., Zhang, J., Kang, B. and Wang, L., 2020. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 909-918).

[ref-3] Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D. and Zhou, J., 2023. VideoComposer: Compositional Video Synthesis with Motion Controllability. arXiv preprint arXiv:2306.02018.

[ref-4] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S. and Kreis, K., 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22563-22575).

[ref-5] Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D. and Dai, B., 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725.

[ref-6] Jiang, B., Wang, M., Gan, W., Wu, W. and Yan, J., 2019. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2000-2009).

Reviewer comment (Evaluation):
From Table 1, the proposed method fails to achieve better performance than Make-A-Video. Even compared with the methods trained on WebVid-10M, the performance improvement is marginal.

Author response:
>Q2: From Table 1, the proposed fails to achieve better performance than Make-A-Video. Even compared with the methods trained on WebVid-10M, the performance improvement is marginal.

We appreciate your thorough review and the effort you put into providing detailed feedback.

In Table 1, we employ the Frechet Video Distance (FVD) and CLIP similarity (CLIPSIM) as evaluation metrics for text-to-video generation. These two metrics are commonly utilized in most of video generation studies. The FVD metric assesses the similarity between real and generated videos, taking into account both video quality and motion dynamics. On the other hand, CLIPSIM measures the video quality and text alignment by averaging the CLIP similarity score of all generated frames.


Our proposed *VersVideo-H* diffusion model, trained with WebVid-10M, consists of 2B parameters. In contrast, **Make-A-Video serves as a strong competing method with significantly larger parameters and training data.** It encompasses 9.72B parameters in total, including a 3.1B text-to-video diffusion model, a 1.3B prior model, a 3.1B frame interpolation model, and more. Make-A-Video is trained using 2.3B images, WebVid-10M videos, and a 10M video subset from HD-VILA-100M. As a result, Make-A-Video achieves the highest CLIPSIM score of $0.3049$, while our proposed VersVideo-H model achieves a slightly lower score of $0.3014$. 

Moreover, it's worth noting that Make-A-Video did not provide results for the FVD metric for motion dynamic evaluation.


When comparing our proposed VersVideo model with the most recent models trained using WebVid-10M, it outperforms them all. For instance, Videocompoer (1.85B) and Videofusion (1.83B) have comparable model sizes, but their performance falls short with FVD scores of $580$ and $581$, and CLIPSIM scores of $0.2932$ and $0.2795$, respectively. In contrast, our proposed model achieves superior results with an FVD score of $421$ and a CLIPSIM score of $0.3014$. The second-best model that comes close to ours is SimDA that is an up-to-date concurrent work within two months of our submission. SimDA achieves an FVD score of $456$ and a CLIPSIM score of $0.2945$. These results highlight the performance of our VersVideo model over other recent video generation models.

Reviewer comment (Experiments):
From Table 2, the performance of the proposed method is a lot worse than the previous STOA method, e.g., VideoFusion. The author explains this by "the VersVideo-L model has a significantly smaller parameter of 500M". Why not use a larger model to verify the performance?

Author response:
>Q3: From Table 2, the performance of the proposed method is a lot worse than the previous STOA method, e.g., VideoFusion. The author explains this by "the VersVideo-L model has a significantly smaller parameter of 500M". Why not use a larger model to verify the performance? 

Thank you for your thorough review.


In this paper, we introduce two variants of *VersVideo*: *VersVideo-L* (500M) and *VersVideo-H* (2B). The primary purpose of *VersVideo-L* is for fast prototyping and conducting ablation studies.  *VersVideo-H* is trained for text-to-video generation and controllable generation tasks. If we were to use *VersVideo-H* for prototyping and ablation studies, it would require training several models with billions of parameters from scratch. This would result in an exceptionally high cost for validating model design and optimizing the network. Therefore, we choose to conduct the ablation study using *VersVideo-L* to validate our model design, and subsequently scale up the model to 2B parameters.

Reviewer comment (Experiments):
Make-A-Video also reports performance on the UCF dataset, and the results should be included.

Author response:
>Q4: Moreover, 1) Make-A-Video also reports performance on the UCF dataset, the results should be included; 2) the author should also provide performance under the same resolution as previous methods for better comparison.

Thank you for your suggestions. 



We update Table 2 in the paper and include the performance of Make-a-video. Both the MEC and MoE-STA have demonstrated their effectiveness as design elements in improving the performance of the video diffusion model. These elements consistently contribute to performance enhancements. Although the *VersVideo-L* model does not surpass state-of-the-art models like Make-a-Video and VideoFusion, which boast billions of parameters, it is worth noting that the *VersVideo-L* model has a significantly smaller parameter count of 500 M. Notably, the *VersVideo-H* model achieves an IS score of $81.3$ and an FVD score of $119$, making it the second-best performing model after Make-A-Video.

| Method | Resolution | $\mathrm{IS} \uparrow$ | $\mathrm{FVD} \downarrow$ |
| :---: | :---: | :---: | :---: |
| TGAN (Saito et al, 2016) | $64 \times 64$ | 15.83 | - |
| MoCoGAN-HD (Tulyakov et al., 2017) | $128 \times 128$ | 12.42 |  |
| CogVideo (Hong et al., 2022) | $160 \times 160$ | 50.46 | 626 |
| DVD-GAN (Clark et al., 2019) | $128 \times 128$ | 32.97 | - |
| TATS (Ge et al., 2022) | $128 \times 128$ | 79.28 | 332 |
| VideoFusion (Luo et al., 2023) | $128 \times 128$ | 80.03 | 173 |
| **Make-A-Video (Singer et al, 2022)** | $256 \times 256$ | **82.55** | **81.25** |
| VersVideo-L (baseline) | $256 \times 256$ | 60.2 | 355 |
| VersVideo-L (+ MEC) | $256 \times 256$ | 68.8 | 287 |
| VersVideo-L (+ MoE-STA) | $256 \times 256$ | 63.5 | 302 |
| VersVideo-L (+ MEC + MoE) | $256 \times 256$ | 72.9 | 207 |
| **VersVideo-H** | $576 \times 320$ | **81.3** | **119** |


In terms of resolution, it should be noted that FVD features are extracted with an I3D model trained on Kinetics-400 and the Inception Score (IS) calculated with a C3D model [ref-1]. During metric computation, the C3D model resizes all videos to a spatial resolution of $112\times112$, as determined by its internal structure.  Consequently, we believe that there would be no substantial difference in the feature level between resolutions of $256\times256$ and $128\times128$. If you would like to explore additional evaluation issues concerning FVD and IS, please refer to the Section C in the Appendix of StyleGAN-V [ref-2].


*Reference*:
[ref-1] Tran, D., Bourdev, L., Fergus, R., Torresani, L. and Paluri, M., 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 4489-4497).
[ref-2] Skorokhodov, I., Tulyakov, S. and Elhoseiny, M., 2022. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3626-3636).

Reviewer comment (Experiments):
The author should also provide performance under the same resolution as previous methods for better comparison.

Author response:
>Q4: Moreover, 1) Make-A-Video also reports performance on the UCF dataset, the results should be included; 2) the author should also provide performance under the same resolution as previous methods for better comparison.

Thank you for your suggestions. 



We update Table 2 in the paper and include the performance of Make-a-video. Both the MEC and MoE-STA have demonstrated their effectiveness as design elements in improving the performance of the video diffusion model. These elements consistently contribute to performance enhancements. Although the *VersVideo-L* model does not surpass state-of-the-art models like Make-a-Video and VideoFusion, which boast billions of parameters, it is worth noting that the *VersVideo-L* model has a significantly smaller parameter count of 500 M. Notably, the *VersVideo-H* model achieves an IS score of $81.3$ and an FVD score of $119$, making it the second-best performing model after Make-A-Video.

| Method | Resolution | $\mathrm{IS} \uparrow$ | $\mathrm{FVD} \downarrow$ |
| :---: | :---: | :---: | :---: |
| TGAN (Saito et al, 2016) | $64 \times 64$ | 15.83 | - |
| MoCoGAN-HD (Tulyakov et al., 2017) | $128 \times 128$ | 12.42 |  |
| CogVideo (Hong et al., 2022) | $160 \times 160$ | 50.46 | 626 |
| DVD-GAN (Clark et al., 2019) | $128 \times 128$ | 32.97 | - |
| TATS (Ge et al., 2022) | $128 \times 128$ | 79.28 | 332 |
| VideoFusion (Luo et al., 2023) | $128 \times 128$ | 80.03 | 173 |
| **Make-A-Video (Singer et al, 2022)** | $256 \times 256$ | **82.55** | **81.25** |
| VersVideo-L (baseline) | $256 \times 256$ | 60.2 | 355 |
| VersVideo-L (+ MEC) | $256 \times 256$ | 68.8 | 287 |
| VersVideo-L (+ MoE-STA) | $256 \times 256$ | 63.5 | 302 |
| VersVideo-L (+ MEC + MoE) | $256 \times 256$ | 72.9 | 207 |
| **VersVideo-H** | $576 \times 320$ | **81.3** | **119** |


In terms of resolution, it should be noted that FVD features are extracted with an I3D model trained on Kinetics-400 and the Inception Score (IS) calculated with a C3D model [ref-1]. During metric computation, the C3D model resizes all videos to a spatial resolution of $112\times112$, as determined by its internal structure.  Consequently, we believe that there would be no substantial difference in the feature level between resolutions of $256\times256$ and $128\times128$. If you would like to explore additional evaluation issues concerning FVD and IS, please refer to the Section C in the Appendix of StyleGAN-V [ref-2].


*Reference*:
[ref-1] Tran, D., Bourdev, L., Fergus, R., Torresani, L. and Paluri, M., 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 4489-4497).
[ref-2] Skorokhodov, I., Tulyakov, S. and Elhoseiny, M., 2022. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3626-3636).

Reviewer comment (Experiments):
Results in Table 2 and Table 3 are from different training datasets. Better to keep it consistent for better comparison.

Author response:
>Q5: Results in Table 2 and Table 3 are from different training datasets. Better to keep it consistent for better comparison.

Thank you for your thoughtful comments. Table 2 and Table 3 represent two distinct experiments that utilize different training datasets for specific reasons.

Table 2 aims to evaluate the effectiveness of the **denoising UNet** design, which incorporates multi-excitation convolution (MEC) and Mixture of Experts (MoE) attention. To conduct this experiment, we utilize **text-video pairs** sourced from WebVid-10M. This dataset offers high-quality textual descriptions for videos and encompasses a diverse range of content.

On the other hand, Table 3 aims to train the **TCM-VAE** for video reconstruction using only video data.   Unlike Table 2, which requires text-video pairs, video captions are not needed to train the TCM-VAE. We did not use WebVid-10M because its videos contain watermarks. Therefore, we obtained 100,000 high-quality videos from the HD-100M dataset for this experiment.

Reviewer comment (Experiments):
Leveraging pre-trained models could limit the model's versatility and the ability to generalize across different datasets. Is it possible to validate the proposed designs to other SD models?

Author response:
Thank you for your review and valuable suggestions. 

Incorporating temporal structures for video generation is a widely adopted approach to enhance existing text-to-image models, such as Stable Diffusion (SD). Several methods have been employed in this regard, including training the entire network [ref-1; ref-2; ref-3] or freezing the pretrained image network to ensure compatibility with other SD models [ref-4; ref-5; ref-6].


Our designs can be utilized for other video diffusion models that are based on SD in three different ways:
1. The over-simplified temporal resblock/transformer can be replaced with the proposed MEC and MoE transformer of the denoising UNet.
2.  The image VAE can be replaced by our proposed TCM-VAE. For example, we can enhance the image VAE decoder of *VideoFusion* and *VideoComposer* by using our TCM-VAE decoder, without additional training. This module significantly reduces flicker artifacts, as shown in our [demo](https://docs.google.com/document/d/e/2PACX-1vQyICnc6WH8CgCCRuX0avsOI7MoO4gyPRQEro-v1UGu0nI60CTCiDksLUXOV5y_A597X41rL5mQdyER/pub).
3.  Designing an unified control strategy outlined in our paper, rather than developing individual motion structure conditions with multiple individual ControlNets for video generation.


Reference:

[ref-1] Esser, P., Chiu, J., Atighehchian, P., Granskog, J. and Germanidis, A., 2023. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 7346-7356).

[ref-2] Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X. and Shou, M.Z., 2023. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 7623-7633).

[ref-3] Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y. and Feng, J., 2022. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018.

[ref-4] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S. and Shi, H., 2023. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439.

[ref-5] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S. and Kreis, K., 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22563-22575).

[ref-6] Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D. and Dai, B., 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725.

Reviewer comment (Presentation):
It might be beneficial to include a section discussing the limitations of your current approach and possible future work to address these limitations.

Author response:
Thank you for your insightful comments and suggestions.

Despite its strong generative capabilities and versatile guidance inputs, the proposed *VersVideo* has some limitations in its current state:
1. The generated videos sometimes contain watermarks due to the training dataset WebVid-10M. To ensure high-quality generation, it is necessary to have watermark-free videos with textual descriptions.
2. Our primary focus is on generating videos of a fixed length (e.g., $L = 24$). However, an interesting future direction would be to extend our method for long video synthesis, considering clip-by-clip generation.
3. *VersVideo* inherits certain common challenges from image diffusion, including hand generation and text alignment. Specific solutions employed in image generation could serve as inspiration for video generation.
4. The stability of image-to-video generation is not perfect. The input image cannot be consistently maintained throughout the generated video. It requires additional control mechanisms to preserve the spatial appearance of the input image more accurately.

[A snapshot of our revision.](https://drive.google.com/file/d/1f1UdXPXBKIKyhMWgVA8SiF3w7FOx4Nt1/view?usp=sharing)

Reviewer comment (Reproducibility):
If possible, consider releasing the code or at least providing more detailed implementation notes. This could help others replicate your results and build upon your work.

Author response:
Thank you for taking the time to review our paper.

We provide details (e.g., hyperparameter, model, and optimizer) and experiment setups (e.g., datasets, metrics) in Section 3.2 and Appendix A, B, and C. We incorporated additional details regarding the key modules to facilitate implementation and enhance understanding.
1. MoE spatial-temporal transformer.  In our paper, we introduce a new MoE scheme that trains experts without a router network, reducing the number of parameters introduced by the router and simplifying training. Our MoE assigns tokens to experts by randomly partitioning them into groups. At the end of each training iteration, we perform weight averaging on each MoE's experts. After training, we can simply average the experts of each MoE into a single FFN, without increasing inference cost.  Merging weights of models for better performance is widely acknowledged in large language models and image classification. This simple yet effective idea has been proven to improve accuracy and robustness. Similarly, our proposed MoE merges experts' weights to efficiently enhance the performance of the spatial-temporal transformer. Unlike conventional model ensembles, we average the model weights without incurring any additional inference or memory costs. [A snapshot of our revision](https://drive.google.com/file/d/1WXwEmOOBMi4Mr7dxF2U65-viJ1lf2A5J/view?usp=sharing).
2. TCM-VAE. We introduce Temporal Compensation Modules (TCMs) to enhance temporal consistency, and a refinement UNet for quality enhancement, building upon an image pretrained VAE. To optimize training efficiency, we freeze the encoder and concentrate on training the video decoder. We incorporate TCMs into the decoder blocks to infuse temporal information into multi-level intermediate features. The implementation of TCMs leverages the multi-excitation module in Section 2.2.  More details about the TCM-VAE can be found in **Section2.3, page 5** and **Section C of the Appendix**. [A snapshot of our revision.](https://drive.google.com/file/d/1r6jnbaAk_OLFXTVXUyc1ogPlxD7ZxrrB/view?usp=sharing)

Reviewer comment (Writing):
The details of the MoE attention are not explained very well. In section 2.2, the authors mentioned that they use MoE design to enhance attention. If my understanding is correct, the attention is performed on all channels, but the feedforward network after attention is applied to different groups. If the MoE attention is applied like this, how can the feature or attention be enhanced?

Author response:
Thank you for highlighting this issue; we apologize for any confusion caused. 

Our design might be more appropriate to refer to it as the 'MoE Spatial-Temporal Transformer' rather than 'MoE Spatial-Temporal Attention.' 

The role of the FFN in transformers is to process the information aggregated by the attention mechanism. It can learn to recognize and generate more intricate patterns based on the information it receives from the attention block. 

The most recent prevalent training approach for transformers involves replacing the FFN layer with a sparse MoE [ref-1; ref-2; ref-3]. The sparse MoE includes multiple expert FFNs, each with unique weights, and a trainable routing network. During both the training and inference phases, this routing network selects a sparse set of experts for each input, enabling efficient scalability of transformer models through sparse computation.

In our paper, we introduce a new MoE scheme that trains experts without a router network, reducing the number of parameters introduced by the router and simplifying training. Our MoE assigns tokens to experts by randomly partitioning them into groups. At the end of each training iteration, we perform weight averaging on each MoE's experts. After training, we can simply average the experts of each MoE into a single FFN, without increasing inference cost. 

**Why does this MoE work?** Merging weights of models for better performance is widely acknowledged in large language models [ref-6] and image classification [ref-4; ref-5]. This simple yet effective idea has been proven to improve accuracy and robustness. Similarly, our proposed MoE merges experts' weights to efficiently enhance the performance of the spatial-temporal transformer. Unlike conventional model ensembles, we average the model weights without incurring any additional inference or memory costs.



Please kindly refer to **Section B in the Appendix** where we have added more details and descriptions of the MoE design. [A snapshot of our revision](https://drive.google.com/file/d/1WXwEmOOBMi4Mr7dxF2U65-viJ1lf2A5J/view?usp=sharing).

_Reference_:

[ref-1] Du, N., Huang, Y., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A.W., Firat, O. and Zoph, B., 2022, June. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (pp. 5547-5569). PMLR.

[ref-2] Fedus, W., Zoph, B. and Shazeer, N., 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1), pp.5232-5270.

[ref-3] Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N. and Chen, Z., 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668.

[ref-4] Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S. and Schmidt, L., 2022, June. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (pp. 23965-23998). PMLR.

[ref-5] Ainsworth, S.K., Hayase, J. and Srinivasa, S., Git re-basin: Merging models modulo permutation symmetries, 2022. URL https://arxiv. org/abs/2209.04836.

[ref-6] Jin, X., Ren, X., Preotiuc-Pietro, D. and Cheng, P., 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849.

Reviewer comment (Writing):
Several details are missing for the temporal compensate decoder. In the paper, the authors do not mention what S1 stands for.

Author response:
Thank you for pointing out this issue. We sincerely apologize for such confusions.

In Fig. 3, $\{A_1, A_2,...,A_M\}$ are encoder blocks; $\{S_1, S_2, ..., S_M\}$ are decoder blocks. See [Fig. 3](https://drive.google.com/file/d/1-OtQhJ5zHNQGo4zTq-BQgAZxIwI9Az1x/view?usp=sharing) in the revised paper. 

Most existing video latent diffusion models utilize image VAEs to reconstruct videos from latents. However, an image VAE without temporal modules fails to capture the dependencies between frames. This leads to a loss of temporal information in videos, which in turn results in flickering artifacts and temporal inconsistency.

We introduce Temporal Compensation Modules (TCMs) to enhance temporal consistency, and a refinement UNet for quality enhancement, building upon an image pretrained VAE. To optimize training efficiency, we freeze the encoder and concentrate on training the video decoder. We incorporate TCMs into the decoder blocks to infuse temporal information into multi-level intermediate features. The implementation of TCMs leverages the multi-excitation module in Section 2.2. 

More details about the TCM-VAE can be found in **Section2.3, page 5** and **Section C of the Appendix**. [A snapshot of our revision.](https://drive.google.com/file/d/1r6jnbaAk_OLFXTVXUyc1ogPlxD7ZxrrB/view?usp=sharing)

The effectiveness of TCM-VAE in improving video quality is supported by quantitative metrics presented in **Table 3** of the paper, as well as visual examples available in the demo for video generation and reconstruction: [demo for video generation](https://docs.google.com/document/d/e/2PACX-1vQyICnc6WH8CgCCRuX0avsOI7MoO4gyPRQEro-v1UGu0nI60CTCiDksLUXOV5y_A597X41rL5mQdyER/pub) and [demo for video reconstruction](https://anonymous-pages.github.io/video_demos/).

Reviewer comment (Experiments):
For the Multi-excitation conv, the motivation for using four branches is not addressed clearly. The effectiveness of each branch is also not thoroughly ablated.

Author response:
We appreciate your thorough review and the effort you put into providing detailed feedback.


**Multi-Excitatin Convolution (MEC) motivation.** In contrast to existing video diffusion models, which often restrict spatial-temporal performance due to the over-simplification of standard 3D operations, we propose multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes. 

The proposed MEC is constructed by assembling four paths in parallel, allowing for the **activation of multi-type information in videos**. Our design is fundamentally inspired by the squeeze-and-excitation (SE) block [ref-1; ref-2], as it explicitly models channel/temporal interdependencies. Drawing inspiration from these two previous works, we design the excitation paths:
1. Factorized 2D spatial + 1D temporal convolution. This path is a **simplified adaptation of 3D convolution** and is the only temporal module widely used in most existing video diffusion models [ref-3; ref-4; ref-5]. 
2. Spatial-temporal excitation. This path pools the 5-D tensor $(B, C, T, H, W)$ along channel dimension, resulting in the **spatial-temporal activation** mask $(B, 1, T, H, W)$ which is later processed with 3D convolution. 
3. Channel excitation. This path get global spatial information $(B, C, T, 1, 1)$ of the input feature $(B, C, T, H, W)$ by spatial average pooling.
4. Motion excitation. The **motion information** is modeled by adjacent frames using the same squeeze and unsqueeze strategy. Related works have explored this, aiming to model motion information based on the feature level instead of the pixel level [ref-2; ref-6].



**Ablation study of MEC.** An ablation study is conducted to validate each excitation path of MEC, which includes the MoE-STA component. MEC includes factorized 2D spatial + 1D temporal convolution (2D+1D), spatial-temporal excitation (STE), channel excitation (CE), and motion excitation (ME). Various versions of the *VersVideo-L* model are trained with and without these paths, followed by an evaluation of their class-conditional generation using the UCF-101 dataset. The results suggest that the excitation modules improve the performance of MEC. When integrating all four excitation paths, the MEC demonstrates superior performance in both IS and FVD.

| 2D+1D | STE | CE | ME | IS $\uparrow$ | FVD $\downarrow$ |
| :---: | :---: | :---: | :---: | :---: | :---: |
|  | $\checkmark$ | $\checkmark$ | $\checkmark$ | 69.5 | 249 |
| $\checkmark$ |  | $\checkmark$ | $\checkmark$ | 68.0 | 264 |
| $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 67.7 | 266 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 68.3 | 260 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 72.9 | 207 |

Please refer to [**Section D in the Appendix**](https://drive.google.com/file/d/12SAET1FOi6bYzyCwwc3UDzT3qP0OBZHS/view?usp=sharing) for our revision.

*Reference*:

[ref-1] Hu, J., Shen, L. and Sun, G., 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).

[ref-2] Li, Y., Ji, B., Shi, X., Zhang, J., Kang, B. and Wang, L., 2020. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 909-918).

[ref-3] Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D. and Zhou, J., 2023. VideoComposer: Compositional Video Synthesis with Motion Controllability. arXiv preprint arXiv:2306.02018.

[ref-4] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S. and Kreis, K., 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22563-22575).

[ref-5] Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D. and Dai, B., 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725.

[ref-6] Jiang, B., Wang, M., Gan, W., Wu, W. and Yan, J., 2019. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2000-2009).

Reviewer comment (Experiments):
The authors did not provide a visual comparison of the results. Providing a visual comparison would better demonstrate the effectiveness of the method. Besides, a user study is also needed to verify the effectiveness of the method.

Author response:
Thank you for your valuable feedback. 

We incorporate more visual examples of competing methods for comparison in our demo, which can be found at [link](https://docs.google.com/document/u/1/d/e/2PACX-1vTpTofxtnoetQwgFcVaHWx01Offc2sc4T5zRwY6I-tueiECmpZKbnrYqRQ9pXTdXsbvJelg2Y6OaW4n/pub).  Additionally, we provide a comparison of conditioned generation at the end of [demo](https://anonymous-pages.github.io/video_demos/).

To conduct the user study, we selected four widely acknowledged and open-sourced models, namely Text2video-zero (Khachatryanet al., 2023), VideoFusion (Luo et al., 2023), Zeroscope (Hugging Face, 2023), and VideoComposer (Wang et al., 2023d), for visual comparison. The evaluation set consisted of 30 video prompts, and 16 evaluators compared the inter-frame consistency and overall video quality (including inter-frame consistency, alignment of text-video, and perceptual factors) between two videos - one from competing methods and one from our method, shown in a random sequence. Figure 11 shows that our method, *VersVideo*, was preferred over VideoFusion $83.3\%$ of the time in terms of video quality, and over VideoComposer $88.5\%$ of the time. Our method also performed significantly better than the baseline methods in terms of inter-frame consistency in the user study.

For detailed results, please refer to [**Section E in the Appendix**](https://drive.google.com/file/d/1N0yYBc__UTsc6sABkA3FPkN0VOINHWpP/view?usp=sharing).

Reviewer comment (Evaluation):
The quality of the video-to-video editing results currently shown in the paper is not much better compared to other methods such as Render-A-Video and TokenFlow. The results generated in the current demo have severe color changes. Therefore, I doubt the effectiveness of this method in video fidelity.

Author response:
Thank you for your valuable feedbacks.


In our demo, the first example of video-to-video editing showcased severe color changes due to the text prompt ***"cyberpunk city"* which is associated with the metaverse, characterized by rapid color changes and flashing lights.**

There are two concurrent studies, Render-A-Video and TokenFlow, for video editing. The Render-A-Video is a video-to-video model that employs multiple pipelines. It leverages the off-the-shelf image model from [CivitAI](https://civitai.com/), a public platform that enables artists to share their personalized models. In addition, TokenFlow utilizes pre-trained image models and the PnP image diffusion model [ref-1] for video editing.

It would be reasonable to expect that their rendering results are more promising in terms of fidelity. This is because the video model is based on pretrained image models that were trained using high-quality images. However, there is a significant difference between their approach and the proposed *VersVideo* method. While Render-A-Video and TokenFlow are specialized for video-to-video applications, our model is designed to be versatile, allowing for a wider range of applications such as text-to-video, image-to-video conversion, and stylization.


*Reference*:

[ref-1] Tumanyan, N., Geyer, M., Bagon, S. and Dekel, T., 2023. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1921-1930).

Reviewer comment (Evaluation):
Since many modules in the framework designed in the paper are related to temporal consistency, please provide indicators to measure temporal consistency. Currently, the paper only lists indicators of generation quality.

Author response:
Thank you for your thoughtful comments.

We use the widely accepted Frechet Video Distance (FVD) [ref-1] and CLIP Similarity (CLIPSIM) [ref-2] as evaluation metrics for text-to-video generation. CLIPSIM assesses video quality and text alignment by averaging the CLIP similarity score of all generated frames. The FVD metric evaluates the similarity between real and generated videos, considering **both video quality and motion dynamics**. Therefore, FVD can serve as an indicator of temporal consistency. As most prior studies report FVD and CLIPSIM as the gold standard, we follow this practice to ensure a fair comparison.


*Reference*:

[ref-1] Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M. and Gelly, S., 2018. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717.

[ref-2] Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G. and Duan, N., 2021. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806.

Reviewer comment (Experiments):
Can the current framework handle the situation with multiple condition inputs?

Author response:
Thank you for your insightful comments.  Our model can handle multiple condition inputs.

The global condition, such as the text prompt and style, is inherently compatible with other conditions. For instance, the final two rows in Fig. 6 utilize depth maps for motion structure control, while simultaneously employing images for global style control. 

When it comes to the composition of multiple motion structures, such as the concurrent use of depth, canny, and HED, we offer several examples to demonstrate its practicality with the [visual examples](https://docs.google.com/document/d/e/2PACX-1vQTN8K7LDRgAEAEniaQYD2ePbogQ__Hht7ewRqlzh0RNCfI13bIEPPzZwl67AK-gluB-UsYBagLd-R0/pub).