Paper: VDT: General-purpose Video Diffusion Transformers via Mask Modeling (ICLR-2024)

Reviewer comment (Experiments):
While the authors conducted validations on various tasks, the ablation experiments in Table 2 are insufficient. Verification in unconditional video generation tasks would enhance the persuasiveness of the study.

Author response:
**A:** Thank you for the suggestion! We have conducted a detailed ablation study on UCF101 using DiT-S. The results show that reducing the Patchsize, increasing the number of Layers, and increasing the Hidden Size all further improve the model's performance. The position of Temporal and Spatial attention and the number of attention heads do not significantly affect the model's results. Further comprehensive analysis indicates that generally, **an increase in GFlops leads to better results, demonstrating the scalability of VDT**. When maintaining the same GFlops, some trade-offs in design are necessary, but overall, the model's performance does not differ significantly. And we have included these experiments in the Appendix A of our paper.

| Patch Size, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 4    |   1.9   | 643.9      |
| 2    |   7.7   | 554.8      |
| 1    |   30.9   | **466.2**      |


| Head, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 3      |   7.7      | 559.8      |
| 6      |   7.7      | **554.8**      |
| 12     |   7.7       | 598.7      |

| Depth, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 6       |   3.9     | 580.7      |
| 12      |   7.7      | 554.8      |
| 18      |   11.6     | **500.6**      |


| Hidden Size, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 192     |   1.9         | 704.2    |
| 384     |   7.7         | 554.8      |
| 768     |   30.9         | **464.2**      |


| Architecture, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| Spatial First   |   7.7         | 550.2      |
| Temporal First   |   7.7         | 554.8      |

Reviewer comment (Experiments):
The method is limited to lower resolutions, diminishing its practical applicability.

Author response:
**A:** Thank you for your question regarding the resolution constraints of our patch-based method. We understand your curiosity, and the limitations are indeed primarily due to data availability and computational resources:

1. **Data Availability:** The development of high-resolution models (e.g., Align Your Latents) often require access to large and proprietary datasets, which are not publicly available now. Unfortunately, there is a notable scarcity of high-quality, high-resolution video datasets in the public domain. This lack of available data poses a significant challenge in training and testing our method at higher resolutions.

2. **Computational Constraints:** Addressing the computational aspect, for a video with a resolution of 16x256x256, post VAE encoding, the token length reaches approximately 16,384 (16x32x32). If we were to scale up to a resolution of 16x512x512, the token length would increase to around 65,536 (16x64x64). While our computational resources might be capable of handling this increased scale, it would considerably impact our Batchsize. This factor is crucial for the efficiency and practicality of diffusion models. It's noteworthy that even recent high-resolution works, such as 'Imagen Video' and 'Align Your Latents', still rely heavily on super-resolution (SR) models to manage the computational demands associated with higher resolutions.

These factors combine to create a practical ceiling on the resolution we can effectively work with, given current public data availability and our computational resources. While we acknowledge this limitation, we believe our approach still offers valuable insights and applications within its operational resolution range.

Reviewer comment (Novelty):
The core idea of this paper is simple and reasonable. Replacing the original CNN with transformer is an effective method that has been widely verified in the whole field. In addition, this paper adopts the basic attention mechanism, and the mask strategy proposed is also commonly used in video tasks. So, from this perspective, the contribution is slightly less obvious.

Author response:
**A:** Thank you for your comments on the core idea of our paper. We appreciate your perspective regarding the simplicity and effectiveness of our VDT design. 

*  We concur that our Video Diffusion Transformer (VDT) design, while simple, is influenced by established methodologies in the field. Nevertheless, we firmly believe that the essence of innovation often lies in simplicity and effectiveness. Our aim with VDT was not to overcomplicate the design merely for the appearance of novelty, but rather to focus on functional efficiency and adaptability. VDT marks a pioneering effort in applying transformer structures specifically for video diffusion tasks, which, despite its conceptual simplicity, represents a significant and non-trivial leap in this domain.

*  Meanwhile, one of the key strengths of VDT is its ability to unify a broad spectrum of video generation tasks within a single, scalable model, without the need for specialized or task-specific designs. This aspect of VDT, which goes beyond the complexity of its structure, is one import contribution - offering a versatile and robust framework for general purpose video generation tasks.

*  Furthermore, VDT's token-concat approach for conditioning can be seen as  coarse-grained training strategy for autoregressive models. Given the proven success of autoregressive approaches in other domains, VDT's methodology could potentially spearhead significant advancements in large-scale computer vision models.

We agree that the structural design of VDT might not appear novel in isolation. However, the application of this concept in the realm of video diffusion is innovative. The value of our work lies not only in its design but also in its application, scalability, and the potential it offers for video generation using transformer architectures.

Reviewer comment (Experiments):
The work in this paper can be a good benchmark in this field. However, this paper only verifies that it can be effective on multiple video tasks. As a good benchmark, this paper should give more detailed experimental analysis of ablation. For example, why does space-time pay attention to time before space? In this paper, the influence of each module on the video generation effect, the comparison experiment of hyperparameter and so on.

Author response:
**A:** Thank you for the suggestion! We have conducted a detailed ablation study on UCF101 using DiT-S. The results show that reducing the Patchsize, increasing the number of Layers, and increasing the Hidden Size all further improve the model's performance. The position of Temporal and Spatial attention and the number of attention heads do not significantly affect the model's results. Further comprehensive analysis indicates that generally, **an increase in GFlops leads to better results, demonstrating the scalability of VDT**. When maintaining the same GFlops, some trade-offs in design are necessary, but overall, the model's performance does not differ significantly. And we have included these experiments in the Appendix A of our paper.

| Patch Size, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 4    |   1.9   | 643.9      |
| 2    |   7.7   | 554.8      |
| 1    |   30.9   | **466.2**      |


| Head, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 3      |   7.7      | 559.8      |
| 6      |   7.7      | **554.8**      |
| 12     |   7.7       | 598.7      |

| Layer, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 6       |   3.9     | 580.7      |
| 12      |   7.7      | 554.8      |
| 18      |   11.6     | **500.6**      |


| Hidden Size, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| 192     |   1.9         | 704.2    |
| 384     |   7.7         | 554.8      |
| 768     |   30.9         | **464.2**      |


| Architecture, Iteration=40k| GFlops | UCF101 FVD |
|------------|------------|------------|
| Spatial First   |   7.7         | 550.2      |
| Temporal First   |   7.7         | 554.8      |

Reviewer comment (Presentation):
Tab.5, 142.3 (blacken) is not the best one.

Author response:
**A:** Thank you for pointing it out! We have revised it.

Reviewer comment (Experiments):
According to Table 4, the FVD values of the diffusion model pre-trained with U-Net are significantly better than VDT. However, the authors mention GPU resource limitations as a potential issue. I suggest that the authors compare the results of relevant tasks to provide further clarification.

Author response:
**A:** Thank you for your positive comments and for suggesting a more detailed comparison regarding the FVD values of our diffusion model in relation to U-Net pre-trained models.

We fully recognize the value of such a comparison in providing a clearer understanding of our model's performance. However, as you rightly pointed out, we face significant limitations due to computational resource constraints. This restricts our ability to perform direct comparisons with highly resource-intensive models like Make-A-Video, which benefits from pre-training on extensive datasets (for instance, 2.3 billion image-text pairs and 20 million image-text pairs).

To circumvent this limitation and still offer a meaningful comparison, we have expanded the analysis in Figure 4 (also presented below) to include additional video diffusion models. This broader comparison set includes models such as Make-A-Video, PyoCo, VideoGen, and VDM, which employ similar structures. Our findings indicate that our VDT model competes favorably with non-pretrained structures like VDM and PyoCo, achieving FVD scores of 225.7 compared to their scores of 295.0 and 310.0, respectively.



| Diff. based on U-Net with Pre: |            |       |
|--------------------------------|------------|-------|
| VideoGen                       | 16×256×256 | 345.0 |
| Make-A-Video                   | 16×256×256 | 81.3  |
| **Diff. based on U-Net:**          |            |       |
| PYoCo                          | 16×64×64   | 310.0 |
| VDM                            | 16×64×64   | 295.0 |
| **Diff. based on Transformer:**    |            |       |
| VDT                            | 16×64×64   | 225.7 |

This comparative analysis underscores the efficiency and potential of the VDT network structure, even in the absence of extensive pre-training.

Reviewer comment (Evaluation):
It is unclear why MCVD-concat performs better than VDT in FVD.

Author response:
**A:** Thank you for your excellent question regarding the performance comparison between MCVD-concat and VDT, specifically in the context of the FVD metric.
* On the Cityscaps dataset, we observed that while VDT outperforms MCVD in terms of SSIM, it shows a slightly weaker performance in FVD compared to MCVD. The FVD metric primarily assesses how closely the sample space of generated videos resembles the original video space. MCVD, which is specifically tailored for video prediction, creates shorter video clips (3-5 frames) based on previous 2 conditional frames and then concatenates them. This approach allows MCVD to maintain the features of the conditional frame effectively in the spatial domain, thereby giving it an edge in the FVD metric. On the other hand, our VDT model generates the entire video sequence in one go, based on the conditional frame, which slightly impacts its performance in FVD. However, it's important to note that VDT still achieves results very close to MCVD in FVD.

* The SSIM metric, contrasting with FVD, measures the similarity between predicted and target frames for each video, necessitating higher accuracy in video prediction. As MCVD generates only a few frames at a time, it can suffer from issues like color jitter and speed discrepancies across the entire video sequence. Our VDT model, which models the entire prediction process holistically, manages to maintain overall video coherence and accuracy. Consequently, VDT significantly outperforms MCVD in the SSIM metric, demonstrating its strength in ensuring consistency and precision in video prediction.

We hope this explanation clarifies the distinct advantages and limitations of both MCVD-concat and VDT in different evaluation metrics and their respective implications for video prediction performance.

Reviewer comment (Presentation):
Due to the resolution, it is difficult to distinguish the differences between Videofusion and VDT in the TaiChi-HD section.

Author response:
**A:** Sorry for the challenge in differentiating between Videofusion and VDT in the TaiChi-HD section due to the resolution issue.

To address this, we have uploaded high-resolution comparison images in our appendix **(Figure 8)**, which showcases the differences more clearyly. Both VideoFusion and our VDT can generally produce videos. However, VideoFusion's generated results often exhibit less coherent movements (e.g., shaky hands, making it difficult to discern the shape of the hands), while VDT demonstrates more fluid and dynamic movement. Also, in terms of video details, VideoFusion generates clothing folds that are less natural, whereas the folds in VDT adjust more naturally with the movement of the characters. These demonstrate VDT's superior performance in maintaining details and dynamic motion.

Reviewer comment (Writing):
The presentation was found to be easy to follow by the reviewer. However, there were some typos in the paper that caused misunderstandings. For example, in one instance, the paper mentioned that "the input can either be pure noise latent features or a concatenation of conditional and noise latent features." The word "conditional" should have been replaced with "conditions". Also, the paper stated that "the results of convergence speed and sample quality are presented in Figure 3 and Table 2, respectively." However, the actual result was shown in Figure 7.

Author response:
**A:** Thank you for pointing them out! We have revised the paper accordingly.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.

Reviewer comment (Evaluation):
Although the current quantitative analysis indicates that VDT performs better, the experimental results may not support this fact. Can you provide further analysis and results with improved quality?

Author response:
**A:** Sorry for the challenge in differentiating between Videofusion and VDT in the TaiChi-HD section due to the resolution issue.

To address this, we have uploaded high-resolution comparison images in our appendix **(Figure 8)**, which showcases the differences more clearyly. Both VideoFusion and our VDT can generally produce videos. However, VideoFusion's generated results often exhibit less coherent movements (e.g., shaky hands, making it difficult to discern the shape of the hands), while VDT demonstrates more fluid and dynamic movement. Also, in terms of video details, VideoFusion generates clothing folds that are less natural, whereas the folds in VDT adjust more naturally with the movement of the characters. These demonstrate VDT's superior performance in maintaining details and dynamic motion.