Paper: SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR-2024)

Reviewer comment (Novelty):
The paper lacks baselines and novelty, as the authors fail to cite a few very related works that accomplish similar tasks enabling frame prediction and interpolation.

Author response:
**A1** We will include the listed works in the Related Works in the revised manuscript

We delve into the study of a new task, termed as "generative transition", which aims to produce diverse and creative video segments to bridge two distinct scenes. To this end, we propose a short-to-long video diffusion model as an efficient solution, utilizing the initial and final frames of a given scene as inputs. Our model incorporates a text-based and video conditioning method to generate smooth transitional videos. This approach enables us to effectively manage the transition process, ensuring both the coherence and visual quality of the generated videos. Unlike traditional methods that rely on a pre-defined effect or simple interpolation for conditional transition effects, our model can handle more complex scenes and allows for controllable text prompts. We believe this method will inspire greater creativity in video production creators and the industry, while also presenting an intriguing research topic for the community. 

We summarize our contributions as follows: 
1) We propose a new task, namely generative video transition and prediction, aiming at coherent “story-level” long video generation with smooth and creative scene transition and varying lengths of videos; 
2) We adopt the random-mask-based diffusion model for generating smoothness transition video; 
3) we propose three assessing criteria for transition and extensive experiments demonstrate our method the superior performance on the metrics, as well as its ability to be applied across versatile applications

We argue that Masking is a general technique, and has been first used in NLP, e.g., BERT[1]. MCVD and RaMViD also adopted this technique and applied it to diffusion-based video generation. Deviating from them, we propose a latent-mask-based approach, which is more efficient than pixel-based approaches (e.g., MCVD and RaMViD). SEINE is able to conduct masking in more compressed space and directly learn the distribution of entire latent spatiotemporal distribution rather than explicit video distribution. 

[Reference]

[1] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.

Reviewer comment (Novelty):
MaskViT [1] and MAGVIT [2] are MaskGit-like models trained on tokenized video frames which, given the masked learning objective, can typically generalize to enable generation, prediction, and interpolation, with MAGVIT being explicitly trained for this.

Author response:
We argue that Masking is a general technique, and has been first used in NLP, e.g., BERT[1]. MCVD and RaMViD also adopted this technique and applied it to diffusion-based video generation. Deviating from them, we propose a latent-mask-based approach, which is more efficient than pixel-based approaches (e.g., MCVD and RaMViD). SEINE is able to conduct masking in more compressed space and directly learn the distribution of entire latent spatiotemporal distribution rather than explicit video distribution. 

[Reference]

[1] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.

Reviewer comment (Novelty):
MCVD [3] and RaMViD [4] appear nearly identical to the proposed method, where a video diffusion model is trained with masked latents. The exception is the lack of text-conditioning and scale in [3,4], which the reviewer does not believe meets the bar for novelty.

Author response:
We argue that Masking is a general technique, and has been first used in NLP, e.g., BERT[1]. MCVD and RaMViD also adopted this technique and applied it to diffusion-based video generation. Deviating from them, we propose a latent-mask-based approach, which is more efficient than pixel-based approaches (e.g., MCVD and RaMViD). SEINE is able to conduct masking in more compressed space and directly learn the distribution of entire latent spatiotemporal distribution rather than explicit video distribution. 

[Reference]

[1] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.

Reviewer comment (Novelty):
The authors should clarify how their method is novel compared to the prior work mentioned above.

Author response:
We delve into the study of a new task, termed as "generative transition", which aims to produce diverse and creative video segments to bridge two distinct scenes. To this end, we propose a short-to-long video diffusion model as an efficient solution, utilizing the initial and final frames of a given scene as inputs. Our model incorporates a text-based and video conditioning method to generate smooth transitional videos. This approach enables us to effectively manage the transition process, ensuring both the coherence and visual quality of the generated videos. Unlike traditional methods that rely on a pre-defined effect or simple interpolation for conditional transition effects, our model can handle more complex scenes and allows for controllable text prompts. We believe this method will inspire greater creativity in video production creators and the industry, while also presenting an intriguing research topic for the community. 

We summarize our contributions as follows: 
1) We propose a new task, namely generative video transition and prediction, aiming at coherent “story-level” long video generation with smooth and creative scene transition and varying lengths of videos; 
2) We adopt the random-mask-based diffusion model for generating smoothness transition video; 
3) we propose three assessing criteria for transition and extensive experiments demonstrate our method the superior performance on the metrics, as well as its ability to be applied across versatile applications

Reviewer comment (Experiments):
It is necessary to compare against a subset of the mentioned methods as baselines (or a similar model), since there are currently no baselines explicitly trained for the prediction/interpolation tasks.

Author response:
**A2.**
Thank you for the constructive suggestion. In our paper, we compared our model's predictions in Fig. 8 (b) with those of TATS, a state-of-the-art method for video prediction. Our methods outperformed the compared method in terms of overall quality and degradation. To perform further analysis with the random mask-based method, we used MCVD as a baseline for comparison. We adhered to the predictive settings on UCF-101. We sampled 16 frames from both our model and MCVD for comparison. The results shown below demonstrate that our method achieves a better FVD than MCVD.
|  Method| MCVD | Ours |
| --- | --- | --- |
| FVD | 1143 | 406 |

Reviewer comment (Evaluation):
From the qualitative result shown in Figure 6, it seems that the transition is more like an "interpolation" between two scenes. For example, the frames in (row1, col4), (row2, col2), and (row2, col3) are not very natural.

Author response:
**A1. Re More like a "interpolation" :**
For Figure 6, our results differ significantly from those obtained through interpolation. In the newly uploaded rebuttal.pdf of Fig. R3, we have included animated gif results for better visualization. These results demonstrate our approach's diversity and creativity. For example, in sample 1, a raccoon appears to sway to the right along with the ocean waves before morphing into a figure wearing a trumpet. Sample 2 showcases a scene transition achieved through the shimmering effects of stage lighting. Meanwhile, sample 3 depicts a transition from the ocean to the stage, as if unveiled by a theater curtain. In contrast to the interpolation methods (refer to the first row in Fig. R4), the VQGAN-based and Morphing techniques seem to merely employ a simplistic fade-in, fade-out approach.

**Re unnatural results:**
We recognize the presence of motion distortion in the intermediate frames during video transitions. Viewing these frames in isolation might seem unnatural, as they are transitional states. However, we've observed that within the video format in Fig. R3, these distortions integrate smoothly, resulting in natural transition frames when viewed as part of the entire video sequence.

Reviewer comment (Experiments):
In general, it's hard to see if the proposed method provides a good solution to this new task. The paper also lacks enough ablation study of the model architecture design. More discussions and intuitions about this task would be helpful for future works.

Author response:
**A3. Re Ablation study of the model architecture design:**
We add an ablation study of our model design. Our primary focus is on the exploration of a rarely studied task: generative video transition. To address this, we introduce a random mask layer into the base T2V model, following the architecture proposed by Lavie (Wang et al., 2023). This architecture incorporates a stable diffusion v1.4 and temporal attention layers.
To assess the effectiveness of the introduced random mask layer, we conducted an ablation study on the zero-shot UCF-101 dataset. In this study, we utilized fixed masks to mask the first one or two frames. We autoregressively sampled the first 60 frames from a set of 10,100 video samples in the UCF-101 dataset and used Fréchet Video Distance (FVD) as the evaluation metric for the models.
Our findings reveal that our model outperforms fixed mask models, indicating that the random mask model exhibits greater robustness. Additionally, our random mask model demonstrates the ability to perform Image-to-Video generation, generate long videos, and execute video transitions within a single model.

|fvd	|mask 1 frame	|mask 2 frame2|
|---|---|---|
|fixed mask|	683.4	|733.3|
|random mask(ours)	|672.7	|727.3|

**Re More Discussion:**
In addition to the failure cases we provided in the paper,  we will add discussions about the following two points in our final version: 

1) Large transition and small transition: our model is capable of small transitions of similar scenes for a single object and also the large transition of different scenes. While small transitions may also be achieved by interpolation or morphing methods, our model can also generate creative and diverse results. For a more comprehensive explanation and discussion, please refer to the responses provided for Question 1 and Question 6.

2) text-controllable transition: The generated transition effect is influenced by both the input image and the prompt. When there is an inherent connection between the input images, the model automatically identifies similarities and generates transitions accordingly, even when the prompt is reduced or omitted. However, when specific prompt requirements are introduced, the model generates transitions based on those requirements. We conducted an ablation study to investigate controllable transition generation. For a more comprehensive explanation and discussion, please refer to the response provided for Question 5.

**Re Intuitions:** 
We delve into the study of a new task, termed as "generative transition", which aims to produce diverse and creative video segments to bridge two distinct scenes. To this end, we propose a short-to-long video diffusion model as an efficient solution, utilizing the initial and final frames of a given scene as inputs. Our model incorporates a text-based and video conditioning method to generate smooth transitional videos. This approach enables us to effectively manage the transition process, ensuring both the coherence and visual quality of the generated videos. Unlike traditional methods that rely on a pre-defined effect or simple interpolation for conditional transition effects, our model can handle more complex scenes and allows for controllable text prompts. We believe this method will inspire greater creativity in video production creators and the industry, while also presenting an intriguing research topic for the community.

Reviewer comment (Writing):
In Sec. 2, the citation for PYoCo is missing.

Author response:
**A4.1** Thank you for the reminder. PYoco (Ge et al., 2023) is an advanced text-to-video generation model that presents a noise prior approach and utilizes a pre-trained eDiff-I (Balaji et al., 2022) as initialization. We will add the reference in the updated version.

[Reference]

Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. ICCV 2024.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022

Reviewer comment (Writing):
In the last paragraph of Sec. 3, the sentence describing "Long video" is incomplete.

Author response:
**A4.2** Thank you for your reminder. We apologize for the typo and we will complete the sentence in our updated version: "Long video involves recursive using the last few frames of the generated video as input and utilizing masks to predict the subsequent frames. By recursively iterating this process, the model can generate longer video sequences.

Reviewer comment (Presentation):
In Figure 10, the image on the left part has red-green-blue watermarks. Is that example from Gen-2 instead of SEINE?

Author response:
**A4.4** In Figure 10, we show a comparison of our results and Gen-2. In each case, our result is shown on the left and the results of Gen-2 are shown on the right. We found that our model reaches comparable results for image-to-video animation.

Reviewer comment (Experiments):
For controllable transition generation, do we give the first and last frames unmasked to the model for each prompt? If this is the case, I'm wondering maybe the model can also generate smooth zoom-in/out transitions without explicitly adding "camera zoom-in/out" in the prompt. It would be nice to provide ablation study that removes "camera zoom-in/out" in the prompts and see if the generation quality deteriorates.

Author response:
**A.5** Thank you for your question and suggestion. We have added an ablation study for controllable transition generation. We find that the effect of the transition is determined jointly by the given image and the prompt. In the example of the panda, even when we remove the zoom in/zoom out prompt, we still observe that the transition video is performing a zoom-in. This is because the ratio of the panda in the video changes between the two given frames. To achieve a smooth transition, the model tends to zoom in during generation. Additionally, for further research, we used more complex images without a clear single subject for transitions, and the results are shown in Fig. R5. We can see that the transition effects can be somewhat controlled based on different prompts.

Reviewer comment (Evaluation):
As mentioned in the above part, it would be nice if the author can provide some discussions about what kinds of scene transitions (small transition vs large transition, same object vs different objects) their model is good at.

Author response:
**A6.** Thank you for your constructive comment.  Compared to the previous method, we summarize our model is good at generating complicated scenes and large transitions. 
For large transitions involving complex scenes, our model performs exceptionally well when compared to previous methods. It can generate diverse and creative video segments, effectively bridging distinct scenes. This proficiency is evident from the results demonstrated in Fig. R1, where it managed a variety of objects with ease and creativity.  We compared our model against other methods, which are illustrated in Fig. R2. This comparison further highlights the strengths of our approach over conventional techniques.

When it comes to handling transitions involving the same object but within different scenes, as depicted in Fig. R3,  We found our model is good at handling different scenes. In contrast, Morphing and VQGAN interpolation-based method output videos often result in blurred effects and fail to create meaningful or creative content, and the SD-based method struggles to produce videos with temporal coherence.

Reviewer comment (Evaluation):
[Major] My main concern is that there is not enough quantitative evaluation of video transitions. This paper conducted quantitative experiments by randomly selecting one caption from MSRVTT and determining CLIP-TEXT. However, since video transitions occur when scenes change, it does not seem appropriate to evaluate video semantic correlation. Also, no video quality evaluation metrics (such as FVD etc.) have been considered. This makes it difficult to quantify the exact quality of generation.

Author response:
Video transition is a relatively unexplored area in research. In our approach, we tackle this task by providing the initial and final frames of the preceding and succeeding scenes, supplemented with text control. To evaluate the effectiveness of our approach, we utilize text-video alignment. While our model generates two distinct scenes, our primary goal is to facilitate a smooth transition where the generated video frames maintain semantic coherence between the two scenes.
Simultaneously, devising a comprehensive measure of video quality remains a challenge. Therefore, we've incorporated human evaluation as an auxiliary means to assess overall video quality. Encouragingly, both the metrics, CLIPSIM-Scenes and CLIPSIM-Frames, show positive correlations with the results from human evaluations, indicating consistency between quantitative metrics and human perception.

To further scrutinize the quality of our video transitions, we carried out FVD analysis along with frame-wise FID score calculations. The outcomes are presented in the table below. The table reveals that our model consistently delivers superior video quality compared to other methods examined in our study.

Method| Morphing | VQGAN-based | SD-based | Ours |
| --- | --- | --- | --- | --- |
FVD | 573.9 | 445.6 | 502.0 | 181.6 
FID | 35.3 | 36.7 | 62.0 | 13.4

Reviewer comment (Evaluation):
[Major] Several details related to the human evaluation are missing. (such as number of frames in the generated video, the dataset used, and the questions posed in the user study.) Was the user study appropriately reflective of temporal coherence, text-video alignment, and semantic similarity?

Author response:
In our human evaluation, our primary objective is to ascertain the quality of the videos generated by our model. To maintain impartiality in this assessment, we selected a random assortment of results from our quantitative evaluation. These results were generated through rigorous experiments conducted on the MSR-VTT dataset and each video comprises 16 frames.

For the user study, we presented the participants with pairs of videos - one generated by our model and the other by a comparative method. Rather than guiding their assessment, we encouraged the participants to evaluate the overall quality of each video independently. Subsequently, they were asked to cast their vote for the video they perceived as superior in quality within each pair.

Reviewer comment (Experiments):
[Minor] For transitions to be applied in the real-world, it would require generating more than 16 frames. Would the quality be maintained if more frames are generated?

Author response:
Our model can potentially maintain quality while generating additional frames. We utilize a combination of prediction and transition techniques, guided by prompts, to ensure the desired transition effects. This approach has been successfully demonstrated in longer transitions, as seen in our long video demo on the website in Supplemental Material “Adventure of a panda” from 0:11 to 0:14. However, it's important to note that this method involves multiple sampling stages, which poses a challenge for efficiently generating high-quality transitions over an extended number of frames. The effectiveness of maintaining quality over more frames remains an interesting problem.

Reviewer comment (Presentation):
[Minor] In Figure 5 related to video transition, the frame numbers and details are omitted.

Author response:
We appreciate your suggestion and will include this detail for clarity in the next version. We aimed for a fairly balanced selection of intermediate frames and chose the 5th, 9th, and 13th frames to showcase.