Paper: Consistent Video-to-Video Transfer Using Synthetic Dataset (ICLR-2024)

Reviewer comment (Novelty):
The paper's technical contribution seems to be incremental. The proposed long video sampling strategy seems to be pretty similar to the sliding window operation but stated in a more formal way.

Author response:
* Our technical contributions include the creation of a synthetic dataset and the development of an effective method for achieving inter-batch consistency in video editing. We also trained a model capable of conducting video editing without additional control and free of per-video tuning.
    * The Long Video Sampling Correction (LVSC) is not merely a sliding window operation. While a simple sliding window method can process video frames in overlapping segments, it does not ensure that the overlapping frames yield consistent results. The key to our LVSC lies in the algorithmic approach (detailed in Equations 4 and 5 of our paper), which is designed to align predictions across overlapping frames in different batches, ensuring consistency and coherence in the final video output. 
    * We have revised our paper to include a new illustrative figure in Appendix C, which delineates the interaction rules between preceding and subsequent batches. This addition aids in better understanding the LVSC's functionality and its distinction from a naive sliding window approach. We kindly request the reviewer to refer to the updated Appendix C for a comprehensive overview of these enhancements.

Reviewer comment (Reproducibility):
Will the dataset be released?

Author response:
* We're undergoing the release process of dataset. Alongside the dataset, we also plan to make available the code necessary for reproducing the data generation, allowing for broader accessibility and replicability of our work.

Reviewer comment (Evaluation):
The generated video shows visual appeals in the given content with different styles while presenting severe jitter in the newly added content.

Author response:
* We acknowledge the issue of jitter, particularly in newly-added content, which stems from the higher complexity in generating these elements without adequate reference data in the conditional video. To address this, a possible solution is the integration of more advanced diffusion models with better generative capacities, which may offer better handling of the complexities involved in newly added content. Additionally, enhancing the diversity of our training data could improve the model's ability to generalize, thus mitigating jitter.

Reviewer comment (Experiments):
The performance of the proposed method relies on the synthetic paired video dataset. Though it gives reasonable sampling strategies on the generated data, if this dataset doesn't closely match real-world scenarios, it may limit the model's utility.

Author response:
* The quality of our synthetic paired video dataset is indeed pivotal to our model's performance. We utilized real-world video captions to craft this dataset, aiming to mirror real-world scenarios more closely. Nevertheless, we acknowledge potential discrepancies between our dataset and actual scenarios. Future efforts could involve using text-to-video models that align more closely with real-world distributions, further bridging this gap for practical applications.

Reviewer comment (Evaluation):
It would be better to quantify the difference between the generated video dataset and some reference one, e.g., computing FID between them. It may also be helpful to validate the effectiveness of the given sampling criteria.

Author response:
* A quantitative comparison, like computing the Fréchet Inception Distance (FID) between our synthetic video dataset and a reference one, provides insightful data on how our dataset mirrors real-world scenarios. Beyond just assessing data realism with FID, future work can involve examining inductive biases by exploring statistical measures like the diversity in video types, their generation success rates, and variety. This will help in accurately measuring disparities with real-world scenarios using various metrics.  
* Regarding our current CLIP-based sampling, which is heuristically driven and biased by human feedback, we note that certain prompts seem more successful in generating synthetic video pairs. Identifying whether this bias originates from the generative model or the sampling criteria is an open question. Future work can involve analyzing successful generation distributions across different prompts and their alignment with the generative model's biases, as well as assessing our sampling criteria's impact on the dataset's diversity and real-world representativeness.

Reviewer comment (Experiments):
How compatible is this approach with different types of video content, various editing instructions, and text-to-video generation methods?

Author response:
* Exploring the compatibility of our method with diverse video content, editing instructions, and underlying video generation models is indeed crucial. In our research, the dataset employed for testing encompasses videos from three distinct sources, and the editing instructions cover four types. This diversity leads us to believe that our method is adept at handling a reasonable range of editing scenarios. However, our evaluation predominantly focuses on real-world videos, which are most prevalent in everyday applications. The method's compatibility with more artistic forms, such as cartoons, or semantically ambiguous videos (for instance, abstract art videos), remains less clear. We anticipate potential challenges with the latter, especially in scenarios where semantic objects are not clearly defined. Regarding editing instructions, our approach performs well when the edits do not significantly alter the video's structure, maintaining similarity between the source and target videos. However, edits that induce structural changes (like transforming a sunset scene into sunrise, where the sun's movement is reversed) pose a challenge, as our synthetic dataset does not model such scenarios effectively.  
* Concerning Text-to-Video Generation Methods: The selection of a specific text-to-video model plays a critical role due to the inherent biases each model brings. Compiling a comprehensive synthetic dataset to account for these variances is a task that demands extensive resources. At the time of our submission, our analysis was constrained by the limited availability of text-to-video models for extensive examination. Future research directions include delving deeper into this aspect to validate the versatility of our method across a spectrum of text-to-video generation techniques.

Reviewer comment (Experiments):
The proposed sampling method to maintain long video consistency is a variation of inpainting sampling methods, which is also widely used in image/video generation tasks. The experimental section provides detailed numerical metrics for various evaluation indicators and user study. However, the compared baseline lacks strength in video editing tasks, there already are some video editing models based on image diffusion models, such as Pix2Video[1], Render A Video [2], TokenFlow[3], which also do not require fine-tuning on a single video. The proposed method in this paper did not compare itself with those models mentioned.

Author response:
* We respectfully disagree with labeling our sampling method as just an inpainting variation. Our approach is specifically designed for diffusion models, integrating correction steps at each denoising stage's end, distinguishing it from standard inpainting methods in image or video generation.
    * Regarding the inclusion of recent video editing models for comparison, we focused initially on methods that support extensive editing capabilities, such as significant changes in object shapes. However, we agree that incorporating recent advancements in tuning-free video editing methods is crucial for a thorough comparative analysis. Therefore, in our final camera ready version, we would like to include comparisons with the mentioned tuning-free video editing approaches [1,2,3]. We initially did not include Rerender A Video[2] in our comparison due to its primary focus on stylization editing. Nevertheless, we recognize the importance of encompassing a variety of approaches in our analysis and will aim to provide a more comprehensive comparison in the updated version of our paper.

Reviewer comment (Evaluation):
From the generated video results in the provided supplementary, it seems that the proposed method does not achieve superior results.

Author response:
* We recommend referring to the numerical comparisons in the experimental section for a more reliable assessment of our method's performance. The supplementary videos are intended to provide additional context but may not fully represent the method's capabilities.

Reviewer comment (Experiments):
In Section 3.1, the authors mentioned that the temporal attention layers are also replaced to adapt to video editing tasks. However, in the showcased results, most videos are style transferred frame by frame. Is there any attempt to simultaneously change the style of the video and modify the motion of the video, such as transforming a person walking towards the left to walking towards the right?

Author response:
* The temporal attention layer replacement in Section 3.1 is primarily aimed at ensuring structural similarity between the input and edited videos. This might give the appearance of frame-by-frame style transfer, but in reality, the frames are processed collectively in batches, not individually, which aligns well with our video editing framework.
    * Regarding altering the motion in videos, such as changing the direction of a person walking, it presents an intriguing yet complex editing challenge. In scenarios like these, the structure of the input and output videos would significantly differ, which is challenging under our current approach. Our data generation pipeline is designed to produce videos with similar visual structures. While our existing model isn't equipped to handle such extensive structural changes, exploring this capability certainly presents an interesting avenue for future research.

Reviewer comment (Evaluation):
In Section 3.2, how is the success rate calculated? Are there any other methods to improve the success rate?

Author response:
* The success rate is calculated as the ratio of successfully generated synthetic videos that pass the CLIP filter criteria (mentioned in Section 3.3) to the total number of generation attempts.
    * To improve this rate, one approach is to source better video descriptions, as discussed in Section 3.2. Other potential methods include employing video generation models with higher capacity or refining the generation algorithm to produce more accurate and relevant synthetic videos.

Reviewer comment (Experiments):
Does using video diffusion model based methods have any advantages over using optical flow in image diffusion models in video editing tasks?

Author response:
* Utilizing optical flow in image diffusion models for video editing is a concept worth exploring, yet its application is not straightforward. There are two primary challenges with this approach: Firstly, when frames are processed sequentially, errors can accumulate rapidly, leading to significant inconsistencies. Secondly, occlusions in the video can create regions where warping operations struggle to maintain consistency, further complicating the editing process.
    * In contrast, video diffusion models inherently embed consistency within their architecture, offering a learnable approach to maintaining coherence across frames. A key advantage here is that frames are processed together in a single batch, which helps in preserving the continuity and overall structure of the video more effectively than methods reliant on optical flow in image diffusion models. This integrated processing approach of video diffusion models provides a more robust and cohesive solution for video editing tasks.

Reviewer comment (Novelty):
The techniques used in the paper are not completely new -- a large portion of them has followed Instruct-Pix2Pix.

Author response:
* Indeed, our work is inspired by the Instruct Pix2Pix framework, particularly in adopting the principle of generating paired data and tuning a model on synthetic data. However, our approach significantly extends this concept into the video domain, which involves unique challenges not present in image editing.
    * A key innovation in our work is the application of the Prompt-to-Prompt (PTP) method to video, where we identified that temporal self-attention replacement is crucial for successful video PTP generation, as detailed in Section 3.1. This adaptation is non-trivial and vital as it ensures the paired synthetic videos retain a similar visual structure.
    * Additionally, while previous works have focused on intra-batch consistency in video editing, our Long Video Sampling Correction (LVSC) and Motion Compensation (MC) mechanisms are pioneering techniques for maintaining inter-batch video consistency in diffusion models. These contributions represent an advancement in the field of text-guided video editing.

Reviewer comment (Evaluation):
Even with all the measures in place, the generated videos are still far from being temporally consistent.

Author response:
* Compared to existing works, our approach takes a step towards an ultimate solution for generating highly consistent videos. The implementation of Long Video Sampling Correction (LVSC) in our method represents an advancement in enhancing temporal consistency. We acknowledge that achieving absolute perfection in this domain remains a challenging and ongoing endeavor. Nonetheless, our method's progress in this area is a testament to its potential and the direction for future research to fully realize the goal of flawless temporal consistency in video editing.

Reviewer comment (Reproducibility):
Which LLM did you use for the in-context learning?

Author response:
* For the in-context learning process, we utilized the MPT-30B model, as described in Section 3.2.