Paper: LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation (ICLR-2024)

Reviewer comment (Experiments):
Data-related issues: The authors do not compare their method to datasets used in their previous work like "tune-a-video." This omission may undermine the fairness of the experimental results.

Author response:
In Section 5.1 of our paper, we have claimed that the dataset used for our experiments is DAVIS, which is the same dataset used in "Tune-A-Video." We acknowledge that in some cases, such as in Figure 1, we choose to include partial images which is not from DAVIS. Our intention is to demonstrate the performance of our approach under challenging conditions and highlight its effectiveness on handling complex objects and  backgrounds.

Reviewer comment (Experiments):
Base model choice: The authors employ ControlNet as the base model instead of using LDM directly. ControlNet offers strong structural control, which might make the improvement from LatentWarp seem relatively small. It would be beneficial to provide experimental configurations with LatentWarp combined with LDM or Tune-a-Video to showcase this point.

Author response:
Firstly, it is important to note that many of the comparative methods we evaluated in our study also utilized ControlNet, such as Rerender a video. This choice was made to ensure a fair and consistent baseline comparison across different techniques. However, we understand the concern that ControlNet's strong structural control may potentially diminish the observed improvement from LatentWarp. To address this concern, we have conducted experiments comparing the performance of LatentWarp with and without controlnet in the ablation experimentaion in the [supplementary material](https://diffusion-latentwarp.github.io/sm/supp.html). Such experiments demonstrate the distinct contributions and improvements offered by LatentWarp using LDM directly.

Reviewer comment (Experiments):
Compared methods: It is advisable for the authors to compare their method with a broader range of existing approaches, such as Video-P2P and more recent methods.

Author response:
We conducted extensive experiments to demonstrate the effectiveness and superiority of our proposed method in comparison to more recent approaches,  including Rerender a video, Tokenflow, Video-P2P, and others. The detailed results of these video-based comparisons can be found in in the [supplementary material](https://diffusion-latentwarp.github.io/sm/supp.html).

Reviewer comment (Evaluation):
User surveys: The authors did not provide user surveys as prior works have done. This is important to evaluate the visual effect.

Author response:
Good suggestion. We refer to previous works[1,2,3] and adopt a Two-alternative Forced Choice (2AFC) protocol, where participants are shown the original video, the translation result of a compared baseline method and the result of our method. The participants are asked to choose which result they prefer.

   We select 15 video-prompt pairs for evaluation and we collect 420 votes from 28 users. The results are provided in the table below, which also verifies the superiority of our method considering realistic user evaluations. We will include this result in our revised paper.

   | Methods                       | Tune-A-Video | Text2Video-Zero | TokenFlow |
   | ----------------------------- | ------------ | --------------- | --------- |
   | User preference of our method | 88.6%        | 80.0%           | 72.9%     |

Reviewer comment (Evaluation):
Running costs: Editing time and GPU resource consumption, should be reported and compared to help readers understand the resource requirements and efficiency.

Author response:
We have measured the inference time and GPU resource consumption of different models for reference. Because the number of generated frames  are limited for some methods, for fair comparison,  the video length is set to be 8 and the image size is set as 512 x 512 for all the methods. 50-step DDIM sampling strategy is adopted. All the experiments are conducted on one NVIDIA A100 GPU. From the table, it can be seen that our method is efficient concerning both the inference time and GPU memory consumption. 

   This could be attributed to reasons of two aspects.  The main advantage of our method is that our method skips the DDIM inversion process, which is typically adopted in video editing tasks to preserve the temporal prior while it consumes lots of time. However, the objective could be achieved through latent alignment operation alternatively, which is obviously more efficient. Moreover, since the warping operation is performed on the latent space, a batch of frames could be denoised on parallel. In contrast, Rerender-A-Video perform the warping operation on the pixel level, which makes the frames need to be generated one by one. 

**Inference time.**

| methods          | flow extraction | tuning | inversion | sampling | total |
| ---------------- | --------------- | ------ | --------- | ------- | ----- |
| video p2p        | -               | 1200s  | 720s      | 55s     | 1975s |
| tune a video     | -               | 1200s  | 8s        | 13s     | 1221s |
| tokenflow        | -               | -      | 55s       | 16s     | 71s   |
| text2video-zero  | -               | -      | -         | 18s     | 18s   |
| rerender a video | 20s             | -      | -         | 182s    | 202s  |
| Ours             | 18s             | -      | -         | 23s     | 41s   |

**GPU memory consumption.**

| methods          | flow extraction | tuning | inversion | sampling | max  |
| ---------------- | --------------- | ------ | --------- | ------- | ---- |
| video p2p        | -               | 10G    | 29G       | 29G     | 29G  |
| tune a video     | -               | 10G    | 29G       | 29G     | 29G  |
| text2video-zero  | -               | -      | -         | 31G     | 31G  |
| tokenflow        | -               | -      | 14G       | 14G     | 14G  |
| rerender a video | 3G              | -      | -         | 13G     | 13G  |
| Ours             | 2G              | -      | -         | 16G     | 16G  |

Reviewer comment (Experiments):
In section 5.1, the overall denoising step number is set to 20 and the proposed method is only applied to the first 16 steps. It would be good if there could be an ablation study about the two stages of the denoising steps.

Author response:
1. **Ablation studies on performing latent alignment on the last few denoising steps.**

​    We conduct the experiment of applying latent alignment operation on the first 16 steps and on the last 4 steps in the denoising process, and present the video results in the [supplementary material](https://diffusion-latentwarp.github.io/sm/supp.html#Ablations).  Although the visual consistency can also be maintained through the last few steps, we observe that errors in the warping operation and optical flow estimation would be introduced in the denoising process especially in the last few steps and reflected on the generation result, hampering the generation quality.

Reviewer comment (Experiments):
Is the proposed method sensitive to the hyper-parameters \alpha and threshold as well as the optical-flow methods? I would like to see some ablation studies on that.

Author response:
2. **Ablation studies on hyper-parameter tuning and optical flow network choice.**

   Please refer to the global response for abltion on hyper-parameter. We also provide the video results with the optical flow extracted by GMFlow[1] and there are no obvious differences.

Reviewer comment (Novelty):
Introducing the optical flow into diffusion-based video processing has been studied by Rerender-A-Video, though it is not applied to the latent.

Author response:
Although introducing optical flow into diffusion-based video processing has been explored in prior work, such as Rerender-A-Video, there are several key differences in our approach: 

   1. Firstly, we extract both forward optical flow maps and backward optical flow maps between adjacent frames and apply a  warping operation on the latent of the last frame to align with the current frame using these flow maps.
   2. The main way we calculate the binary mask for preserving or replacing parts of the warped latent differs from previous optical-flow based diffusion methods, such as Rerender-A-Video. Our methods utilizes a binary mask to determine which regions of the warped latent should be preserved and which should be replaced with latent of the current frame. The details is illustrated in method 4.3.
   3. There are also differences in the fusion stage between our method and Rerender-A-Video. This stage involves combining the warped latents with the generated latents to produce the final output.

Reviewer comment (Theory):
Authors introduce the occlusion mask for indicating the warped region and unwrapped region, but how to guarantee the consistency on the unwrapped region?

Author response:
​      We preserve the consistency of the unwarped regions through cross-frame attention mechanism, i.e., sharing the *key* and *value* tokens across attentions of different frames, to encourage the temporal consistency. As illustrated in the [ablation study, ](https://diffusion-latentwarp.github.io/sm/supp.html#Ablations) without cross-frame attention, the left part of the scene exposed to us with the camera turning left would appears significantly inconsistent with the other regions, hampering the video quality, while our method effectively avoids this issue.

Reviewer comment (Experiments):
Some related works are not compared, such as Edit a Video, Rerender-A-Video, and etal.

Author response:
​      Since Edit-A-Video has not released the code so far, we provide the comparison with Rerender-A-Video and other methods in the [supplementary material](https://diffusion-latentwarp.github.io/sm/supp.html). We will cite and discuss these methods in our revised paper.

Reviewer comment (Evaluation):
I understand that warped query can enhance the temporal consistence but why does it improve visual details (Figure 5) as well?

Author response:
Sorry for this misunderstanding. In Figure 5, we aim not to show the improvement of our method, but to demonstrate that latent alignment effectively preserves the fine-grained visual details. We will clarify this in our revised paper.

Reviewer comment (Novelty):
Since there are lots of open-sourced video diffusion models, which can naturally ensure the temporal consistence, what's the benefit of using Image diffusion model for video editing? Longer video?

Author response:
Compared with video diffusion models, our zero-shot video translation technique has the following advantages:

   1) One significant advantage is the ability to handle longer videos. Image diffusion models typically operate on individual frames, allowing them to handle videos of arbitrary length without the computational and memory limitations that may arise with video diffusion models.

   2) Furthermore, image diffusion models provide flexibility in the editing process. They allow for more fine-grained control over individual frames, enabling targeted modifications and adjustments to specific parts of the video without affecting the entire sequence. This can be particularly useful in scenarios where precise video editing is required, such as in film production or content creation.