Paper: FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR-2024)

Reviewer comment (Experiments):
From my perspective, the text information for editing is only acquired by cross attention with editing prompts. Other editing techniques like prompt-to-prompt are not used. This might cause the inaccurate editing of the background. For example, in the first example of Figure 5, the grass also turns yellow, while TokenFlow and FateZero can better keep the background.

Author response:
**1. The text information for editing is only acquired by cross attention.**

Now we fully understand your concern! We checked the TokenFlow paper carefully and we find that **TokenFlow didn’t use prompt-to-prompt** [1]. They use the image editing technique named PnP-Diffusion [2] (as their default setting: https://github.com/omerbt/TokenFlow/blob/master/configs/config_pnp.yaml). We follow their default setting when generating their samples. Maybe you're confused because of the acronym **PNP** and **prompt-to-prompt**? The editing ability of TokenFlow also purely relies on cross-attention. The better textual editing ability is probably because we don’t need to inject a lot of features/attention weights as TokenFlow to keep motion/structure for very dynamic source videos (e.g., the “truck”). 

Furthermore, we find that the design of visual attention can also affect textual cross-attention in the experiments. They are not completely independent. In the ablation study, we have tried two ways to integrate the flow-guided attention into the framework while keeping the other modules the same (see Figure 7 in the manuscript). However, even though the cross-attention modules are identical, the editing results are different.

Following your suggestion, we use the same setting as their configuration (https://github.com/omerbt/TokenFlow/blob/master/configs/config_pnp.yaml) to edit their source video “woman-running.mp4”.

Our result video: https://i.imgur.com/7ospOcz.mp4 

We cannot reproduce the result shown on their webpage (maybe they used a different configuration than they released). We followed their official code as well as their configuration (https://github.com/omerbt/TokenFlow/blob/master/configs/config_pnp.yaml) and got the result:

Result video from TokenFlow: https://i.imgur.com/mapBHhK.mp4

In this case, our result shows better structure preservation and a more stable background. We also would like you to consider the overall quantitative results. Moreover, since the concurrent work TokenFlow was released 2 months before ICLR submission (and the code was released 2 weeks before the submission), we ask for your understanding that a comprehensive comparison with it was not feasible at the point of submission.

[1] Amir Hertz, et al. Prompt-to-prompt image editing with cross attention control.

[2] Narek Tumanyan, et al. Plug-and-play diffusion features for text-driven image-to-image translation.

Reviewer comment (Evaluation):
DSTA conducts cross attention across all 32 frames, which takes many computational resources. It is better to compare the computational cost and inference time with other methods.

Author response:
**2. Computational Cost.**

Thanks for your notice! The GPU that we used is A100 40GB which has 40 GB VRAM (usually used 27GB). We have carefully checked the code of ControlVideo. They also use DSTA for the whole video by default (Line447-448 https://github.com/YBYBZhang/ControlVideo/blob/master/models/attention.py). The reproduced results of ControlVideo in our paper also follow this default setting.  We also tried to run FLATTEN on RTX4090 which usually allows 27 frames. We will take your suggestion and improve the memory efficiency in the future to edit more frames on common GPUs! As DSTA is not our kernel idea, it can be easily replaced, e.g. by cross-frame attention to two/four/... adjacent frames. We sincerely hope that you will consider our main contribution, which is effective and efficient. **Our flow-guided attention (FLATTEN) has great potential in this case.** You see, we limit spatial attention in FLATTEN while allowing attention to look farther in time, thus saving memory and computation. 

Again, we sincerely thank you for your valuable time and constructive comments! We appreciate your consideration of our contributions!

Reviewer comment (Experiments):
The performances are better than many previous works. What are the samples that this method validates? I am quite curious about the videos that they evaluated.

Author response:
For the quantitative study, we use the same evaluation set (LOVEU) as described in the main paper. The videos and prompts used for the quantitative results can be found at: https://sites.google.com/view/loveucvpr23/track4.  We also use some in-the-wild videos for qualitative results (e.g. the cute cat in Figure 1).

Reviewer comment (Evaluation):
Are there any qualitative or quantitative results about the trajectory patches in the aforementioned cases in question [1]? I want to see if the trajectory is truely following the flow of objectives in the video.

Author response:
Sure! The patch trajectories are predicted well even in complex scenarios. The qualitative results are shown as follows (Reviewer UKpY is also interested in).  The patches on the trajectories are marked with red dots. Note that our approach does not rely on any specific flow estimation module. We use the SOTA flow estimation module RAFT [1] in this work, and the trajectory could be more precise/robust with better flow models in the future.

Occlusion: https://i.imgur.com/8PErmVI.gif

Appearing: https://i.imgur.com/R22S7Ri.gif

Disappearing: https://i.imgur.com/w3nRRwK.gif

Large motion: https://i.imgur.com/fmyPK9q.gif

[1] Raft: Recurrent all-pairs field transforms for optical flow.

Reviewer comment (Theory):
The method is only suitable for scenarios where every pixel in the original video aligns spatially with the generated video. For misaligned areas, the optical flow trajectory of the original video is not appliable for the motion of the generated video, leading to incorrect key and query identifications. For instance, in the example of transforming a cat to a tiger in Figure 1, the tiger's face is larger than the cat and thus there exist pixels that belong to the tiger's face and belong to the background in the cat example. For the original video with the cat, these pixels belong to the background with an optical flow near zero. However, for the tiger, it's part of the face and should rotate with the head, requiring an optical flow describing a leftward movement. This sets too high a requirement for editing scenarios.

Author response:
Our method intentionally avoids the requirement for pixel-level alignment between the original and generated videos. The optical flow is downsampled before being used as guidance (e.g. from 512 * 512 to 64 * 64) and the flow-guided attention works in low-resolution latent space. This improves the tolerance for misaligned areas between the original video and the generated video. In the example of “cat to tiger”, even though the tiger's face is larger than the original cat's. The pixels that belong to the tiger's face move naturally in the generated video and the tiger's face does not stick to the background.

The quantitative and qualitative results demonstrate that our model can output high-quality and highly consistent videos. The edited videos by FLATTEN have natural visual appearance and the same dynamic motion as the original videos. This allows our method to be credibly applied in many application scenarios including style transfer, coloring, texture editing, and shape editing (e.g., editing short videos on social media platforms). 

Your concern about misaligned areas is insightful! In fact, it is challenging not only for our method but also for all current video editing approaches. The other methods using optical flow [1,2] or structural information (e.g., edge/depth map) [3,4] from the original video also face this misalignment problem. Tune-A-Video [5] with finetuning has tried the large-shape editing but the original motion is lost in the result (e.g., https://i.imgur.com/NVT5QbY.gifv). For the task of video editing, the edited video should follow the motion/structure from the source video. Large misaligned areas are somehow in conflict with this task. This issue might be better solved using video generation techniques. 

[1] VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet

[2] Tokenflow: Consistent diffusion features for consistent video editing.

[3] Text2video-zero: Text-to-image diffusion models are zero-shot video generators.

[4] Controlvideo: Training-free controllable text-to-video generation.

[5] Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.

Reviewer comment (Experiments):
In the provided MP4, there is only one visual comparison, which is too limited. It's suggested not to cherry-pick comparisons so that the effectiveness of the method can be judged intuitively.

Author response:
The supplementary video is only a brief introduction and there is an upload limitation (<100MB). Therefore we only showed a challenging example with larger motion. We randomly sample more examples and release them at the anonymous link: https://youtu.be/dkXcjVrp9x0. Further examples will be added to Appendix B in the revised manuscript.

Reviewer comment (Experiments):
The pipeline relies on flow field computation, and errors introduced in that step may affect the results. The paper is missing some discussion, experiments, or results on how robust the method is to mistakes in the flow field computation.

Author response:
We appreciate your positive recognition of our work and the insightful comments regarding the reliance on the pre-computed flow field. We will address your comments as follows:

We would like to clarify that our method does not directly employ the flow field on the pixel level. Instead, we utilize a downsampled flow field as guidance in the latent space and incorporate an attention mechanism. This can effectively mitigate the negative impact of potential errors in the pre-computed pixel displacement field, which is also one key novelty of our work. Furthermore, the flow estimator [1] we used has already demonstrated its superior performance in many applications. To strengthen our idea, we will add the results/answers to the following questions in the revised manuscript.

[1] Raft: Recurrent all-pairs field transforms for optical flow.

Reviewer comment (Evaluation):
How well can the proposed method handle large motion (large displacement in the flow field) or abrupt motion? Are there any examples the authors can provide?

Author response:
**Q1: How well the proposed method can handle large motion or abrupt motion?**
The proposed method based on RAFT overall performs well, even when there is a large motion/abrupt motion. In fact, "racing trucks" is the most challenging example in the dataset. It is a long-term video with large motion, camera movement, and object movement. Our model can still output a high-consistent edited video. The generated video has the same motion as the input video. Several patch trajectories in the example are visualized (the patches on the trajectory are marked with red dots) as follows:

 https://i.imgur.com/8PErmVI.gif

 https://i.imgur.com/R22S7Ri.gif (starts in the middle)

Sometimes some patches (e.g. the background) look quite similar and the predicted trajectories are not 100% accurate. However, they have little negative impact on the editing results and our method is robust in these cases.

https://i.imgur.com/bpHWf7k.gif

We also show some additional editing results for the large motion videos:

https://i.imgur.com/zlGGXpP.mp4

https://i.imgur.com/D53sbDA.mp4

https://i.imgur.com/fQQ8AOP.mp4

Reviewer comment (Experiments):
How well can the proposed method handle videos in which both global motion (camera movement) and local motion (object movement) are present? Are there any examples the authors can provide?

Author response:
**Q2: How well the method can handle videos in which both camera and object movement are present?**

Our method works well with videos where camera movement and object movement occur at the same time. "Racing trucks" is a good example. Another example of “car drift” is shown:

Source video: https://i.imgur.com/qN3bsGS.mp4

Edited video: https://i.imgur.com/Sc81qzN.mp4

Trajectories:

 https://i.imgur.com/fmyPK9q.gif

https://i.imgur.com/w3nRRwK.gif  (stops in the middle)

https://i.imgur.com/0Y37Exi.gif  (starts in the middle)

Reviewer comment (Evaluation):
Are there any situations where the method can do "more harm than good," meaning that errors in the flow-field computation cause the method to produce worse results than the baseline? How often do such cases occur?

Author response:
**Q3: Are there any situations where the method can do "more harm than good"? how often do they occur?**

Our method demonstrates a high level of robustness, and instances where it causes "more harm than good" are very rare. While it's acknowledged that occasional inaccuracies may arise in the flow-field computation, it's crucial to emphasize that the majority of trajectories are accurately predicted. Importantly, the occurrence of all trajectories in a video being inaccurately predicted is an extremely rare anomaly that has not been observed in our extensive testing.

One notable advantage of our approach is the integration of the flow field into the attention mechanism, significantly enhancing adaptability and robustness. In situations where a patch is mistakenly assigned to a trajectory, the attention mechanism mitigates potential harm. When a patch is misassigned to a trajectory, other patches will avoid attending to the wrong patch due to the low similarity with the misassigned patch. Moreover, if the original motion is very complex and many predicted trajectories are not correct, it usually means the original video is also very challenging for the baseline method (which only uses spatial-temporal attention). The results in Table 2 have demonstrated the robustness/effectiveness of our method compared to the baseline.