Paper: LOVECon: Text-driven Training-free Long Video Editing with ControlNet (ICLR-2024)

Reviewer comment (Novelty):
The authors have pieced together too many other people's methods to achieve the goals, so their model capabilities are limited, such as being unable to perform shape transformations, object additions or deletions. So their products are limited to color or style changes.

Author response:
**Q1. The model capabilities are limited, such as being unable to perform shape transformations, object additions, or deletions.**

**Please first refer to the general reply for the clarification of our contributions.** We also clarify that the inability to perform shape transformations, object additions, or deletions does not stem from the fact that our model capacities are not enough. The related works [1, 2, 3] can neither achieve that because we all hinge on pre-trained diffusion models and controlnets for **training-free video editing**. Besides, as argued, one of our main goals is to keep the fidelity of the generations to the source videos, instead of creating new content. We sincerely hope the reviewer can re-evaluate the contribution of this work based on these clarifications.

Reviewer comment (Novelty):
The innovation of the methods proposed by the authors is limited, as they have not effectively established the temporal information of the video. The so-called cross-window is naive (a minor change of ControlVideo **[1]**) and may be helpful for local video smoothing, but it still cannot truly establish the temporal dependence of long videos. This is reflected in the experimental results that although the generated video actions are locally coherent, the overall appearance is somewhat strange.

Author response:
**Q2. The innovation is limited, and cross-window cannot establish the temporal dependence of long videos？**

Thanks for sharing your concerns. We would like to clarify again that we extend the sparse attention mechanism to cross-window attention for long video editing, while vanilla sparse attention or cross-frame attention leads to noticeable transitions between windows. Fig 5 a) shows the result of the separate edit of each window based on naive cross-frame attention. From the experiments, **we can conclude that the cross-window can establish the temporal dependence of long videos to some extent and alleviate the different styles between windows.**

Reviewer comment (Theory):
Cross-window attention is proposed to improve inter-window consistency, then how do the authors ensure consistency within the window?

Author response:
**Q3. Consistency within the window, and worse performance than fully cross-frame attention-based ControlVideo?**

Thanks for digging into the details. In cross-window attention, we **incorporate the features of the last frame in the current window in the KV matrix**, which can ensure consistency within the window. In fact, the hierarchical sampler in ControlVideo serves the same purpose of establishing the temporal dependence of long videos. Our cross-frame attention is a replacement for it in an auto-regressive pipeline. We also clarify that **it is natural to get better results when incorporating more features (as done by fully cross-frame attention-based ControlVideo), but we argue that we can get comparable results with less computation overhead and quite better results than vanilla sparse attention, and visualization results are not cherry-picked.** Besides, with our complete pipeline, we perform better than baselines, which shows that each part makes a difference to the final effects, especially the latent fusion module can provide more accurate controls and further improve consistency (see car_comparision.mp4 in the updated supplementary for the comparison results of the "a red car" case).

Reviewer comment (Evaluation):
As shown in Figure 5 (a), fully crossframe attention-based ControlVideo not only maintains the loyalty of the background, but also maintains the temporal consistency of the target subject. On the contrary, the author's method blurs the background and changes the target subject in the temporal sequence (the car window turns red).

Author response:
**Q3. Consistency within the window, and worse performance than fully cross-frame attention-based ControlVideo?**

Thanks for digging into the details. In cross-window attention, we **incorporate the features of the last frame in the current window in the KV matrix**, which can ensure consistency within the window. In fact, the hierarchical sampler in ControlVideo serves the same purpose of establishing the temporal dependence of long videos. Our cross-frame attention is a replacement for it in an auto-regressive pipeline. We also clarify that **it is natural to get better results when incorporating more features (as done by fully cross-frame attention-based ControlVideo), but we argue that we can get comparable results with less computation overhead and quite better results than vanilla sparse attention, and visualization results are not cherry-picked.** Besides, with our complete pipeline, we perform better than baselines, which shows that each part makes a difference to the final effects, especially the latent fusion module can provide more accurate controls and further improve consistency (see car_comparision.mp4 in the updated supplementary for the comparison results of the "a red car" case).

Reviewer comment (Novelty):
The main contribution of long-video editing is limited in paper. The cross-window attention mechanism is common in long video editing/generation [1].

Author response:
Thanks for pointing out the connection between our method and Gen-L-Video. From our limited understanding, Gen-L-Video proposes bidirectional cross-frame attention, which is used with a short video clip instead of the whole video, and suggests learning clip identifier $e^i$ for each short video clip to guide the model in denoising the corresponding clip.  But **our proposed cross-window attention is for the whole video**, which is the main difference.

In this field, cross-frame attention is widely used and different papers set different keyframes but we think they serve the same purpose. However, for the whole video consistency when dealing with long videos, we design the cross-window attention mechanism. **Refer to the general reply for more clarifications of our contributions.**

Reviewer comment (Novelty):
The first drawback of the proposed method is that it is very similar to the VideoControlNet, where the main difference is the modification to tackle long video editing cases. 1) The cross-window attention is a special sparse attention mechanism, which has been widely used by other vision models. 2) The latent fusion module has also been used in other works. 3) The frame interpolation mechanism is a long-history video frame smoothing approach. Although combining existing approaches to solve a challenging problem is meaningful, I am hesitant to give a high score on this work since the "added" modules are all very common and bring limited insight into the community.

Author response:
Thanks for pointing out such a connection. Here we clarify that in VideoControlNet, the editing process for a video involves multiple stages. They acquire inpainting masks based on optical flows, generate the keyframes, and interpolate the inner frames irrelevant to the diffusion pipeline. This resembles the hierarchical sampler in [1]. Instead, we edit the whole video with our diffusion pipeline auto-regressively. We think **the two methods are fundamentally different. Please refer to the general response for more clarifications of our contributions.**

Reviewer comment (Experiments):
In the visual comparison, I find the main visual benefit of the proposed method is that it only edits the target region while keeping the other part intact, which, according to my understanding, is contributed by the latent fusion module, making it hard to evaluate the effectiveness of the cross-window attention. I think more visual (like fig. 5a) and quantitative comparisons without the latent fusion would help.

Author response:
Thanks for the suggestion. We **add a new baseline** in Fig 5 a), the separate edit of each window based on naive cross-frame attention. **Note that the experiments do not include a latent fusion module.** Comparing our cross-window attention without the latent fusion module to this baseline, we can **conclude that cross-window attention can effectively improve consistency in long video editing.**

Reviewer comment (Evaluation):
The experiments are only based on 30 videos, which is far from enough to show the robustness of the proposed method. Moreover, I feel CLIP-based metrics may not be reliable enough to evaluate the overall quality of the generated videos. IS and FVD (if having a larger evaluation set as the reference) should be used for the quality comparison.

Author response:
Thanks for pointing it out. We clarify that the number 30 was carefully chosen after thoughtful consideration. In the quantitative comparison in [2], only 25 videos were selected for comparisons with baselines. In [3], 40 videos from the Davis dataset and other videos from the wild were used for comparisons. 

Regarding metrics, unfortunately, there is currently no standardized validation set or evaluation metric available. Therefore, we have followed the practice of utilizing metrics from the CLIP series and conducting user experiments, which align with previous works like [4] in this field. We also add Con-L2 as a metric to evaluate consistency. We welcome more specific suggestions on the metric.