Paper: TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR-2024)

Reviewer comment (Evaluation):
Compared to other baselines, like Text2LIVE and Gen1, TokenFlow still shows some "flickering" when there are high-frequency patterns. For example, in "Comparisons to Baselines" SM, in the first example "running dog", the ground has severe flickering compared to Gen 1. Also in the third example "cutting bread", there is more flickering in the bread and background compared to Gen 1. Same thing also happened in the comparison to Text2LIVE in "Additional Qualitative Comparisons". Why is TokenFlow not able to maintain the consistency for the high-frequency patterns?

Author response:
**High frequency patterns**  
As discussed in Sec 6 of our paper, the high frequency patterns in our results are attributed to the inherent nature of the latent diffusion model. Given that the latent space of the diffusion model is that of a VAE, which involves lossy compression, some high frequencies in the final RGB frames are hallucinated by its decoder. Such patterns are also evident when encoding-decoding a natural video without any further manipulations. We further discuss this limitation in Sec 6 of the paper.

Reviewer comment (Evaluation):
Based on the previous point, the authors could consider analyzing the reason behind the flickering and include it in **Limitations**.

Author response:
**High frequency patterns**  
As discussed in Sec 6 of our paper, the high frequency patterns in our results are attributed to the inherent nature of the latent diffusion model. Given that the latent space of the diffusion model is that of a VAE, which involves lossy compression, some high frequencies in the final RGB frames are hallucinated by its decoder. Such patterns are also evident when encoding-decoding a natural video without any further manipulations. We further discuss this limitation in Sec 6 of the paper.

Reviewer comment (Evaluation):
It would be great if the authors could also include runtime comparisons.

Author response:
**Runtime**  
Please see our global response for runtime analysis.

Reviewer comment (Experiments):
Why use NN for the correspondences instead of using optical flow? Can pixel-level correspondences like dense optical flow be used for this token-based framework? Can we apply downsampled optical flow maps to the feature maps?

Author response:
**Optical flow**  
Since our method performs propagation in the feature space, the swapped tokens should be ultimately interchangeable for the diffusion model. The most direct way to achieve this is through matching the features explicitly, rather than using subsampled pixel-level optical flow.

Reviewer comment (Experiments):
Is the current framework able to handle some extreme cases, like occlusion? For example, a dog running through some poles but sometimes the dog is occluded by a pole.

Author response:
**Occlusions**  
Please see our global response.

Reviewer comment (Reproducibility):
What is the maximal length TokenFlow can handle?

Author response:
**Video length**  
On an A100 it is possible to run ~300 frames. We believe some optimisations can be made to increase the amount of frames (for example, computing the extended attention on random subsets of the keyframes rather than on all of them).

Reviewer comment (Reproducibility):
What is the post-processing deflickering that is used in the SM?

Author response:
**Post-process in SM**  
We used [this script](https://github.com/OndrejTexler/Few-Shot-Patch-Based-Training/blob/master/train.py).

Reviewer comment (Evaluation):
Regarding the quantitative evaluation, the *edit fidelity* measured by CLIP score does not provide useful/discriminative information. It might also need to include a user study on the visual quality and fidelity.

Author response:
We wish to note that (a) measuring edit fidelity using CLIP score shows that our method is on par in terms of editing capabilities w.r.t. the baselines and other methods; we do not claim to improve visual quality w.r.t. per-frame editing (e.g., PnP); we will make an honest effort to further evaluate visual quality with a user study in the revised version. (b) Ours and the competing methods (besides TAV) are designed to preserve the structure of objects, hence we find this metric valid.  Note that in addition to this metric,  the paper includes a thorough user study to evaluate temporal consistency.

Reviewer comment (Evaluation):
Regarding the quantitative evaluation, the *temporal consistency* measured by optical flow and warping might over-penalize edits that change in shape and tend to favor edits that involve only color/texture changes.

Author response:
We wish to note that (a) measuring edit fidelity using CLIP score shows that our method is on par in terms of editing capabilities w.r.t. the baselines and other methods; we do not claim to improve visual quality w.r.t. per-frame editing (e.g., PnP); we will make an honest effort to further evaluate visual quality with a user study in the revised version. (b) Ours and the competing methods (besides TAV) are designed to preserve the structure of objects, hence we find this metric valid.  Note that in addition to this metric,  the paper includes a thorough user study to evaluate temporal consistency.

Reviewer comment (Novelty):
The supplementary material shows some results of per-frame editing using ControlNet. Is it possible for ControlNet to be used not only for editing but also for providing optical flow guidance? If so, how would it differ from TokenFLow?

Author response:
We would appreciate further clarification of this question by the reviewer.

Reviewer comment (Novelty):
The novelty of the proposed framework is slightly limited, considering that the key components (keyframe sampling, feature aggregation and propagation across frames) are introduced in prior works. Also, it is unclear which part of Section 4.1 is newly proposed in the paper and which is borrowed from other works. It would be great if the authors can elaborate on the main differences from prior methods and specify the novel components/modifications.

Author response:
Thank you for the supportive feedback.  
**Differences w.r.t prior work**  
**To the best of our knowledge, we are the first to introduce a method that performs keyframe editing and propagation directly in the feature space of the model, during the generation process** (as pointed out by reviewers FeQk and Mqtb). Furthermore, our work provides new findings about diffusion features across video frames, supported by empirical analysis (as acknowledged by reviewers FeQk and 6aWr). On the technical side, our framework involves several key and novel components including: TokenFlow propagation – directly extracting and enforcing inter-frame correspondence in the feature space, and a dedicated generation process that alternates between keyframe editing and feature propagation.  

This is in contrast to prior work that either: (i) solely rely on extended attention (Sec. 4.1) , which is insufficient for temporal consistency, as we thoroughly demonstrated in Sec. 5, (e.g. [1], [2]) or (ii) edits keyframes relying on optical flow estimation between them, and propagates edits in RGB using an off-the-shelf propagation method (e.g., [3]); an approach which results in long range inconsistencies and artifacts (see our SM video comparisons), since optical flow estimation between distant frames (i.e. keyframes) is prone to errors.  

[1]Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.09535, 2023.  

[2] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. ArXiv, abs/2303.13439, 2023a.  

[3]  Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation, 2023.

Reviewer comment (Experiments):
The ablation study on keyframe sampling only covers fixed and random sampling. It would be good to also ablate on sampling interval (tradeoff between computation overhead and temporal smoothness). I’m also curious if a dynamic keyframe sampling scheme would further improve the results, especially for occlusion cases.

Author response:
**The effect of interval size**  
We thank the reviewer for the interesting suggestion. The interval size is equivalent to changing the number of keyframes. We note that as less keyframes are used in our framework: (i) runtime decreases (editing less keyframe is faster and requires less memory), and (ii) temporal consistency is improved (since the same tokens are shared across more frames). Nevertheless, too few keyframes will result in inaccurate correspondences which may result in artefacts. Following this comment, we evaluated the effect of the interval size numerically on a subset of videos of 96 frames:  

| interval size | warp error (x 10^-3) | runtime |
|---|---|---|
| 4 | 2.3 | 1656 |
| 8 | 2.2 | 656 |
| 12 | 2.1 | 435 |
| 16 | 2.0 | 359 |



Empirically, we found in all our experiments that an interval size of 8 is a robust choice, balancing runtime and quality of the result.  

**Occlusions**  
Please see our global response.

Reviewer comment (Experiments):
Have you tried to replace some other features such as ResBlock features or attention masks?

Author response:
Thank you for the supportive feedback.  
**Feature selection**  
It is indeed only the self attention tokens that are being replaced. We selected these features as they have been shown to capture fine-grained spatial information, and have been used for controlled image generation [1]. This is in contrast to the cross attention maps that capture only rough layout and hence would not be a good fit for our goal of fine-grained temporal correspondence [2].  
Note that the residual block is followed by a self attention block in every layer, thus the residual features are also overridden (up to skip connections) by our token replacement. 


[1] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.  
[2] Hertz, Amir, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. ICLR (2022).$