Paper: FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR-2024)

Reviewer comment (Evaluation):
From looking at the generated videos, although the proposed method can more cleanly generate longer videos, it seems that the spatial structure of the video (e.g. location of a cat) is very similar throughout the entire video. I believe this may be due to the repetitive nature of shuffled noise repetitions which are generally highly correlated with the structure of the resulting video. So it seems that the method may have a hard time generating more dynamic changes in long videos, such as a cat walking across the screen or scene / camera changes. Could the authors comment on this, or if there are generated video examples with larger structural changes through the video?

Author response:
> **Q1: Spatial structure of the video (e.g. location of a cat) is very similar throughout the entire video**

In some cases, the displacement of the subject is limited due to the reused noises of FreeNoise. However, FreeNoise does not obliterate motion variation or thoroughly fix the spatial structure of objects. Please refer to the common response for more details.

Reviewer comment (Experiments):
How would the proposed window fusing (weighted by frame index distance) compare to a simpler scheme such as just merging the current and/or prior window (i.e. similar to a one or two hot version of the frame index weighting).

Author response:
> **Q2: Advantages of the proposed weighted window fusion compared to the naive version that uses one-hot weights**

The naive version tends to cause abrupt content changes at window boundaries, and the weighted window fusion can suppress such artifacts by smoothing the temporal transition.

Reviewer comment (Experiments):
Have the authors explored other noise augmentation other than shuffling? In general, shuffling does not seem to be the most intuitive method for perturbing gaussian noise, as it also constrains the epsilons (per frame) to the original samples.

Author response:
> **Q3: Exploration of other noise augmentation other than shuffling**

We have explored some other strategies in our early experiments. We have tried mixed noise and progressive noise to make the fragments generated by each window more correlated [1]. However, it brings poor quality results due to the training-inference gap. In addition, we have tried to flip noise frames spatially. Although it brings more new content, abrupt changes in content are also introduced. We have uploaded video results to the anonymous website: https://free-noise.github.io/ (A: Other Noise Scheduling).

[1] Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

Reviewer comment (Theory):
Section 3.1 mentions that "temporal attention is order independent". Does this imply that the VideoLDM does not have temporal positional embeddings? Or how would it be order independent if it did?

Author response:
> **Q4: Clarification on the statement that temporal attention is order independent**

Sorry for the confusion. Through the paper, we use a pre-trained video diffusion model, whose temporal blocks consist of temporal convolution and temporal Transformer with no temporal positional embeddings used. However, if the temporal Transformer uses temporal positional embeddings, the statement that temporal attention is order independent is no longer correct because the position embedding is order sensitive. At that time, temporal attention could be viewed as the superposition of order-independent parts and order-dependent parts. As a result, our method also works for T2V models that use temporal positional embeddings, such as the higher-resolution results (576x1024 model variant of VideoCrafter) demonstrated on our anonymous webpage.

Reviewer comment (Evaluation):
Demo quality. From the demos, we can see that the motions of most generated results are restricted. For example in the ablation study, when the horse running, the background and the position of the horse do not change (even though the legs are moved), which is not a reasonable motion. Therefore, I would say that the proposed method corrupts the motions of the original diffusion models.

Author response:
> **Q1: Demo quality: motions of most generated results are restricted**

In some cases, the movement of the subject is limited due to the reused noises of FreeNoise. However, FreeNoise does not obliterate motion variation or thoroughly fix the spatial structure of objects. Please refer to the common response for more details.

Reviewer comment (Novelty):
Many ideas of the paper are used in previous works already. (1) For noise rescheduling, Reuse and Diffuse [1] proposed to reuse previous noise in the later frames. (2) For window-based temporal attention, Align Your Latents [2] applies a similar idea for long video generation, which does not change the temp conv part, but uses sliding local attention to resue the trained temporal attention. I think there's no intrinsic difference. (3) The motion injection part: interpolating the context is already proposed in Gen-L [3].

Author response:
> **Q2: Many ideas of the paper are used in previous works already.**

(1) Reusing previous noise in later frames (Reuse and Diffuse [1]): [1] extends the noises by simply adding new noises to the reused noises which is significantly different with our noise rescheduling. In practice, the noise reusing scheme of [1] is infeasible to our proposed method because newly added noises will hurt content consistency across frames. Besides, [1] can not be regarded as a previous work since its first submission to arXiv was on **Sept 7th** and the submission date of ICLR is **Sept 28th**.  

(2) Sliding local attention (Align Your Latent [2]): this is a commonly used strategy, which can not address the longer video generation itself, as illustrated in Figure 1 (d). In Table 1 and Figure 4, we treat it as one baseline and our results are significantly better than it. In FreeNoise, we divide the whole clips into several overlapped windows with a fixed stride. When the stride = 1, it degenerates into naive sliding attention. However, stride = 1 means that we can not conduct noise rescheduling, causing repetitive results. Therefore, we need to use window-based attention (stride > 1), which is then combined with our proposed weighted fusion of local window attention and noise rescheduling, making FreeNoise effective and non-trivial.

(3) Context interpolation (Gen-L [3]): Simply applying textual interpolation is infeasible to achieve multi-prompt video generation. Our contribution lies in the novel insight and thorough exploration of how to make it work, i.e. using a basic prompt through certain layers and timesteps, then injecting the target prompt for the rest.

Reviewer comment (Reproducibility):
I’m interested in detailed experiment settings. Please include diffusion sampler configurations, sample resolutions, frame stride, etc. in your future revision.

Author response:
Thanks for your advice. We have added more experiment details in our revised paper. Due to space limitations, we have moved this part to the appendix.

Reviewer comment (Experiments):
Is direct inference, sliding, GenL, and FreeNoise sharing the same pretrained text-to-video model? If it is, then the evaluation is very convincing since they can all generate the same short video using the prompts but only FreeNoise can achieve good long video results.

Author response:
Yes. All the methods are evaluated with the same pre-trained T2V model. In addition, their inference parameters (e.x. DDIM steps, scale of classifier-free guidance) are all consistent except for the design of the method itself.

Reviewer comment (Evaluation):
In which case FreeNoise may fail? A discussion over the limitations is welcomed.

Author response:
FreeNoise has certain limitations in introducing new content along with the frame length increasing because repeated noises are used. Besides, FreeNoise is a tuning-free approach, it inherits the video generation quality from the base T2V model while can not address the originally existing failure cases.

Reviewer comment (Experiments):
100 evaluation prompts have limited diversity. If it is feasible, please add more evaluation prompts to make the comparison more convincing.

Author response:
Thanks for your advice. We use 512 prompts from a standard evaluation paper EvalCrafter [1] to rerun the evaluation. Each prompt is sampled four times with different initial noises and totally 2048 videos are generated for each inference method. The updated results are provided in the revised paper (Table 1).

Reviewer comment (Experiments):
The authors claim FreeNoise achieves the maximum long video of 512 frames due to GPU memory limit. Is it possible to unlock even long video generation ability by using the CPU offload technique?

Author response:
It is feasible but the inference time is crazy for the CPU. In the future, we will explore some technologies to reduce computation [2] and that will be slightly friendly to the CPU.

Reviewer comment (Experiments):
With the help of ControlNet, is it possible to generate more diverse motion with FreeNoise?

Author response:
This is an interesting idea. We have added a depth2video demo at the anonymous website: https://free-noise.github.io/ (B6: Depth2Video). FreeNoise works for ControlNet and additional condition helps to generate more diverse motions. However, naively applying FreeNoise with ControlNet can not work perfectly, because the frame-wise variated depth conditions introduce extra variation to the context. It requires further exploration to make this combination work properly.

Reviewer comment (Presentation):
The method described in this paper lacks suitable diagrams to help illustrate it.

Author response:
Thanks for your valuable comments. For your raised concerns, please refer to our responses to the "Common Response" and "Ability to Generate Videos with Significant Movement". And we have added the pipeline diagram to help understand FreeNoise in the first revision (Figure 3).