Paper: How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
From the theoretical point of view, the presented approach relies on the warping field being a diffeomorphism, while in practice to the best of my knowledge optical flows estimated with off-the-shelf methods are rarely invertible mappings.

Author response:
It is true that the theoretical derivation relies on the warping field being a diffeomorphism, which is generally not the case for optical flow maps. In particular, disocclusions can reveal empty spaces that aren’t mapped to any part of the initial frame. 

We solve this in practice by replacing missing values with newly sampled noise. On very long sequences, we also resample the high-resolution noise periodically, which effectively updates the anchor frame. We will add a more detailed explanation regarding how we handle warping in practical settings in the Appendix.

Reviewer comment (Evaluation):
According to Tab. 1, while the proposed method typically increases temporal coherence, at the same time it worsens frame-wise image plausibility metrics such as LPIPS or FID.

Author response:
For the appearance transfer example, we used FID as a measure of image quality, since there is no ground truth in this case. FID is computed between the LSUN Bedroom dataset and a dataset we create from frames of the video results we have. Unfortunately, FID simultaneously measures sample quality and diversity, while we are only interested in quality. Since we are computing it over a sequence of frames from the same video, the diversity is naturally very low, which explains the overall high values we have in Tab. 1 (typical FID values are <50). Additionally, less temporally coherent methods like ‘Random Noise’ will likely have more variations between frames, which improves diversity. This explains why random noise performs the best wrt. FID, and our method typically performs a bit worse. All in all, the main take-away from the FID comparison is that our method performs in the same range as other Gaussian noise priors (random, fixed, PYoCo, Control-A-Video) and much better than the other interpolation methods. This, in complement with the warp error metric, shows that we are able to take the “best of both worlds”.

Regarding the LPIPS metric for video super-resolution and restoration, we do observe a slight decrease in image quality according to the metrics, which we cannot fully explain. Similarly, the main take-away is that we are performing in the same range as the other noise priors methods, while having much better temporal coherency.

Reviewer comment (Experiments):
None of the video samples in the supplementary except for fluid dynamics provides the results of noise handling with the method proposed by Control-A-Video model. Please note that this model is more close to the presented method in spirit since it also takes input video into account while PYoCo does not.

Author response:
The residual noise from Control-A-Video behaves like fixed noise in regions of uniform color, and like random noise around moving edges between objects. Our experiments have shown that this generally leads to the “worst of both worlds'', as both flickering and texture sticking happens. We have updated the results with more comparisons with the residual noise prior. They can be found on our project webpage: https://warpyournoise.github.io/

Reviewer comment (Experiments):
As indicated in Tab. 1, bilinear interpolation often results in quite decent quality (excl. Video SR task). Haven't the authors considered a small modification of bilinear interpolation which is able to preserve the variance of pixels, namely, replacing the commonly used weighting coefficients with their square roots? Probably, this simple modification can even further improve its performance. Also note that Eq. 5 is also essentially "Gaussian averaging" of subpixels, i.e. averaging using variance-preserving weighting coefficients of $1/\sqrt{k}$ instead of $1/k$.

Author response:
We have indeed considered the exact same modification in our early experiments! It seemed like a very simple idea for solving our problem, but the experiments revealed some issues with this method:
* First, this averaging will only preserve the variance if it averages samples that are independent. This means we cannot simply warp from the previous frame as in standard bilinear interpolation. Rather, we have to go back to the first frame as in our proposed method.
* Second, even by doing so, such a variance-preserving bilinear interpolation would still create correlations between neighboring pixels in the warped noise. This translates to some loss of high frequency details. Therefore, while it preserves the variance, it does not preserve the independence of the pixels.
* Lastly, we tried using these in diffusion models. And because the noise has less high frequency but still has unit variance, the diffusion model is unable to denoise it properly. This leads to even worse artifacts than simple bilinear interpolation.

We included a visualization of the noise advected with the root-bilinear approach, which can be found at https://warpyournoise.github.io/#noise-comp. Additionally, we reported measurements of efficiency for this approach in our [answer to Reviewer 2 (3JNo)](https://openreview.net/forum?id=pzElnMrgSD&noteId=nVUOv3C1dX).

Reviewer comment (Evaluation):
As mentioned above, the presented method is less computationally efficient than the baselines. It would be nice to provide a quantitative evaluation of that (in)efficiency.

Author response:
Our method is indeed computationally less efficient than most other noise prior methods. This is a known trade-off that we decided to make, as our intent was to solve Gaussian noise warping as accurately as we can to see how far that could bring us. Interestingly, it is still comparable to DDIM inversion in runtime. Additionally, our method has two parameters $k$ and $s$ which control the resolution of the upsampled noise and the discretization level of the pixel polygons respectively. These parameters provide users with the possibility to trade off accuracy in the temporal correlations with efficiency.

Please refer to our [answer to Reviewer 2 (3JNo)](https://openreview.net/forum?id=pzElnMrgSD&noteId=nVUOv3C1dX) for further details regarding the method’s efficiency.

Reviewer comment (Reproducibility):
From the manuscript, it is a bit unclear how the 9 triangulation points are warped in Fig. 1a. How exactly is the estimated optical flow $\mathcal{T}^{-1}$ which typically has the same resolution as the image itself, applied to non-integer pixel coordinates? In other words, how is the operation $\textrm{WARP}_{\infty}$ from Algorithm 2 implemented?

Author response:
Following standard practices in physics simulation of fluids, we warp the triangulation points by upsampling the flow map with bicubic interpolation. We will include this in the updated manuscript.

Reviewer comment (Theory):
I suggest adding an explicit proof of why $\textbf{Z}$ is a Gaussian vector (Appendix B.2). Probably, the shortest proof is the one considering the linear combinations $\kappa^T \textbf{Z}$ for all possible $\kappa$, which are obviously Gaussian random variables.

Author response:
We will integrate this into the updated manuscript.

Reviewer comment (Theory):
I recommend adding more comments on how Eq. 24 for the continuous case turned into Eq. 5 for the discrete case. Why is it valid to still rely on the first-order approximation?

Author response:
It is indeed an important aspect of our contribution, we will clarify this in a new section in the Appendix. Intuitively the discretization does not affect the theoretical guarantees because we are only approximating the true warped pixel polygon with a "discretized" one. But the set of all warped polygons still creates a partition of the domain, so all the guarantees on independence and variance preservation still hold. In a sense, our re-interpretation of noise as a continuous field allows us to transfer the error from value space (e.g. bilinear interpolation) to the geometrical representation of the domain (approximation on the shape of the polygon).

Reviewer comment (Evaluation):
The proposed method is computationally inefficient as compared to prior arts. Quantitative comparisons on this aspect would have been more helpful for future research works.

Author response:
We estimate both wall time and CPU time for all methods on a video sequence from the surrogate rendering application (resolution at 256x256). By evaluating the run time for different sub-sequence lengths, we can linearly regress the average per frame computation time of each method. This is evaluated on a GeForce RTX 3090 GPU and an Intel i9-12900K CPU. The table below summarizes the run time of our method against previous approaches. We will add this table to the updated manuscript appendix.

| Method | Wall Time (in ms/frame) | CPU Time (in ms/frame) |
| :-- | :--: | :--: |
| Random | 0.01| 0.01|
| Fixed | 0.01  | 0.01|
| PYoCo (mixed) [1] | 0.01| 0.01 |
| PYoCo (progressive) [1]  | 0.01| 0.01|
| Control-A-Video [2]| 6.08   | 95.46|
| Bilinear  | 5.26| 76.76|
| Bicubic | 6.00| 87.73|
| Nearest| 5.17| 75.73 |
| Root-bilinear| 7.66 |103.78|
| DDIM inversion (20 steps) [3] | 853.42 | 2226.6|
| DDIM inversion (50 steps) [3] | 2125.5  | 3608.3 |
| **$\int$﻿-noise (ours, k=3, s=4)**| **981.76**| **2953.6**|

Our method is indeed less efficient. This is a trade-off that we decided to make, as our intent was to solve Gaussian noise warping as accurately as possible, to understand how far that could bring us. It is worth noting that we have explored several simpler, less computationally heavy ways to warp the noise by making different simplifying assumptions (such as the root-bilinear interpolation methods shown in the table, a modification also suggested by Reviewer 3 (H5Nc) in Question 1). However, none of those led to satisfying solutions.

An analysis of the table shows that simple noise sampling methods like PYoCo, fixed or random noise can be executed efficiently on GPU. Methods that rely on information from the input sequence in a simple way such as Control-A-Video or interpolation methods are almost 3 orders of magnitude slower than the simple noise sampling methods. Finally, our warping method requires ~1s per frame to warp the noise. While being considerably less efficient than aforementioned methods, the proposed approach is comparable to DDIM inversion. With a standard setting of 50 DDIM steps, DDIM inversion takes longer than our method to produce a noise map, while being similar to ours in run time if we reduce the number of steps to 20. Please refer to our answer to Reviewer 1 (13WQ) for more qualitative comparisons with DDIM inversion.

| k \ s | 1| 2| 3| 4|
| -- | -- | -- | -- | -- |
| **0**| 10.0 | 10.3| 10.7| 12.1 |
| **1**| 10.1 | 10.5| 10.9| 12.2 |
| **2**| 10.3 | 10.5| 10.9| 12.3 |
| **3**| 12.2 | 12.5| 13.0| 14.2 |
| **4**| 17.9 | 17.9 | 18.4  | 19.5 |

Additionally, our method has two parameters $k$ and $s$ which control the resolution of the upsampled noise and the discretization level of the pixel polygons respectively. These parameters provide users with the possibility to trade off accuracy in the temporal correlations against efficiency. We refer to the table above to demonstrate how the parameters impact the performance. The measurements represent the wall clock time in seconds that our method requires to compute noise for a video sequence of 24 frames at resolution 256x256. Additionally, we refer to the Section 4 of the paper on how these parameters affect the quality of the results. 

Lastly, our implementation was not optimized for efficiency. We believe it can still be drastically improved, which would make the computational cost of using our noise scheduler negligible.

Reviewer comment (Theory):
The major weakness from the practical point of view is the implicit assumption that temporally correlated noise maps can induce temporally consistent video editing results, which is often not true. This limitation, however, has been acknowledged and explained by the authors in the paper.

Author response:
While there is no theoretical guarantee that diffusion models produce temporally-consistent video editing results with a given temporally-correlated noise map, we observe that this is often the case in practice. Take the translating pizza example on Appendix E: if we add a noise that undergoes the same translation as the input video, the generated images are more temporally coherent. For better visualization and understanding of these results, we refer to the videos at https://warpyournoise.github.io/#warp-ldm. 

Other examples in the paper also highlight that transformations that are applied to the noise will produce similar image transformations, even though that’s not explicitly enforced during training of diffusion models. With this in mind, we believe that our proposed temporally coherent noise would indeed improve the training of video diffusion models, similar to PYoCo [1].

Reviewer comment (Experiments):
I’m wondering how sensitive the method is with respect to the underlying estimated flow map? In other words, how do the errors in the estimated optical flows affect the results?

Author response:
Overall, as one would expect, accurate optical flow is important for achieving good results. From our experiments, we observed that poorly estimated optical flow will still be able to warp the noise, but it tends to create visual artifacts in the form of temporal misalignment, i.e. fine details in the video will not seem to be perfectly synchronized with the motion of larger structures. 

That being said, it is worth noting that the method is still relatively robust to small errors. For the video super-resolution and JPEG restoration task we show in the submission (Figure 4), the optical flow is computed on the degraded, low-resolution video, and our method is still able to produce decent, coherent results.

Lastly, as our noise warping method is agnostic to the way the deformation field is computed, it can also work with more robust methods such as Neural Layered Atlases [2], when flow-based methods struggle to provide accurate motion estimation. These methods map each pixel in each frame to a canonical space common to all frames. In this case, the canonical space is considered the “first frame” and the mapping replaces the accumulated flow map in our algorithm. Please refer to our answer to Reviewer 3 for a more complete discussion about the properties of the warping velocity field.

Reviewer comment (Experiments):
One principled way to obtain a sequence of correlated noise maps is to perform inverse DDIM on the reference video frames (assuming such video is available, which is true in most of the use cases demonstrated in this paper). The authors did mention that technique in the paper, but did not elaborate further and did not show visual results or comparison. I’m wondering how will the noise maps obtained that way compare to the ones obtained from the ∫-noise in terms of the final video quality?

Author response:
The main issue that prevents a fair and straightforward comparison with DDIM inversion is that the latter only produces one single noise map per image, whereas the applications we show generally use DDPM and require as many noise maps as there are steps in the denoising process.

We included a new visual comparison for the bedroom SDEdit surrogate rendering results with DDIM noise inversion that can be visualized here: https://warpyournoise.github.io/#ddim. There are two ways we can think of to use DDIM inversion with SDEdit. 
* The first consists in partially inverting the image to an intermediate noisy version using DDIM inversion and denoising it with forward DDIM. This expectedly leads to a reconstruction of the input synthetic scene with no additional realistic details.
* The second way consists in fully inverting each frame to obtain a set of noise maps, which we treat similarly to the other noise priors we compare to in the original submission. Since the synthetic renders we use as input video are far from the distribution of the LSUN dataset, the inverted noises are not Gaussian. Our results demonstrate this creates noticeable artifacts. 

Lastly, DDIM inversion deterministically maps each frame to a noise, which removes any diversity in the results. In comparison, our method can apply the same warping to different noise maps to generate different equally temporally-consistent appearances.