Paper: Space-Time Attention with Shifted Non-Local Search (ICLR-2024)

Reviewer comment (Experiments):
Experiments: Results show that the proposed method works as expected, but so many details are not explained and hence it is hard to see how the method really works and how it behaves for different hyper-parameters under ablation studies. For example, patch size P, feature extractor for patches, details predicting offset F, K of top-K.

Author response:
- We have updated Figure 7 in Section 4.1 to address your interest regarding how parameters (such as patch size and query stride) impact the search quality. We also gently note in the supplemental Section 6.2, an experiment uses a patch size of 7 and query stride of 4. Additionally, Sections 6.3 and 6.4 include ablations experiments regarding the key stride and space-time window sizes.

Reviewer comment (Experiments):
Offset instead of alignment/denoising: Experimental results demonstrate how the method works in applications, however, directly investigating the learnt offset value F would be helpful for evaluating the proposed method.

Author response:
- Directly investigating the offsets, $\mathbf{F}$, is important. This is a motivation for Figure 3 in Section 3.3. However, directly comparing offsets to evaluate the quality of a search module will be misleading. This is because attention modules act as a kernel function on the input video. Thus different offsets may produce similar outputs, depending on the content of the value video ($\mathbf{V}$).  Written agin, when the two offsets are not equal, $\mathbf{F}^{\text{Ideal}}  \neq \mathbf{F}^{\text{Shifted-NLS}}$, the final output might be similar, $\mathbf{X}^{\text{out,Ideal}} \approx \mathbf{X}^{\text{out,Shifted-NLS}}$. It may also be true that when the two offsets are similar (but not equal), the final output might be dramatically different. This ambiguity is resolved by comparing offsets through their impact on the attention module's final output. The Video Alignment experiment in Section 4.1 provides an interpretable output target for the attention module. We thank you for pointing out this gap in our explanation and we have updated the text in Section 4.1 to clarify this point.

Reviewer comment (Experiments):
The way authors show that optical ﬂow only needs small spatial corrections is from the results of Sintel-Clean benchmark, however, this setting is far from the real-world dataset, where blur and degradation could happens. Moreover, these results are from methods with high computational cost, which is not feasible for the online setting.

Author response:
**Small Optical Flow Errors on the Sintel-Clean Benchmark.** We have updated our text in Section 3.3 to clarify our point regarding the Sintel-Clean dataset. We agree with you: the results from the Sintel-Clean benchmark will be overly optimistic. Our discussion in Section 3.3 is to illustrate that even recent, large-scale deep networks still report an end-point-error of about 1 even on this “far from real-world” dataset. Yet, using OpenCV’s implementation of Farnback’s 2003 optical flow method (a computationally “cheap” method) combined with a large grid search of 41x41 pixels, the most similar pixels (i.e. the pixels used for attention) exist in a small radius surrounding the offset values. We further acknowledge this radius may grow as noise increases, which is a motivation for the Video Alignment experiment in Section 4.1.

Reviewer comment (Novelty):
The idea is already explored in video enhancement task, such as BasicVSR++ [1] RVRT [2], where the deformable convolutions/attentions' offsets are computed on top of the SpyNet predicted optical flow. Moreover, IART [3] propose an cross-attention scheme by searching around the OF-shifted window.

Author response:
**RVRT uses Predicted Offsets.** We agree that RVRT does indeed use an auxiliary network to predict offsets (named “Predicted Offset” in the paper) with optical flow and video features as inputs. In Table 1 from Section 4.2, we replace this small network with our Shifted Non-Local Search and observe the video denoising quality improve. We have updated the text in Section 4.2 to clarify this point.

Reviewer comment (Novelty):
This paper is the mixture of 

[1] BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (CVPR2022)

[2] Recurrent Video Restoration Transformer with Guided Deformable Attention (NeurlPS2022)

[3] An Implicit Alignment for Video Super-Resolution (arXiv:2305.00163)

Author response:
**Not a Mixture.** Our paper is not a mixture of the three listed papers. 
- First, BasicVSR++ [1] is not considered in this paper as our method is inspired from non-local denoising methods, such as Video Non-Local Bayes [4]. 
- Second, RVRT [2] uses Predicted Offsets while our Shifted Non-Local Search outperforms this method by 3 - 7 dB on video alignment in Figures 6 and 7 (Sec 4.1). When replacing RVRT’s Predicted Offsets with our Shifted Non-Local Search, RVRT’s denoising quality improves by 0.30 dB in Table 1 (Sec 4.2).
- Third, IART [3] is an unpublished paper that provides less functionality than the Shifted Non-Local Search and requires 81 times the memory ([relevant code link](https://github.com/kai422/IART/blob/05ee7d57fcc2c7d028e8389c330b50b5c797aa41/archs/implicit_alignment.py#L144)) when searching a window of size 9x9.

[4] Pablo Arias and Jean-Michel Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 2018.

Reviewer comment (Novelty):
The reproduced results for RVRT 39.29 seems deviate a lot from the original paper 40.57 for DAVIS sigma=10, why this is the case? Moreover, I think RVRT already implement offsets on top of predicted optical flow, which I think is the same method with the proposed one.

Author response:
**Q1. Reproducing RVRT.** One reason our reproduced RVRT network yields lower quantitative results may be due to our computational setup. We use a stand-by queue on SLURM, which requires training in 4-hour increments. While existing open-source packages claim to fully support this functionality, in reality, this may be unreliable [examples of previous issues: [1](https://github.com/Lightning-AI/lightning/issues/18588), [2](https://github.com/Lightning-AI/lightning/issues/12812)]. Still, the STAN network is trained in the same fashion and yields significantly better results. Training time may also be a factor, which we discuss in more detail in Supplement Section 6.4.  

**RVRT uses Predicted Offsets.** We agree that RVRT does indeed use an auxiliary network to predict offsets (named “Predicted Offset” in the paper) with optical flow and video features as inputs. In Table 1 from Section 4.2, we replace this small network with our Shifted Non-Local Search and observe the video denoising quality improve. We have updated the text in Section 4.2 to clarify this point.

Reviewer comment (Experiments):
Is there direct comparison on STAN alignment with guided deformable attention? (i.e. comparing results by only changing the alignment module, keep the original backbone unchanged for RVRT.)

Author response:
**Q2. Comparing STAN and RVRT.** Since STAN processes multiple frames in parallel and RVRT does not, we are unable to directly compare the two networks. The updated text at the start of Section 4 hopefully clarifies this point. Notably, in Section 4.2 we do directly compare our proposed grid search against the auxiliary network used within the Guided Deformable Attention module (within RVRT). Table 1 shows the direct comparison between offsets computed from an auxiliary network (“Predicted Offsets” ) and our grid search (“Shifted Non-Local Search”). We also include ablation experiments for both RVRT and STAN in Sections 6.3 and 6.4, respectively.

Reviewer comment (Writing):
The technical explanations of the implementation of the approach are difficult to follow. Section 3.1 and 3.2 could certainly be clarified and simplified. For example, specify the meaning of the indices, use different letters for different variables (what is the difference between I and \tilde_I?; if K_v is the variable for the Keys then do not use K again to denote the number of neighbor, etc).

Author response:
Thank you for the comments on our presentation. We have updated Sections 3.1 and 3.2 to improve the readability of the method. In Sections 4.1 and 4.2, we have added more details to explain the intention of the experiments.

- [Sections 3.1 and 3.2]: We have changed our presentation to guide the reader through increasingly sophisticated search methods within attention modules. I.e. Global Attention -> Neighborhood Attention -> Non-Local Search -> Shifted Non-Local Search.

- [Section 4]: We have added a paragraph summarizing our experiment section.

- [Section 4.1]: We add motivation for using the Video Alignment task as our evaluation of the different search methods.

- [Section 4.2]: We add more detail to explain how RVRT’s auxiliary network (“Predicted Offsets”) are replaced with our grid search (“Shifted-NLS”).

Reviewer comment (Presentation):
The results section is not clear to me. I suggest the authors to start the experiments section by summarizing how they will attempt to demonstrate what are the advantages of their approach through several specific applications using different datasets and different evaluation metrics.

Author response:
Thank you for the comments on our presentation. We have updated Sections 3.1 and 3.2 to improve the readability of the method. In Sections 4.1 and 4.2, we have added more details to explain the intention of the experiments.

- [Sections 3.1 and 3.2]: We have changed our presentation to guide the reader through increasingly sophisticated search methods within attention modules. I.e. Global Attention -> Neighborhood Attention -> Non-Local Search -> Shifted Non-Local Search.

- [Section 4]: We have added a paragraph summarizing our experiment section.

- [Section 4.1]: We add motivation for using the Video Alignment task as our evaluation of the different search methods.

- [Section 4.2]: We add more detail to explain how RVRT’s auxiliary network (“Predicted Offsets”) are replaced with our grid search (“Shifted-NLS”).