Paper: Video Deblurring with Adaptive High-frequency Extraction (ICLR-2024)

Reviewer comment (Theory):
The motivation that unsharp masking is useful for deblurring is not clear. The only thing used in the proposed network is extracting high-frequency information.

Author response:
The primary goal of video deblurring is to recover high-frequency (HF) details that are typically lost in the blurring process. This recovery of HF information is essential. We employ the unsharp masking technique, a strategy that specifically targets HF components, to aid in retrieving these lost details. This approach underscores the importance of HF information in the deblurring process.

Reviewer comment (Novelty):
The only novelty of this paper is adaptive extracting high-frequency information which is too marginal.

Author response:
Our approach introduces a novel perspective in video deblurring, drawing inspiration from the classic technique of unsharp masking. This unique focus on HF information sets our work apart, as no other studies have explored deblurring through this specific lens, despite its obvious relevance. Our method for extracting HF details goes beyond black box learning with neural networks, which are typically unable to discern such nuances on their own.

Reviewer comment (Experiments):
Why does the network use pre-defined high-frequency kernels to extract information? Why not allow the network to learn it directly?

Author response:
We utilize pre-defined high-frequency kernels because learning a kernel directly without explicit specifications does not guarantee the creation of high-pass filters. The pre-defined high-frequency kernels allow the predicted kernel to be exactly a high-pass filter and eases the computational load by solely concentrating on the prediction of coefficients.

Our experiments, shown in Table 4, compare direct kernel learning to variations of 3D convolution or "Naive Kernels" (when using a linear combination of directly learned kernels). We observed a performance decrease of 0.07dB and 0.09dB in these methods.

Reviewer comment (Evaluation):
The authors do not provide enough analysis to demonstrate the effectiveness of the high-frequency information. More analysis is needed.

Author response:
We have conducted extensive experiments to demonstrate our method's effectiveness, including distortion performance analysis (Table 1-3), visual comparisons (Figure 2-3), and ablation studies on high-frequency extraction techniques (Table 4). Additionally, we provide visualizations of the representations (Figure 4), the generated kernels (Figure 5), and their performance across various frequency sub-bands (Figure 6).  Taken collectively, we think this presents a convincing case for the effectiveness of our approach on the HF information.

We are open to specific suggestions for additional analysis.  Alternatively, we welcome specific questions about the effectiveness which can guide us in providing more analysis.

Reviewer comment (Evaluation):
Some improvement is too marginal, e.g. Table 2 and Figure 2 (with artifacts).

Author response:
In Table 2, our model outperforms STDANet-Stack by 0.14dB, a significant achievement given that STDANet-Stack is over 10 times more complex, as shown by their GMACs in Table 1. The disparity in complexity across various video deblurring models makes direct comparisons challenging. For instance, the complexity range in the models compared in Table 1 spans a factor of 139. A more balanced assessment is presented in our ablation study (Table 3), where our method boosts performance by 0.39dB with only a 7.7% increase in GMACs. This underlines the efficiency of our HF extraction technique in deep models. 

Regarding Figure 2, some artifacts are present due to the severe blurring of the input. However, the key improvement is our model's ability to differentiate details, such as clearly distinguishing the "55" at the top row, where models like CDVD-TSP fail.

Reviewer comment (Evaluation):
My understanding is that the experimental results are presented for a single run per experiment. To better understand the significance of the claimed improvements, it is necessary to run the main experiments multiple times with a set of random number generator seeds and to present the results with properties of their distributions, such as mean and standard deviation under the assumption of a normal distribution.

Author response:
**A1:** This is a solid suggestion. We will add the mean and the standard to the reported table in the revision.

Reviewer comment (Experiments):
I am interested to see the performance of the proposed method compared to previous works on a real-world deblurring dataset, such as [1].

[1] Zhong, Zhihang, et al. "Real-world video deblurring: A benchmark dataset and an efficient recurrent neural network." *International Journal of Computer Vision* 131.1 (2023): 284-301.

Author response:
**A2:** Transferring models to real-world datasets requires careful consideration. We will add the comparison on this real-world dataset [1] in the revision!  

[1] Zhong, Zhihang, et al. "Real-world video deblurring: A benchmark dataset and an efficient recurrent neural network." International Journal of Computer Vision 131.1 (2023): 284-301.

Reviewer comment (Theory):
This work is not well motivated. This paper is proposed based on a statement that neural networks prefer learning low-frequency components. While the work [a] has analyzed that convs are high-pass filters. Please discuss in detail whether these two statements conflict. [a] Park and Kim, How do vision transformers work? ICLR 2022.

Author response:
**A1:** Thank you for your intriguing question! We are not sure about the contradiction and a direct comparison is challenging because the ICLR’22 work does not discuss spectral bias. The experimentation in the two works are also not aligned. An in-depth comparison that could clarify this discrepancy would be intriguing, but it well-exceeds the scope of our current research on video deblurring.

We note however, that spectral bias in neural networks [1] is well-established and cited by hundreds of papers. This idea forms the foundation of our study. Our experimental results confirm that integrating the high-frequency extraction operation enhances the information extraction in both low and high frequency sub-bands in Figure 6 and results in an remarkable improved performance in Table 3 and 4.  In that regard, regardless of the contradiction, we find the basis of spectral bias sufficient for our work.

[1] N. Rahaman et al., On the spectral bias of neural networks, ICML 2019

Reviewer comment (Experiments):
In the third paragraph of Section 1, the authors state that integrating spatial and temporal gradients into a neural network for video deblurring leads to an improvement of 0.39dB. Where does the result come from?

Author response:
**A2:** 0.39dB comes from Table 2 by subtracting the “RGB" (32.39dB) from “RGB+$\nabla x$ + $\nabla_t x$" (32.78dB).

Reviewer comment (Experiments):
The authors mention several times that they utilize a set of predefined high-pass building kernels. How are these kernels predefined? Are they the simple kernels defined in “Building kernels”? They are too simple to extract sufficient high-frequency information. What about learning these kernels?

Author response:
**A3:** We define high-pass filter kernels by fixing their weights as the standard high-pass filters, such as derivative filters. The pre-defined filters are proven to be effective in Table 3. The actual high-pass filter is further predicted adaptively from the pre-defined kernels. 

These predefined kernels are not utilized in all building blocks of the network. Instead, we employ them in the network's early stages, where they prove to be adequately effective.

Learning an arbitrary kernel directly does not guarantee that the resulting kernels will be high-pass filters; they could instead function as low-pass filters. To mitigate this, predefined kernels are employed as constraints, ensuring that the learned kernels are high-pass filters. This also eases the computational load by solely concentrating on the prediction of coefficients. Furthermore, we have explored the direct learning of the kernels without constraints. Our findings are shown in Table 4. This direct learning approach is equivalent to a "3D conv" or "Naive Kernels" (if we use linear combination). Our experimental results reveal a performance decline of 0.07dB and 0.09dB for these respective methods.

Reviewer comment (Experiments):
In the experiments, please add the comparisons with more recent methods, e.g., [b] Li et al., A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift, CVPR 2023. [c] Liang et al., Recurrent Video Restoration Transformer with Guided Deformable Attention, NeurIPS 2022.

Author response:
**A4:** We mainly compare with recent methods of similar complexity in the paper. We will consider expanding our discussion to include additional recent methods in the revision.

Reviewer comment (Evaluation):
Compared to the recent video deblurring methods, e.g., [b] and [c], the proposed method does not achieve the state-of-the-art performance, i.e., at least 1.97dB worse than [b] on the dataset of GoPro and 1.11dB worse than [c] on the dataset of DVD.

Author response:
**A5:** At this time, we don’t think a direct comparison to [b] and [c] based only on PSNR is reasonable nor grounds to reject our paper.  These two methods have significantly greater compute and runtime.  The FLOPs of ShiftNet [b] is 4.23 times greater than ours. RVRT [c] has at least 1.63 times the FLOPs of our approach. When considering runtime, ShiftNet is 9.09 times slower than our model, and RVRT lags behind by a factor of 3.15, largely due to its window and deformable attention mechanisms. Given these substantial disparities in complexity, drawing a fair comparison between these models proves challenging. It's worth noting that the reported FLOPs for ShiftNet [b] appear lower due to their use of a smaller input size in their calculations.

Reviewer comment (Experiments):
Please add more comparisons on real examples.

Author response:
**A6:** Due to the page limit, more comparisons are available in the supplementary material.