Paper: Adaptive Window Pruning for Efficient Local Motion Deblurring (ICLR-2024)

Reviewer comment (Experiments):
Since the confidence predictor only uses MLP layers. How many pixels did you shift? Is the feature shift necessary to enlarge the receptive field of the neighbourhood?

Author response:
**The number of shifting pixels and the necessity.** In each AdaWPT block, we shift 4 tokens, which is half of the window size. The number of shifting pixels is proportional to the token size of each block. For example, for AdaWPT 1 and 9, whose resolution is $512\times512$, the number of shifting pixels is $4\times4$. Similarly, for AdaWPT 2 and 8, the number of shifting pixels is $8\times8$... For AdaWPT 5, the number of shifting pixels is $64\times64$. As the block's resolution decreases, the number of shifting pixels increases, effectively expanding the receptive field. This shifting operation aids in enlarging the receptive field of the local neighborhood, thereby enhancing the deblurring performance. Noticeably, we have trained our proposed network without the Feature Shift/Reverse layer, and the absence of these layers results in a PSNR drop of approximately 0.1dB.

Reviewer comment (Evaluation):
What is the mask prediction accuracy on validation set?

Author response:
**The prediction accuracy on the validation set.** We calculate the prediction accuracy by $(TP + TN) / N$, where $TP$ refers to the number of pixels that our model correctly predicts to be blurry, $TN$ refers to the number of pixels our model correctly predicts to be sharp, and $N$ is the total number of pixels. The mask prediction accuracies vary in different AdaWPT blocks. The highest is 94.51\%. Figure 12 in the new version of our Appendix reports the accuracy of each AdaWPT block. Most AdaWPT blocks exhibit high prediction accuracies, indicating the effective recognition of local blurs by our proposed model. We have incorporated this discussion into the new version of Appendix E.6.

Reviewer comment (Experiments):
How did you decide the border when annotating the masks for blurry moving objects?

Author response:
**About the blur mask borders.** We subtract the sharp and blurred images to observe the differences in local regions. This difference map helps us to determine the approximate position of the blurriness and the border. To prevent leakage, we extend the blurry region by 5 pixels, ensuring that all pixels within the annotated mask are considered blurry. Regarding pixels between blurry and sharp areas, we adhere to a principle: a diffuse spot encompassing more than 5 pixels is categorized as a blurry region, while conversely, it is classified as a sharp region. We add this discussion in the new version of Appendix C.

Reviewer comment (Reproducibility):
If a patch is always abandoned, how is it processed? What layers it will be passed into during inference?

Author response:
**How abandoned patches are processed.** Because the abandoned patches are sharp, they do not go through Transformer layers. They directly go to the Window Compound layer during inference. The Window Compound layer is incorporated \textcolor{red}{to integrate the abandoned windows and selected windows into a unified feature map in inference. This is explained in the new version of Section 2.2, highlighted in red.

Reviewer comment (Experiments):
Why did you only report real-world results on large images? Do you have a chart for comparison on PSNR/FLOPs/runtime under different image resolutions.

Author response:
**Results under different image resolutions.** In the initial version of our paper, we presented results based on large images due to the ReLoBlur testing data exclusively comprising images with substantial resolutions. In this revision, we augment our findings by including results for various resolutions in the subsequent chart. Employing the nearest interpolation, we down-sample the ReLoBlur testing data and conduct comparisons between our proposed model and baselines using both middle-resolution and small-resolution images. The results depicted in the following chart demonstrate that our proposed LMD-ViT outperforms other methods across all resolutions, delivering rapid and efficient inference. This discussion has been incorporated into the updated version of Appendix E.4.

| Image Resolution | Methods  | $\uparrow$PSNR | $\uparrow$SSIM  | $\uparrow$PSNR$_w$ | $\uparrow$SSIM$_w$ | Inference time | FLOPs |
|----------|----------|----------|----------|----------|----------|----------|----------|
| ReLoBlur* ($1076\times718$) | Restormer [1] | 34.28 | 0.9130  | 29.01 | 0.8510 | 0.99s | 1.723T |
| ReLoBlur* ($1076\times718$)   | Uformer [2] | 32.88 | 0.8747  | 28.38 | 0.8293 | 0.25s | 1.158T|
| ReLoBlur* ($1076\times718$)   | LMD-ViT (ours) | 34.59 |  0.9176 | 29.39 | 0.8586 | 0.23s | 266.074G|
| ReLoBlur* ($538\times359$)     | Restormer [1] | 33.10 | 0.8903  | 28.53 | 0.8376 | 0.26s | 450.210G |
| ReLoBlur* ($538\times359$)   | Uformer [2] | 32.88 | 0.8747  | 28.38 | 0.8293 | 0.25s | 321.657G|
| ReLoBlur* ($538\times359$)   | LMD-ViT (ours) | 33.52  |  0.8978 | 28.85  | 0.8451  | 0.10s |162.627G|

[1] Zamir et al, Restormer, CVPR 2022

[2] Wang et al, Uformer, CVPR 2022

Reviewer comment (Experiments):
The key of this method is the adaptive window pruning. It is better to provide an ablation study with the rest tricks as a baseline (i.e., no window pruning in training and testing).

Author response:
**Ablation study of no pruning and rest tricks.** In the original version of our paper, we discussed the performance without pruning strategy in Section 3.4 and Table 2. Line 5 in Table 2 are results of our proposed model without pruning strategies. Except for the pruning trick discussion, we discuss the effectiveness of blur mask annotation, W-LeFF, and feature channels in Section 3.5, Appendix E.1, and E.2, respectively. We add ablation studies of the other tricks in the new version of our paper, including the effectiveness of joint training in Appendix E.3, and local deblurring results under different resolutions in Appendix E.4, and the effectiveness of our backbone in Appendix E.5. All the modifications are highlighted in red.

Reviewer comment (Writing):
Unfinished paragraph in page 2.

Author response:
**About the minor issues.**

1) We delete the unfinished sentence and modified the first paragraph on page 2. We are very sorry for the previous mistake.

Reviewer comment (Presentation):
For figure 1, I think it might be better to visualise the attention window borders, for example, by adding solid lines to show the windows.

Author response:
**About the minor issues.**

2) Thank you for your advice and we really appreciate it! We add the attention windows' borders in Figure 1 in the new version of our paper. Additionally, we deleted the visualization of AdaWPT 9 because the windows are too small to draw the borders. We add the visualization of AdaWPT 8 instead.

Reviewer comment (Presentation):
The summarised contributions are a bit overlapped (point 1 and point 2). I think it's better to claim adaptive window pruning and bluriness confidence prediction as two contributions.

Author response:
**About the minor issues.**

3) We change the first two points of the summarised contributions as: "1) the first sparse vision Transformer framework for local motion deblurring, LMD-ViT, which utilizes an adaptive window pruning strategy to focus computation on localized regions affected by blur and achieve efficient blur reduction without causing unnecessary distortion to sharp regions; 2) a sophisticated blurriness prediction mechanism, integrating a confidence predictor and decision layer to effectively distinguish between sharp and blurry regions". The modifications are highlighted in red in the new version of our paper.

Reviewer comment (Presentation):
I think there is no need to distinguish between AdaWPT-F and AdaWPT-P. Just be simple and united. The name of AdaWPT is enough.

Author response:
**About the minor issues.**

4) We combine AdaWPT-F and AdaWPT-P as AdaWPT in Figure 2 of the new version of our main paper. Furthermore, we delete relative descriptions distinguishing them.

Reviewer comment (Presentation):
Comparison in Figure 6 is not visible.

Author response:
**About the minor issues.**

5) We replace Figure 6 with a more visible picture. The petals and wall tiles reconstructed with our pruning strategy are clearer than those without pruning.

Reviewer comment (Experiments):
Is there an example on a globally clear image (e.g., a still scene). Contrary to Figure 7, will the decision map be all zeros?

Author response:
**About the minor issues.**

6) Please refer to the reply to your fifth question. When inputting static sharp images, the masks are all zero, and the decision maps are nearly in all-black color. The results of these extreme experiments demonstrate the model’s ability to effectively distinguish between blurred and sharp images, showcasing our model's robustness. We add the results and discussion in the new version of Appendix E.7, highlighted in red.

Reviewer comment (Reproducibility):
The authors did not mention whether the presented method LMD-ViT requires blur mask annotation during inference. This is crucial because if the method does require blur masks before inference, it would be helpful to provide instructions on how to generate them beforehand and assess their practicality.

Author response:
**Whether LMD-ViT requires blur masks in inference.** In the inference phase, the input consists solely of locally blurred images without accompanying blur masks. During training, blur mask annotations are utilized to guide the prediction of blurriness confidence. However, in the inference phase, our network is capable of autonomously predicting the locations of blurred regions without requiring additional blur mask annotations. We add this explanation in the second paragraph of Section 3.1, highlighted in red.

Reviewer comment (Experiments):
In the user study experiment, the absence of an explanation regarding the camera equipment used is notable. This is important because when images from the same camera share a common source, the blurriness often exhibits a consistent pattern. Therefore, including images from the same camera would allow us to assess the proposed method's robustness.

Author response:
**The details of acquiring real-world locally blurred images in the user study.** In the user study experiment, we acquired locally blurred images using a static Sony industrial camera and a static Fuji XT20 SLR camera. The camera sensors and shooting environments in this experiment differ from those in the ReLoBlur dataset and the GoPro dataset. Consequently, the blur patterns observed in the user study may exhibit variations compared to the training data. The visual results presented in Figure 4 demonstrate the robustness of our proposed model in effectively addressing real-world instances of local motion blur. We add the details in the third paragraph of Section 3.2, highlighted in red.

Reviewer comment (Novelty):
Some references are missing, like “Window-based multi-head self-attention” in page 2, and “LeFF” in Section 2.4.1.

Author response:
**About the references.** Thank you for your advice. We have added the references of `"Window-based multi-head self-attention" on page 2, and "LeFF” in Section 2.4.1. The modifications are highlighted in red in the new version of our paper.

Reviewer comment (Presentation):
The organization of the paper can be improved.

Author response:
**About the organization and writing.** Thank you for your advice. To improve the writing of our paper, we have made several modifications: 

1) We have removed the second paragraph from Section 1 and incorporated discussions about our novelty in the same section.

2) We have consolidated the explanations for AdaWPT-F and AdaWPT-P under a unified term, AdaWPT, in Section 2. Accordingly, we've updated Figure 2 to reflect this change without differentiating between AdaWPT-F and AdaWPT-P.

3) We add more discussions in the new version of our Appendix. Specifically, the blur mask annotation details in Appendix C, the effectiveness of the joint training technique in Appendix E.3, deblurring results under different resolutions in Appendix E.4, the effectiveness of our backbone in Appendix E.5, mask prediction accuracy in Appendix E.6, results of special cases in Appendix E.7, as suggested by the reviewers. 

4) To maintain the consistency of citing LBAG [1], we replace "LBFMG" [1] with "LBAG" [1] in the new version of our paper. (Notably, LBFMG [1] is a blur mask generation method in paper LBAG. In the original version of our paper, we wrote LBFMG [1] when discussing blur mask annotations.) 

All the modifications are highlighted in red in the new version of our paper.

Reviewer comment (Writing):
The methodology (Sec. 2) consists of too many (unnecessary) acronyms. Moreover, there are some inconsistencies when citing previous works (for example, LBAG (Li et al., 2023), LBFMG (Li et al., 2023), etc.). The submission would strongly benefit from polishing the writing.

Author response:
**About the organization and writing.** Thank you for your advice. To improve the writing of our paper, we have made several modifications: 

1) We have removed the second paragraph from Section 1 and incorporated discussions about our novelty in the same section.

2) We have consolidated the explanations for AdaWPT-F and AdaWPT-P under a unified term, AdaWPT, in Section 2. Accordingly, we've updated Figure 2 to reflect this change without differentiating between AdaWPT-F and AdaWPT-P.

3) We add more discussions in the new version of our Appendix. Specifically, the blur mask annotation details in Appendix C, the effectiveness of the joint training technique in Appendix E.3, deblurring results under different resolutions in Appendix E.4, the effectiveness of our backbone in Appendix E.5, mask prediction accuracy in Appendix E.6, results of special cases in Appendix E.7, as suggested by the reviewers. 

4) To maintain the consistency of citing LBAG [1], we replace "LBFMG" [1] with "LBAG" [1] in the new version of our paper. (Notably, LBFMG [1] is a blur mask generation method in paper LBAG. In the original version of our paper, we wrote LBFMG [1] when discussing blur mask annotations.) 

All the modifications are highlighted in red in the new version of our paper.

Reviewer comment (Experiments):
It is not clear why the GoPro dataset is used for training along with the ReLoBlur training set. In previous works, such as LBAG, only the ReLoBlur dataset is used (see Table 4).

Author response:
**Why we use the GoPro [2] and ReLoBlur dataset [1] for joint training.**
We train with the GoPro dataset [2] and ReLoBlur dataset [1] together mainly because of two reasons. Firstly, joint training could prevent our model from over-fitting and improve the model's robustness. Secondly, we expect our proposed LMD-ViT to deblur both globally and locally. Training with the GoPro [2] and ReLoBlur [1] datasets together improves both the local and global deblurring performances. We compare models trained solely on the ReLoBlur dataset [1] and jointly trained with the two datasets in the Table below. The results show improvements (+0.13dB in local deblurring and +0.42dB in global deblurring) when we add the GoPro dataset [2] to train. We've added this discussion to the new version of Appendix E.3.

| Training data | Training data          | Testing data | Testing data |
| ------- | ------- |------- | ------- |
| ReLoBlur      | ReLoBlur & GoPro | ReLoBlur      | GoPro |
|          ✓         |                              |   35.29 dB / 0.9280 |   31.74 dB / 0.9271  |
|                      |              ✓             | 35.42 dB / 0.9285   |   32.16 dB / 0.9318  |

[1] Li et al, Real-World Deep Local Motion Deblurring, AAAI 2023

[2] Nah et al, Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR 2017

Reviewer comment (Novelty):
The novelty of the submission needs to be clarified.

Author response:
**The novelty of our work.** Our novelty lies in three aspects. 
*Firstly,* the key idea of our method lies in pruning redundant tokens to protect sharp regions as well as enhance inference speed for local motion deblurring Transformers. 
*Secondly,* our work is the first to apply sparse ViT in image deblurring. We introduce a novel AdaWPT block, which only allows windows with blurry regions to go through W-MSA, leveraging the blurriness confidence predicted by an end-to-end trained confidence predictor. The training process takes advantage of a reconstruction loss with Gumbel-Softmax re-parameterization and a pruning loss guided by annotated blur masks. By doing so, we significantly reduce inference time by half and enable the network to concentrate on locally blurred regions. Though sparse ViT was previously adopted in the image classification task, they used a top-K strategy to decide how many tokens should be abandoned in each layer. However, this restricts the model expressions in different locally blurred images. Our proposed method selects tokens with predicted blur mask and performs natural and reasonable token pruning.
*Thirdly,* to better adapt LeFF to local motion deblurring, we introduce a modified version called W-LeFF. W-LeFF only performs convolution within blurry windows. Experiments in Appendix D.1 demonstrate that our proposed W-LeFF enables parallel training and faster inference without significantly compromising performance.
*In conclusion,* we have developed an innovative adaptive window pruning Transformer for local motion deblurring that is both effective and efficient. It is the first approach to apply sparse ViT to image deblurring, effectively bridging the gap between efficient local motion deblurring and ViT. Compared to the CNN-based deblurring method, we achieved improved visual and quantitative performances. Compared to Transformer-based methods, we speed up inference. Though we borrow W-MSA from Uformer, we do not regard it as our contribution. We summarize this discussion in the $3^{rd}$ paragraph in Section 1 in the new version of our paper, highlighted in red.

[1] Li et al, Real-World Deep Local Motion Deblurring, AAAI 2023

[3] Wang et al, Uformer: A general u-shaped transformer for image restoration, CVPR 2022

Reviewer comment (Novelty):
It would be better to discuss the differences between LBAG and Uformer. Compared to LBAG, it simply substitutes the CNN architecture with a Transformer. All the other modules, including sparse ViT, W-MSA (Window-based multi-head self-attention), and LeFF (locally-enhanced feed-forward layer), have been introduced in previous deblurring works.

Author response:
**The difference between LBAG [1] and our method.** 

1) LBAG [1] uses a gate structure at the end of its network to mitigate the deblurring impact on non-blurred regions. However, it still involves unnecessary computations as the entire image has already been processed by the previous parts of the network network. Our proposed LMD-ViT predicts blur regions and prunes tokens at the very start. Thus, it saves the computation for Transformers as much as possible. 

2) LBAG [1] is a CNN-based method. In LBAG [1], the interactions between the image and convolution kernels are content-independent and ill-suited for modeling long-range dependencies. We discussed these two differences in the $2^{nd}$ paragraph of Section 1. 

3) We manually annotate blur mask ground truths rather than using that from LBAG [1]. This is because the masks produced by LBAG [1] have holes and noise, which decrease the precision of window pruning. On the contrary, our annotations are smooth and more in line with human vision. They improve the blur detection precision of our methods and other methods, as illustrated in Section 3.5 of our main paper.

**The difference between Uformer [3] and our method.** 

1) Unlike Uformer [3], which conducts attention operations on every token and consequently distorts sharp tokens in locally blurry images (as illustrated in Figure 1 of our main paper), while also wasting computational resources during inference (as demonstrated in Table 1 of our main paper), our method is more efficient: it filters out unnecessary tokens, allowing only the blurry ones to pass through the Transformer layers. This approach significantly reduces computation and is ideally suited for local motion deblurring tasks. We have included this discussion in the second paragraph of Section 1, highlighted in red, in the updated version of our paper.

2) Different from the main block of Uformer [3] which focuses on restoring images globally, the main block of our method, AdaWPT, prunes tokens based on a confidence predictor and a decision layer trained by a reconstruction loss with Gumbel-Softmax re-parameterization and a pruning loss guided by annotated blur masks.

Reviewer comment (Experiments):
The fairness of the experiments is questionable because the transformer baselines, such as Restormer and Uformer-B, are not trained with the local blur masks, which are deployed during training by the proposed methods. This makes the comparison in Table 1 unfair.

Author response:
**The fairness of the experiments.**  To utilize the blur masks to train, we borrow the gated block from LBAG [1], and the weighted loss from our method (introduced in Section ). We apply the two local blur-aware strategies to Uformer [3] and Restormer [4]. The following table shows that when trained with local blur masks, the PSNR and SSIM scores of Uformer [3] and Restormer [4] improve. However, they need more time to infer and the overall performances are still inferior to ours without the pruning strategy and AdaWPT blocks. We've added this discussion to the new version of Appendix E.5.

| Methods       | PSNR | SSIM | PSNR\_w | SSIM\_w| Inference time|
|-----------------|----------|----------|-----------------|----------|----------|
|Restormer w/ mask| 34.88 | 0.9259 | 29.44 | 0.8806 | 3.75s |
| Uformer-B w/ mask| 35.23 | 0.9277 | 30.22 | 0.8935 | 1.33s |
| LMD-ViT (ours)           |35.42 | 0.9285 | 30.25 | 0.8938 |0.56s |

In the original version of our paper, we did not train baselines with local blur masks for two reasons: 1)  we follow the comparison settings of the LBAG paper, which also trains the baseline networks without blur mask annotations; 2) we want to avoid altering the carefully designed network structures of baseline methods. Noticeably, we aim to compare the baselines in a fair and equitable manner without compromising any structure. Being able to use the blur mask is precisely the motivation and advantage of our model. We regard the whole strategy as our novelty rather than the backbone only.

Reviewer comment (Experiments):
Please explain in detail how the authors manually annotate the blur masks for the ReLoBlur dataset.

Author response:
**The details of blur mask annotations.** As illustrated in Section 2.4.2 of our main paper, we manually mark binary masks for the locally blurred images of the ReLoBlur dataset. In our annotated blur masks, pixels with a value of 1 indicate blurriness, while pixels with a value of 0 represent sharpness. This can be seen in Figure 1 and Figure 5 of our main paper. To confirm the blurry regions, we subtract the sharp and blurred images to observe the differences in local regions. This difference map helps us to determine the approximate position of the blurriness and the border. After confirming the blurry regions, we select moving objects on the \textit{EISeg} software, which can automatically segment different components within each object. Then, we manually refine the blurry regions. To prevent leakage, we extend the blurry region by 5 pixels, ensuring that all pixels within the annotated mask are considered blurry. Regarding pixels between blurry and sharp areas, we adhere to a principle: a diffuse spot encompassing more than 5 pixels is categorized as a blurry region, while conversely, it is classified as a sharp region. We add this discussion in the new version of Appendix C.

Reviewer comment (Experiments):
The baseline results reported in Table 1 are higher than those in their original papers, e.g., LBAG. It would be better to give the reasons and more details when introducing Table 1. The baseline results reported in Table 1 are higher than those in their original papers, e.g., LBAG for 34.85 dB.

Author response:
**The reason why our reported scores are higher than that of the LBAG [1] paper.** There are mainly three factors causing the score difference. 1) We use our newly annotated blur masks for supervised training rather than the blur masks provided by LBAG [1] in their original paper, which improves the LBAG's score [1]. 2) We train LBAG [1] with the ReLoBlur dataset [1] and the GoPro dataset [2] together. 3) We train LABG [1] for 160k steps in our paper. The results from different training steps may fluctuate a little, using the official public code of LBAG [1]. Due to the above three reasons, the scores are higher than those reported in the LBAG [1] paper.

Reviewer comment (Writing):
Typo: "Table table 4" in Appendix C.

Author response:
**About the typos.** Thank you for your advice. We have replaced "Gopro" with ``GoPro" in Table 4 in Appendix C. We modified the sentences in Figure 7: " All the values in the decision map are equal to 1". All the modifications are highlighted in red.

Reviewer comment (Writing):
Typo: "Decision map" in Figure 7.

Author response:
**About the typos.** Thank you for your advice. We have replaced "Gopro" with ``GoPro" in Table 4 in Appendix C. We modified the sentences in Figure 7: " All the values in the decision map are equal to 1". All the modifications are highlighted in red.