Paper: Visual Semantic Learning via Early Stopping in Inverse Scale Space (ICLR-2024)

Reviewer comment (Experiments):
While this method serves as a preprocessing measure for input data, there is a notable absence of comparative validation against the performance of similar procedures (for instance, applying varying degrees of Gaussian blur or color perturbation to input images — a data augmentation approach that is, in fact, quite prevalent in network training, rather than solely introducing original clean images). The comparative analysis in Supplementary Material Fig.17 is overly subjective.

Author response:
Our method is not as simple as a data augmentation method, for we do not use any of the original data during our training procedure. A data augmentation method is to use noisy or corrupt image to urge the model to learn more robust information. However, our method directly discovers the most important and robust information, including structural and low-frequency information, and smooths out other suspicious information. It is much different from the data augmentation method and we do not compare the result with them.

Reviewer comment (Novelty):
Numerous studies, such as those found in reference [1,2], have delved into enhancing networks' capacity to process low-frequency information through a series of methods including image denoising and high-frequency noise injection. However, this paper lacks a comparative analysis with the findings of these studies or an exploration of whether there is room for further improvement building on their foundations.

Author response:
We have implemented [1,2] as you suggested, added comparisons with [2,3,4] in Tab.2,7, on the adversarial robustness tasks on CIFAR-10 using ResNet18. [1] performed poorly in our implementations on CIFAR-10, possibly due to its primary focus on ImageNet.

It can be demonstrated that our method outperforms other methods, whether using FGSM or PGD attacks. Also, using our method as preprocessing method, our method can further improve those methods, though it is not a specially designed defence method.

Reviewer comment (Evaluation):
The paper proposes three training methodologies utilizing Total Variation operations, all of which yield certain results, while differing in effectiveness (for example, the method “Fixed Training” appears suitable for low-resolution scenarios, whereas “Finetune” is more applicable for adversarial attacks). However, the paper lacks an analysis explaining these variances, specifically a discussion on the applicable domains for each of the three methods.

Author response:
For iterative method, it can help the model learn the structural information in the images smoothly. For Fixed method, it can learn the information in images with specific sparsity. For Finetune method, it can refine the vanilla model and help it learn more shape information. 

The finetune method outperform others in the adversarial robustness task because through fine-tuning, the model will learn richer information and become less susceptible to attacks, especially white-box attacks (FGSM, PGD). When a sparsity is fixed, the smooth images are very similar to low resolution images, so the fixed model trained on them performs well in low-resolution classification task. As illustrated in Fig.5, the iterative model learns low-frequency information first and then iteratively gains high-frequency information, so it can gain more stable low frequency information. Therefore, it performs well in the classification on low frequency components.

Reviewer comment (Experiments):
The paper lacks a discussion concerning certain parameters, especially an analysis and empirical representation of "early stopping," as mentioned in the title. Additionally, there is an absence of explanation as to why a sparsity of 0.8 was chosen for the Fixed training method, an aspect that could be elucidated with experimental demonstration and analysis.

Author response:
For the early stopping parameter "sparsity", in experiments we have compared the standard classification results of models trained with different sparsity levels in Tab.4, indicating that the outcome remains robust and consistent across varying levels of sparsity, provided it exceeds 0.6. This suggests a relative insensitivity to changes in
sparsity, leading us to focus on a threshold of 0.6 for optimal balance in most of our experiments.

Reviewer comment (Writing):
It would be beneficial to have a more clear and intuitive description of the graph constructed in the "Sparse Projection via Graph Algorithm" section.

Author response:
Thanks for the advice. We have refined this part by summarizing it in a pseudo-code format and providing an additional flowchart of the algorithm in Appendix.A. Hope these information can make it more readable.

Reviewer comment (Writing):
The symbols in formulas need to be specified, e.g., the meaning of β.

Author response:
**A1**: We have improved the explanations for symbols in section 3.1. Specifically, $\beta$ represents the dense parameter for the loss function in Eq. 1. When $\beta$ minimizes the loss, it represents the TV-regularized image. Meanwhile, $\tilde{\beta}$ in Eq. 4 denotes the sparse TV-regularized image derived by projecting $\beta$ onto the subspace formed by the support set of $\gamma$.

Reviewer comment (Evaluation):
The authors claim that they propose an efficient sparse projection method. In addition to superiority in computation and time complexity compared with SVD and LSQR, is there any advantage for performance improvement?

Author response:
**A2**: Our main contribution is to propose the instance smoothing method, and the efficient sparse projection method is to make the time cost tractable. For performance, since our method can derive the analytical solution just as SVD, so theoretically no difference, even results in smaller loss in practice due to numerical issue. LSQR may struggle with convergence speed or precision, relying on iterative solutions.

Reviewer comment (Experiments):
The choice of early stopping time is unclear. Since early stopping is an important operation to disentangle structural information from detailed ones, it’s crucial to illustrate the choice of early stopping time.

Author response:
**A3**: In experiments, we have tried several sparsity levels as the criterion for early stopping, and compared the standard classification results in Tab.4, indicating that the outcome remains robust and consistent across varying levels of sparsity, provided it exceeds 0.6. This suggests a relative insensitivity to changes in sparsity, leading us to focus on a threshold of 0.6 for optimal balance in most of our experiments.

Reviewer comment (Experiments):
The comparison between this method and existing methods is missing. The authors only compare their method with baseline ones, i.e., Vanilla Model and TV Layer.

Author response:
**A4**: TV-layer [1] is a TV regularization method in which we have compared our methods in Tab.1, Tab.2. We also provide new results with [2,3] in Tab.2,7 in the adversarial robustness task. It can be seen that our method can outperform it in adversarial robustness tasks with FGSM and PGD attacks on the CIFAR10 dataset. Also, as a preprocessing method, our method can further improve the accuracy of adversarial images. These results show the effectiveness of our method, though it is not a specially designed defense method. Additionally, we have incorporated LRP [4] visualizations in Appendix M.

Reviewer comment (Experiments):
The authors have not validated the effectiveness and scalability of their method on larger datasets, such as Imagenet.

Author response:
**A1:** We have extended our training to include ImageNet100, with the results presented in Tab.5 and achieving consistent results in standard classification.

Reviewer comment (Experiments):
The authors have not explored the possibility of applying TV regularization on feature maps, which may further improve the robustness and explainability of the models.

Author response:
**A2:** Exploring TV regularization on feature maps is one of our future goals. To date, preliminary trials using TV regularization with our method on feature maps have been made. We train ResNet18 on CIFAR-10 with TV regularization on feature map of the first convolution layer. Evaluate the model using the same experiments in Sec 4.3, we get the results in the following table. As illustrated, the model with TV regularization on the feature map exhibits higher accuracy in capturing low-frequency components while demonstrating lower accuracy in high-frequency components. This observation suggests its capability to effectively learn low-frequency information.

| Model                            | high frequency component | low frequency component |
|------------------|---------------------------|--------------------------|
| Vanilla                          | 30%                      | 43%                     |
| TV regularization on feature map | 20%                      | 65%                     |

Reviewer comment (Experiments):
The authors have not compared with other structure-based methods, such as shape-biased models or edge detection-based models.

Author response:
**A3:** We have compared our method with TVM [1] in Tab.1,2 (named as "TV layer") for noisy and adversarial robustness tasks. Our approach applies TV regularization to image data, distinguishing it from TVM, which requires alterations to the model's structure. For further comparison, we have added comparative analyses with [2,3] in Tab.2,7, focusing on adversarial robustness tasks using CIFAR-10. Additionally, we have incorporated LRP [4] visualizations in Appendix M, as per your suggestion.

Reviewer comment (Experiments):
The authors have not conducted a sensitivity analysis on different TV regularization parameters, which may affect the performance and results of the instance smoothing algorithm.

Author response:
**A4:** We've compared the standard classification results of models trained with different sparsity levels in Tab.4, indicating that the outcome remains consistent across different levels of sparsity, particularly when it exceeds a threshold of 0.6.

Reviewer comment (Experiments):
How do you choose the optimal sparsity level for different tasks and datasets? Is there a general criterion or guideline for selecting the sparsity parameter?

Author response:
**A5:** We clarify that our study did not specifically aim to identify the optimal sparsity for each task. Our method has shown robust performance across various sparsity levels, indicating that the experimental outcomes we observed are largely independent of the chosen sparsity. While most of our experiments were conducted with sparsity settings of 0.6 and 0.8, we anticipate similar results with other levels, such as 0.7.

Reviewer comment (Experiments):
How do you compare your method with other methods that also use TV regularization or other forms of regularization to enhance robustness and interpretability, such as TVM (Yeh et al., 2022b) or LRP (Bach et al., 2015)?

Author response:
**A3:** We have compared our method with TVM [1] in Tab.1,2 (named as "TV layer") for noisy and adversarial robustness tasks. Our approach applies TV regularization to image data, distinguishing it from TVM, which requires alterations to the model's structure. For further comparison, we have added comparative analyses with [2,3] in Tab.2,7, focusing on adversarial robustness tasks using CIFAR-10. Additionally, we have incorporated LRP [4] visualizations in Appendix M, as per your suggestion.

Reviewer comment (Experiments):
How do you handle the cases where the structural information is not sufficient or reliable for the task, such as when the shape is distorted or occluded by noise or other objects? Do you have any strategies to incorporate other sources of information, such as texture or context, to improve the performance?

Author response:
**A7:** Sections 4.1, 4.2, and 4.4 detail experiments demonstrating the resilience of our approach in these challenging conditions with noisy images, adversarial attacks, or low-resolution data. It is shown that structural information is robust in these cases and can help improve the model's robustness.