Paper: Rethinking Label Smoothing as a Tool for Embedding Perturbation Uncertainty (ICLR-2024)

Reviewer comment (Experiments):
As the proposed method is an improvement over label smoothing, I think it is necessary to make comparisons with the original label smoothing method. For example, one reasonable baseline is to add the same constant label smoothing to all images in the original setting.

Author response:
To elaborate further on our ablation studies, the term "STD LS" actually represents standard label smoothing as you have suggested. Since the removal of SPIDER’s perturbation function will nullify the effect of exponential smoothing function, we have instead conducted experiments with standard label smoothing. The ablation table's results indicate that standard label smoothing when used solely fails to increase general model robustness, resulting in no improvements (even inferior) in common corruption robustness. To further support our findings, we are willing to provide additional experiments on standard label smoothing in the near future.

Reviewer comment (Novelty):
I think the related work on estimating the label distribution/uncertainty is highly related to this study and would suggest the authors discuss this connection [1,2].

Author response:
We have detailed the comparative analysis between the works you suggested and our own in the following comment. Your suggestion of these pertinent works is greatly appreciated.

- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning 

The paper's main objective is to explore the enhancement of model performance through the ensemble of Deep Neural Networks (DNN) and the distillation of this ensemble knowledge into another DNN. This research diverges from ours as our interest lies in general model robustness, not in knowledge distillation or ensemble methods. 

- Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting 

This study argues that in adversarial training, inputs altered by adversarial means are often labeled too confidently, highlighting the phenomenon of double descent during this process. This phenomenon occurs when DNNs are trained with noisy labels. To address the gap between the assigned labels and the actual labels of these adversarially altered inputs, the authors propose softening the labels through temperature scaling and interpolating the perturbed and original inputs. 

While our research partially shares common ground in terms of using less confident labels for altered inputs, there are significant theoretical and technical differences. Theoretically, we have demonstrated that isotopic perturbations combined with label smoothing enhance overall model robustness analyzing with boundary thickness and smoothness. The referenced study, however, is focused on the inherent conflict between hard labels and the true underlying labels of adversarially perturbed inputs. Technically, where the cited paper reduces label discrepancies in adversarial training through temperature scaling and interpolation, our approach involves isotopic perturbations with varying degrees of label smoothing based on perturbation magnitude.

Reviewer comment (Novelty):
The two specific related works suggested for discussion are:  
1. Towards Understanding Ensemble, Knowledge Distillation, and Self-Distillation in Deep Learning  
2. Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting

Author response:
We have detailed the comparative analysis between the works you suggested and our own in the following comment. Your suggestion of these pertinent works is greatly appreciated.

- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning 

The paper's main objective is to explore the enhancement of model performance through the ensemble of Deep Neural Networks (DNN) and the distillation of this ensemble knowledge into another DNN. This research diverges from ours as our interest lies in general model robustness, not in knowledge distillation or ensemble methods. 

- Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting 

This study argues that in adversarial training, inputs altered by adversarial means are often labeled too confidently, highlighting the phenomenon of double descent during this process. This phenomenon occurs when DNNs are trained with noisy labels. To address the gap between the assigned labels and the actual labels of these adversarially altered inputs, the authors propose softening the labels through temperature scaling and interpolating the perturbed and original inputs. 

While our research partially shares common ground in terms of using less confident labels for altered inputs, there are significant theoretical and technical differences. Theoretically, we have demonstrated that isotopic perturbations combined with label smoothing enhance overall model robustness analyzing with boundary thickness and smoothness. The referenced study, however, is focused on the inherent conflict between hard labels and the true underlying labels of adversarially perturbed inputs. Technically, where the cited paper reduces label discrepancies in adversarial training through temperature scaling and interpolation, our approach involves isotopic perturbations with varying degrees of label smoothing based on perturbation magnitude.

Reviewer comment (Experiments):
SPIDER can be considered a variant of LS. In the meantime, standard LS can be seen as a specific example of a monotonically decreasing function (yet without a clearly defined definition in the paper). Nevertheless, it remains unclear whether and why SPIDER outperforms standard LS. The results shown in Table 6 demonstrate that standard LS actually performs worse than the baseline in terms of mCE metrics, which raises my doubts about whether the thickness of boundaries is necessarily linked to robustness against common corruptions. Also, there is a lack of comparisons between standard LS and SPIDER in Tables 1-5 and Figure 2.

Author response:
**W1-1. Why does SPIDER instantiation outperform standard LS in terms of mCE metrics? Since standard LS is a marginal case of monotonically decreasing smoothing function, doesn’t the results violate the essence of theorem 1?**

A. You are correct in noting that standard label smoothing (LS) is a form of monotonically decreasing smoothing function. Our research into SPIDER highlights that when such smoothing functions are used ‘together’ with isotropic perturbation, they generally increase model robustness. Our ablation study was designed to explore if each element of SPIDER – the isotropic perturbation and the monotonically decreasing smoothing function – could individually match the overall performance of SPIDER. The results revealed that these individual components either provided limited or only partial improvements in robustness.

**W1-2. There are lack of comparisons between standard LS and SPIDER in Tables 1-5 and Figure 2.**

A. Regarding the comparison between standard LS and SPIDER in our experiments, we believed that a direct comparison with only standard LS was not essential for the aspects explored. In Tables 1 & 2, which focus on boundary thickness, our theorem suggests that any monotonically decreasing label smoothing can enhance boundary thickness in conjunction with isotropic perturbation functions. Hence, there was no implication that solely using standard smoothing methods would increase boundary thickness. Similarly, for Figure 2, our theorems about flatness only indicate that any $L_2$-norm bounded perturbation promotes the discovery of flatter minima. For our main experiments in Tables 3-5, we have partially shown that standalone standard smoothing does not provide overall model robustness, as indicated in our CIFAR-100 ablation experiments (Table 6).

Reviewer comment (Evaluation):
SPIDER improves flat minia compared with ''No LS'', yet not to a sufficiently large extent. The empirical evaluation in Figure 2 indicates that compared with SPIDER, Augmix achieves flatter minia.

Author response:
**W2. SPIDER improves flat minima compared with ''No LS'', yet not to a sufficiently large extent. The empirical evaluation in Figure 2 indicates that compared with SPIDER, AugMix achieves flatter minima.**

A. In response to the concerns about the outlier case where AugMix outperforms AugMix + SPIDER, it is essential to consider the broader context of these findings. While SPIDER generally promotes flatter minima, as shown in our results and illustrated in Figure 2, there are nuances in its interaction with AugMix. In the CIFAR-10 experiments, AugMix achieves notably flat minima on its own, indicating a strong baseline for model generalization through diverse augmentation methods.

In the specific case of 'AugMix + SPIDER', although there is a slight deterioration in flatness compared to AugMix alone, this does not necessarily imply inferior overall performance. The combination 'AugMix + SPIDER' still performs better than other baseline scenarios. For example, 'AugMix+SPIDER' shows a log loss value of around 2.5 at radius 50, which is more favorable compared to 'Naive+SPIDER' at around 3.0, 'PixMix+SPIDER' at around 2.5, and 'DeepAugment+SPIDER' at around 7.0. This indicates that while AugMix's flatness may be slightly reduced when combined with SPIDER, the overall performance remains robust.

Furthermore, SPIDER enhances AugMix's performance in terms of boundary thickness, corruption, and adversarial robustness on CIFAR-10/100, and Tiny-ImageNet, as detailed in Table 3. This suggests that the slight reduction in flatness is a trade-off for improvements in other critical aspects of model robustness. It is important to recognize that the interplay between flatness and other performance metrics is complex and not fully understood yet. Therefore, while the specific case of 'AugMix + SPIDER' may seem like a strong outlier in terms of flatness, it represents a strategic balance, enhancing certain aspects of robustness at the cost of a minimal decrease in flatness. This nuanced understanding is crucial in evaluating the overall effectiveness of model enhancements and underscores the challenges in conducting a systematic analysis in such dynamic and multifaceted scenarios.

Reviewer comment (Experiments):
In Table 6, ''No LS'' corresponds to adversarial training with noise bounded with L2 norm and outperforms the baseline. I am curious about the outcomes when perturbations are applied while employing standard label smoothing, which more rigorously demonstrates the gap between standard LS and SPIDER.

Author response:
**Q1. ''No LS'' corresponds to adversarial training with noise bounded with L2 norm and outperforms the baseline. What will be the outcomes when perturbations are applied while employing standard label smoothing?**

A. On the topic of the “No LS” condition in our ablation study, it refers to models trained exclusively with isotropic perturbations, which are not adversarially generated. The combination of isotropic perturbations and standard label smoothing, as you've mentioned, constitutes another specific form of SPIDER. Thus, the main question to consider is whether this combination, another instantiation of SPIDER, contributes to general model robustness. We have provided some initial results of this SPIDER variant, as shown in the tables below, to illustrate its impact across various metrics and augmentation methods.

| CIFAR10  | mCE original | mCE prtb + std LS | $L_2$ original | $L_2$ prtb + std LS | $L_\infty$ original | $L_\infty$ prtb + std LS |
|----------|--------------|-------------------|-------------|------------------|-------------|-------------|
| no aug.  | 25.57        | 14.02             | 68.57       | 21.50            | 99.52       | 48.85       |
| AugMix   | 11.83        | 9.67              | 36.91       | 26.07            | 93.15       | 63.72       |
| DeepAug. | 13.21        | 10.26             | 54.86       | 22.73            | 95.86       | 45.93       |
| PixMix   | 11.86        | 9.05              | 65.37       | 31.06            | 99.69       | 79.53       |

Reviewer comment (Theory):
For the theoretical results, the link between the proposed method and the theoretical results is unclear. None of the theorems can explain the design choice of $\delta$ and $s$, which is complicated and requires two hyperparameters. If the theorems cannot explain the design choice, there should be an extensive ablation study for the choice of $\delta$ and $s$.

Author response:
**The link between the proposed method and the theoretical results seems unclear.**

  * *i) Theorems on boundary thickness and flatness do not seem to justify the specific design choices for perturbation and smoothing functions.*

    A. Our specific model, i.e., SPIDER, falls into one of instantiations of our theorems regarding boundary thickness and flatness. Precisely, the exact functions used by SPIDER, i.e., $p_\gamma(\|z\|)=\text{Uniform}(0, \tau)$ and $s(z) := \text{clip}(e^{-\lambda(z-\epsilon)}, 0, 1)$ are one of the monotonically decreasing functions and perturbation function bounded by $L_2$ norm, respectively. Because the theoretical results are not restricted to a particular case and cover the broad range of functions of label smoothing and perturbations (including SPIDER), we truly believe that the theoretical advantages are also valid in the case of SPIDER. Therefore, it is hard to say that the link between the theorems and SPIDER is unclear. 

    When focusing on the theoretical reasons for the choices of smoothing and perturbations, we agree that we have not fully explored what the optimal function shape is regarding common corruption and adversarial robustness, as aforementioned in Section 5. While we have not exhaustively examined the optimal function shape, we have conducted minimal experiments with several smoothing functions, including standard smoothing, linear smoothing, and exponential smoothing functions. We found that the exponential smoothing function to be slightly superior in common corruption benchmarks.

  * *ii) Theorems on flatness does not rigorously guarantee SPIDER’s ability to find flatter minima.*

    A. That is correct. Albeit we have not directly formalized the SPIDER’s ability to find flatter minima on general DNNs or MLPs, we have assumed that SPIDER will indeed encourage a model to find flatter minima – which turned out to be generally true in our experimental cases (7 out of 8.) It is obviously challenging to build a theoretical statement covering all possible deep neural architectures, but we believe that the efforts to build a theory for the basic architecture, such as a single-layer perceptron, can be a valuable step for anticipating the behavior of deep models.

Reviewer comment (Theory):
The second theorem is orthogonal to label smoothing and only depends on input perturbations. Considering that the second theorem needs a very limited situation (linear classifier), it would be better if it were more related to the proposed method. There is also a presumption that the theorems hold when $s$ is a constant function, i.e., vanilla label smoothing.

Author response:
Also, as you pointed out, the second theorem only depends on the perturbations but is invariant to the smoothing. When clarifying the reason, the theorem translates the perturbed input to the perturbed parameters but does not connect the smoothed label of the perturbed inputs to the smoothed label of the perturbed parameters. Yet, we remain the label smoothing part on the level of conjecture. Therefore, the remaining job is to add the part of label smoothing to fully justify that the smoothed label on the input side directly links to the smoothed label, which means flatter loss, on the parameters side. We have provided the additional proposition and corollary to rigorously formalize that SPIDER indeed promotes flatter minima. The additional statements are named **Corollary 1** and **Proposition 1**, and we have added them in Appendix D of our revised manuscript (please see the part following the proofs on Theorem 3 and 4. on page 26). If accepted, by considering their importance, we will integrate the contents into our final paper's flatness part in the main paper rather than in the Appendix. Briefly speaking, for the high-level understanding, **Corollary 1** is the final statement, and it directly connects the perturbation on the input side to the flatness of the parameter side, i.e., (when borrowing the notation in the revised manuscript) $\theta_{\gamma}^{*}$, which is trained by perturbation-based augmentation, yields **the flatter behavior of loss surface on the parameter space.**

Reviewer comment (Experiments):
The paper needs more extensive empirical justification. Theorem 1 shows that a scenario with perturbed input and monotonically decreasing label smoothing satisfies better thickness than vanilla training, and Theorem 2 shows that the solution space with the input perturbation is a subset of the "flat" solution space. However, these do not show that the proposed method is optimal for solving each task (thickness, flatness) or better than other methods. The paper should compare the proposed method with other regularization methods. Increasing boundary thickness or flatness may not be a special property of the proposed method, as prior work (e.g., Yang et al.) showed L1 regularization, L2 regularization, large learning rate, early stopping, and cutout can also increase boundary thickness.

Author response:
- **Extensive empirical justification required on theorems.**

  * *i) No theorems are guaranteeing the optimal performance w.r.t. boundary thickness and flat minima metrics.*

    A. You are indeed correct. However, we would like to gently point out that the challenge you mention is an ongoing one in the field of deep learning. To our knowledge, no works have given optimal solutions for finding flat minima, both theoretically and empirically. Regarding boundary thickness, while the authors of the work [4] have made assumptions to show that mixup is optimal solution to increase boundary thickness, the assumptions made are unrealistic for real-world applications (Proposition A.1.) Furthermore, experimental results in ours (Table 6) and the authors’ (Figure 2) hints us that mixup may provide large room for boundary thickness but is inferior to our baseline augmentations in robustness benchmarks. Striking the balance between theoretical and empirical results seems to be the crucial aspect in this machine learning field.

  * *ii) Theorems do not illustrate SPIDER’s superiority over other baseline methods or existing regularization methods (e.g. L1/L2 regularization, large learning rate, early stopping).*

    A. Without intending any provocation, our primary aim was to show SPIDER's applicability in various situations to improve model robustness, rather than proving its superiority over existing methods. However, for comparative purposes, we are currently training the same backbone network (DenseNet) deployed in the original boundary thickness paper and will release the results within 12 hours. (Note: We have updated additional experimental section E.2.3 with results comparing SPIDER against existing regularization methods. Please take a moment to review these new insights.)

Reviewer comment (Experiments):
In terms of empirical contribution, the proposed method always requires input perturbations. Although adding input perturbations is helpful theoretically, it may limit the usage of the method with recent state-of-the-art methods that employ many strong augmentations (e.g., RandAug, strong image view manipulations, or mixed data sample augmentations such as Mixup, CutMix). It is unclear whether the proposed method and recent strong augmentations can work together.

Author response:
- **Since SPIDER is input perturbation based algorithm, it may be hard to be combined with other augmentation methods such as RandAug, Mixup, and CutMix.**

  A. Usage of SPIDER with different augmentations is possible and is indeed desirable to further boost model robustness, as demonstrated in our experiments. When applying SPIDER, we have regarded the augmented image distribution as the true distribution.  Specifically, on top of the augmented images, we have added isotropic perturbations with corresponding label smoothings, i.e., SPIDER. When pointing out the case done in our work, PixMix handles two different images to combine a single augmented image, which follows a similar mixing process to Mixup and CutMix, and we successfully combine SPIDER with the paired-image-based augmentation. PixMix is applied to augment the original images and our method is applied on the augmented dataset.

  All the experimental results in the main table have been obtained with such a training scheme. We strongly believe that our method can be easily combined with other augmentations.

Reviewer comment (Experiments):
The experimental results are based on very few baselines and weak backbones. Weak networks have a large room for improvement, which might just be an issue of optimization. For example, "ResNet strikes back: An improved training procedure in timm" shows that a different optimization setting invariant to other improved techniques can boost performance by about +4%p for ResNet-50. In ImageNet classification, it is important to show generalizability for various backbones, rather than a specific backbone. The method should also be compared with other robustness-targeted methods such as Mixup, CutMix, and ANT, and with other data augmentation methods and label smoothing variants rather than focusing only on the proposed design choice.

Author response:
- **Weak backbone networks are exploited. A weak network has a large room for improvement, which could be a just issue of optimization.**

  A. Paraphrasing your concern, your question is that the reported values may not be optimal for each of the baselines, and SPIDER’s robustness gain may be a result of a certain optimization method. Respectfully, we have adhered to standard backbone models (Wide-ResNet, ResNet) and optimizations (SGD momentum with cosine learning rate decay) used in notable works (PixMix and AugMix). These are the most popular networks that have played as backbones of the immense number of deep learning algorithms, due to their considerable performance and the wide range of applicability. Also, we want to emphasize that while optimizing these methods might further improve robustness, all methods, including ours, are evaluated under the same conditions, ensuring no advantage in optimization techniques. At this moment, to the best of our understanding, it is hard to imagine an unexpected factor that probably leads us to misunderstand somewhat unintended gains as the true gains from SPIDER.

- **Diverse backbone networks should be deployed to demonstrate the generalizability of SPIDER.**

  A. We think it is a meaningful question to inquire about. It has been demonstrated in AugMix (Table 1) and DeepAugment (Table 4) that while precise statistics may vary, the tendency of augmentations on improving model robustness have not changed among varying model architectures. Echoing findings from the prior works, we are currently running SPIDER’s performance over different backbone architectures and updating the results on Appendix E.2 (please see pages 29-30 of the revised paper). Our method shows consistent gains over different backbones. Kindly note that due to our limited computational resources and limited time budget, we are not able to conduct experiments on ImageNet experiments with different backbone networks.

- **Comparisons with augmentation methods like Mixup, CutMix, or ANT are required.**

  A. Due to computational limits, we were unable to test all augmentations mentioned in our Related Work. We chose specific strategies to demonstrate SPIDER's effectiveness. Our decision to exclude Mixup, CutMix, and ANT is explained in more detail: Mixup and CutMix were less effective in the AugMix study (Table 1), and ANT required additional training for noise generators, leading to unstable learning curves, lengthy training, and less effectiveness compared to AugMix.

Reviewer comment (Theory):
There is a question about whether Theorem 1 holds when $s$ is a monotonically non-decreasing function, such as a constant function. This would make $\gamma_j$ constant. From the reviewer’s perspective, Theorem 1 may also hold for a constant function because constant functions are also monotonic, and the proofs only use the monotonic property. If true, this means Theorem 1 is also valid for input perturbation + vanilla label smoothing, which would weaken the justification for adaptive label smoothing since Theorem 2 is irrelevant to it.

Author response:
**Q1. Does theorem 1 hold for vanilla label smoothing + isotropic perturbation?**

A. Yes. It seems your question has two parts: 1. Why we chose an exponentially decaying function over a standard label smoothing function, and 2. Whether the standard smoothing function's similar robustness improvements diminish the significance of our contribution.

To answer the first part, we have chosen the exponentially decaying function simply because it marginally outperformed the standard label smoothing function in CIFAR common corruption benchmarks. In the following table, "exponential" indicates the SPIDER instantiation in the main paper, and "standard" indicates the version of SPIDER with standard label smoothing for function s(.).

| Benchmark | s(.)        | No Aug. | AugMix | DeepAug. | PixMix |
|-----------------|-------------|---------|--------|----------|--------|
| CIFAR10-C |exponential|14.37|9.17|11.18|8.35|
||standard|14.02|9.67|11.26| 9.05|
|CIFAR100-C| exponential |38.62|33.56|34.64|30.98|
||standard|39.07|34.45|34.60| 32.51|

For the second part, we would like to emphasize our key contribution is answering “how label smoothing should be combined with perturbations to broadly enhance model robustness,” not on exponentially decaying function itself. This nuance is conveyed through the titles of Algorithms 1 and 2. Precisely, we have demonstrated that isotropic perturbations encourage flatter minima, and any monotonically decreasing smoothing function w.r.t. the intensity of perturbations will increase boundary thickness, leading to improvements in model robustness.

Reviewer comment (Writing):
There appears to be a typo in Theorem 4 subscription. It may not be $\cup_{i\in [n]}$, but $\cup_{R_i\in [n]}$, considering the Appendix.

Author response:
**Q2. Notation mistake in $\cup_{i\in[n]}$?**

A. We have used $[n]$ as the index set notation indicating integers ranging from 1 to n. Therefore, $\cup_{i \in [n]} R_i$ can be interpreted as $R_1 \cup R_2 \cup \cdots R_n$.

Reviewer comment (Experiments):
It is mentioned in the manuscript that robustness is important in terms of different kinds of distributional shift including domain shift. I understand that the main goal is to mitigate perturbations, but according to the description of the corruption types in the appendix, it seems that some corruptions are quite strong and humanly perceptible. Therefore, the robustness with respect to domain shift also worth studying but such experiments are missing. It is also mentioned in section 3.3.1 that flat minima, which can be achieved by the proposed method according to both theoretical and empirical verifications, can improve robustness with respect to domain shift, but this is not empirically verified for the proposed method.

Author response:
**Q. Domain shift may be a good benchmark for demonstrating SPIDER’s effectiveness.**

A. Indeed, your observation is insightful. While our research has not yet explored SPIDER's performance against domain shift, we expect that SPIDER would likely improve robustness either as a standalone method or in conjunction with prior augmentation methods as in our experimental results on common corruption and adversarial robustness benchmarks. Despite we have attempted to conduct basic experiments using PACS dataset (which is one of the most widely used DG task benchmarks with four different domains), we were not able to conduct experiments successfully within the rebuttal period. Nonetheless, we still expect SPIDER to improve model robustness against domain shifts as well.

Reviewer comment (Experiments):
The proposed method can also improve robustness with respect to adversarial attacks. However, there is no comparison with other popular adversarial learning algorithms.

Author response:
**Q. Methods that directly aim for adversarial robustness is not included in the baseline methods.**

A. Your point is well-taken. Including adversarial training (AT) methods in our experiments would be ideal. We value your suggestion and have run additional experimental results on CIFAR100 regarding adversarially trained model in the below table. AT indicates a typical adversarial training with adding adversarial noise to the clean image. By combining with the adversarial training (AT), SPIDER shows consistent gains and a remarkable gain for the $L_\infty$ case, i.e., over +20.33\% gains. If accepted, we will conduct full adversarial training experiments on the main table's benchmarks and report the statistics in our final paper as your suggestion. 

|     Metrics     |       mCE      |       |      $L_2$     |           |   $L_\infty$   |    |   clean acc.   |      |
|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|
|     Method     |    original    |     +SPIDER    |    original    |     +SPIDER    |    original    |     +SPIDER    |    original    |     +SPIDER    |
| AT($L_\infty$) | 43.16$\pm$0.17 | 42.59$\pm$0.41 | 55.66$\pm$0.15 | 54.69$\pm$0.42 | 80.97$\pm$0.08 | 60.64$\pm$0.40 | 60.27$\pm$0.34 | 60.64$\pm$0.40 |

Reviewer comment (Experiments):
If I’m understanding correctly, the main baselines used in the experiments are pure image-based augmentation strategies. Class labels are not used in these baselines for augmentation. In my opinion, it is possible to incorporate augmented labels for these baseline methods as well. For example, it could be labels with reduced confidence for the ground-truth class for augmented images. It seems that comparing with these non-label augmentations looks unfair for the proposed label-smoothing method as it can leverage more information. Apart from this, ablation results in Table 6 do not show universal improvements under each component.

Author response:
**W3-1. Concerns about the fairness of evaluating SPIDER against baseline augmentations that do not manipulate labels.**

A. We would like to respectfully address this concern by highlighting two key points. Firstly, our primary goal is not to demonstrate the superiority of one method over another. Rather, our focus is on emphasizing the effectiveness of SPIDER in enhancing the overall robustness of models, whether it is used alone or in tandem with other augmentation techniques in diverse circumstances. Moreover, the assertion that SPIDER has an unfair advantage because it alters class labels, unlike other augmentation methods, in some sense might oversimplify the situation. For instance, SPIDER operates without any inductive bias related to the dataset, whereas other baseline methods may have inherent advantages. AugMix, for example, extensively employs image operations known for their beneficial effects on image classification. Similarly, PixMix utilizes additional images beyond the training set, and DeepAugment relies on a pretrained image-to-image model which deploys additional image dataset and network. These aspects could be viewed as providing an edge to these methods when compared with SPIDER, which solely depends on perturbations and label smoothing. Therefore, we humbly suggest that there is no inherent unfair advantage in favor of any particular augmentation approach in our evaluations.

**W3-2. Ablation results in Table 6 do not show universal improvements under each component.**

A. Indeed, this observation is accurate. SPIDER’s two components, which are label smoothing and perturbations, do not show incremental gains when we add one component to the other. Nonetheless, it is pivotal to underscore the collective significance of SPIDER as an integrated system, rather than focusing on its individual components. When standard label smoothing is applied in isolation, it enhances adversarial robustness but fails to make a discernible impact on common corruption robustness. In contrast, the use of isotropic perturbations alone yields only mediocre improvements in robustness, particularly when compared to the comprehensive efficacy of SPIDER or other augmentation methodologies. Consequently, we advocate that the holistic application of SPIDER is critical for achieving substantial and broad-based enhancements in model robustness, steering clear of incremental or minor improvements.

Reviewer comment (Experiments):
In the empirical verification of the flatness of SPIDER, I’m not sure how good the models are in Figure 2. What are the final accuracies of these models and are they the same models used to report the performance in Table 3?

Author response:
**Q1. What are the final accuracies of the models used for flatness analysis in Figure 2? Are they the same models used in Table 3?**

A. That is exactly correct. The models for the flatness analysis exactly show the final accuracies in Table 3.