Paper: Advancing the Adversarial Robustness of Neural Networks from the Data Perspective (ICLR-2024)

Reviewer comment (Novelty):
The most critical issue is the lack of comparison with prior works. It is well-known in the literature that not all data points are equally susceptible to adversarial attack, and this has motivated the design of various variants of adversarial training ([1,2,3,4,5,6]) to (adaptively) focus on a subset of training samples. The authors made a similar observation "it appears beneficial to emphasize the least robust elements during training", but seemed to be completely unaware of this line of research. Without proper discussion and comparison with prior works, it is hard to fairly position and evaluate this work in the vast literature.

Author response:
## ---*First, we address the mentioned weaknesses regarding the lack of comparison with prior works to delineate our contribution.*---  
---
There is indeed a vast corpus of literature regarding adversarial machine learning in general and adversarial training in particular. In order to (re)position our work with respect to the listed papers, we have revised our exposition to include the previous advancements and clarify which “gap” we intend to close with our contribution. More precisely, we focus on “why“ adversarial training shows a weaker effect in regions of non-robust data which cannot be explained entirely by small margins between differently labelled points.

To stress this point, we added to the updated version that the minimal $l_\infty$ distance between two non-robust CIFAR-10 images ($\approx0.2118$) is more than six times as large as the maximum perturbation radius ($\approx0.0314$) which, geometrically, leaves more than enough space for a robust decision boundary. However, it is not found consistently in these regions, as seen in Fig. 1.

Our primary goal and contribution in this work is to provide a new perspective on this phenomenon which builds on a deeper geometric concept, namely curvature (as perceived by a model during training). Whereas one part of this contribution is the theoretical framework connecting the robustness of data to the perceived curvature, our empirical results show that this perspective is meaningful in practice.
Regarding the adversarial training experiments, this view of perceived curvature can explain why focusing on a negligible number of non-robust points during training leads to an increased robust generalisation accuracy beyond the expected local improvements. Indeed, as we clarify in the updated version, the focus on the least robust points entails a better understanding of the pixel distances (the local improvement) and a better understanding of semantic distances (judging by the improved performance on unseen data). Together, this leads to a better understanding of curvature.

Whereas we do improve upon the current state-of-the-art approach from Wang et al. (2023) in a cost-neutral way, the main contribution of our work is a deeper understanding of representation learning. Our focus is not to provide a new method of adversarial training (which is beyond the scope of this work) but to motivate why developing methods based on this deeper geometric insight is promising in the first place.

Reviewer comment (Evaluation):
The clarity of the presentation and the quality of interpretation regarding the empirical observations could benefit from further refinement to enhance understanding.

Author response:
To clarify the presentation of empirical results, we rephrased and refined Section 4 to make it less convoluted and more accessible. We focused on more intuitive descriptions, especially for readers who are not necessarily familiar with the topics.

Reviewer comment (Evaluation):
The improvement in adversarial robustness is very marginal.

Author response:
In absolute terms, an improvement of 0.2 percentage points seems small but, as stated in the paper, amounts to the same increase of extending the training by 800 epochs, that is, one-third of the entire training (based on results from Wang et al. (2023)). Importantly, this benefit is achieved using identical computational resources.
However, the primary goal of these experiments is to show that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Presentation):
In Figure 1, clarification is needed on whether the dataset used is CIFAR10 and how the most/least robust elements are defined.

Author response:
The robustness of a data point is the minimal distance to a differently labelled data point. We added an explicit definition in Section 3 to avoid confusion. However, the absolute robustness (or, reciprocally, sensitivity) value is of secondary importance. It is the order of points when sorted according to their robustness (or sensitivity) values which lets us connect these concepts to the perceived curvature of a model. For this reason, we always refer to the "least and most robust points" instead of "points with a robustness of <x> or smaller/larger than <x>". To make the concept more accessible to readers, we changed the introduction by taking a more intuitive approach to explaining data robustness.

Reviewer comment (Presentation):
In Figure 1, the claim at the bottom of page 1 seems to suggest that by training for another 2000 epochs, the most robust training data (at epoch 400) become 1% more robust (at epoch 2400) and the least robust training data (at epoch 400) become 7.5% more robust (at epoch 2400). Confirmation is needed on whether this interpretation is correct and whether the same training data are being tracked.

Author response:
Correct.

Reviewer comment (Presentation):
In Figure 1, clarification is needed on whether "eval_adversarial_acc" corresponds to the validation accuracy.

Author response:
The "eval_adversarial_acc" traces the adversarial robustness against a PGD-40 attack with maximum perturbation radius 8/255 and step size 2/255 on the 1024 least and most robust elements, respectively. We added more information to the introduction to clarify this.
However, these subsets are also fixed parts of the training data in every epoch, allowing us to enforce a focus on these points during adversarial training. This example demonstrates that the effect of adversarial training is much weaker in non-robust regions compared to robust regions.
We modified the exposition of this example by (i) removing two plots and (ii) changing its explanation, where we chose a more natural way to introduce data robustness beforehand.

Reviewer comment (Experiments):
The term "validation set" appears to be used for a part of the training set, which is considered odd and requires clarification.

Author response:
To avoid confusion, we changed the nomenclature at several points.

Reviewer comment (Experiments):
In Figure 1, the statement "For CIFAR-100, we disregarded the 40 least robust elements because of distorted robustness values due to differently labelled duplicate images" is unclear. Clarification is needed on what "differently labelled duplicate images" are and what happens to the results if these are included.

Author response:
As we found out, the CIFAR-100 set contains identical images with different labels. For such data points, the sensitivity is not defined due to the vanishing quotient (comp. Equation 1). Based on the implementation, the sensitivity computation for these points would either return “Nan”s or extremely large values (as for us), both of which are inadmissible and distort the sensitivity distributions.

Reviewer comment (Reproducibility):
In Figure 2, there is understanding that the absolute sensitivity is computed using Eq1, but clarification is requested on how the relative sensitivity is computed.

Author response:
The relative sensitivity distributions are empirical probability distributions over the absolute sensitivity values (similar to histograms). However, since these plots confuse readers (and add barely any meaningful information), we removed them.

Reviewer comment (Evaluation):
In Figure 2’s analysis, it is stated that "The generated data, in particular, admit much shorter tails, indicating more robust elements overall." However, it appears that the curves of 1m-1 and 1m-2 are quite similar to cifar-10/100/100s, so clarification is needed on this interpretation.

Author response:
This was indeed a mistake in the description of the graph, which we corrected. We also added an explanation as to why pseudo-labeled data show (comparatively) different sensitivity values. From Carmon et al. (2019), we know that these data points are “mined” from the 80 Million Tiny Images (80M-TI) dataset (Torralba et al., 2008), which is the same source as for the CIFAR images (Krizhevsky, 2009). Similar to the generated data from Wang et al. (2023), the labels were assigned using the classifier predictions with the highest confidence. However, Carmon et al. (2019) removed near-duplicates of the CIFAR-10 test set from the 80M-TI set in a preliminary phase which can explain the lack of highly sensitive elements. More precisely, as sensitivity is a local concept (proven in Theorem 1), removing near-duplicates can decrease sensitivity values independent of the reference set.

Reviewer comment (Evaluation):
In Figure 2, clarification is needed on why the pseudo-labelled data have very different sensitivity.

Author response:
This was indeed a mistake in the description of the graph, which we corrected. We also added an explanation as to why pseudo-labeled data show (comparatively) different sensitivity values. From Carmon et al. (2019), we know that these data points are “mined” from the 80 Million Tiny Images (80M-TI) dataset (Torralba et al., 2008), which is the same source as for the CIFAR images (Krizhevsky, 2009). Similar to the generated data from Wang et al. (2023), the labels were assigned using the classifier predictions with the highest confidence. However, Carmon et al. (2019) removed near-duplicates of the CIFAR-10 test set from the 80M-TI set in a preliminary phase which can explain the lack of highly sensitive elements. More precisely, as sensitivity is a local concept (proven in Theorem 1), removing near-duplicates can decrease sensitivity values independent of the reference set.

Reviewer comment (Evaluation):
In Figure 3, the interpretation of the results is not clear. Clarification is requested on the differences in the mechanisms of MDS and RTE that lead to the inverted pattern observed.

Author response:
Whereas MDS (Kruskal, 1964) visualises similarities based on pairwise differences of data points, RTE (Geurts et al., 2006; Moosmann et al., 2006) is based on fitting a random forest to the data points, whose representations are then determined by the leaves they end up in. Naturally, nearby points are more likely to fall into the same leaf. For the details, we refer to the corresponding papers. The inverted patterns are explained by the different implementations. The critical aspect is whether points of the same colour are grouped together, which expresses similarity in both cases.
To clarify this aspect, we added more explanations regarding the functionality of MDS and RTE (due to space constraints, we moved the RTE plots to the Appendix).

Reviewer comment (Writing):
In Figure 3, it would be helpful for interpretation to provide a brief explanation of what MDS and RTE are and why they are used.

Author response:
Whereas MDS (Kruskal, 1964) visualises similarities based on pairwise differences of data points, RTE (Geurts et al., 2006; Moosmann et al., 2006) is based on fitting a random forest to the data points, whose representations are then determined by the leaves they end up in. Naturally, nearby points are more likely to fall into the same leaf. For the details, we refer to the corresponding papers. The inverted patterns are explained by the different implementations. The critical aspect is whether points of the same colour are grouped together, which expresses similarity in both cases.
To clarify this aspect, we added more explanations regarding the functionality of MDS and RTE (due to space constraints, we moved the RTE plots to the Appendix).

Reviewer comment (Presentation):
For all figures, it is suggested to use subcaptions to increase the clarity of the results.

Author response:
We restructured the graphics and divided them into sub-graphics with individual captions.

Reviewer comment (Presentation):
In the exploration experiments setup column of Table 2, clarification is needed on whether minus v ("-v") means removing v from the training set, and whether "(v)" means all the validation accuracy is based on v.

Author response:
Correct.

Reviewer comment (Reproducibility):
In the exploration experiments, clarification is needed on whether the most sensitive data points are computed based on another pre-trained model.

Author response:
No. Sensitivity information is based entirely on the feature and label information of the data (comp. Equation 1). In other words, the most and least sensitive points are defined in a model-agnostic way, allowing us to predict which regions a model perceives as areas of high or low curvature -before- a model is trained.

Reviewer comment (Novelty):
The basic observation made by the paper regarding the relationship between robustness, sensitivity, and sample importance during training is interesting, but well-known. To strengthen the contribution and significance of the work, the authors should clarify the contribution of their analysis in the context of the previous work, or demonstrate some actionable insights—e.g., an algorithm that exhibits superior adversarial robustness relative to existing techniques.

Author response:
To explain and delineate our contribution from existing work and to show the significance of the proposed link between data robustness and the perceived curvature of a model, we reframed our exposition in the updated version.

As mentioned by the reviewer, it is known that nearby differently labelled points (= non-robust points) enforce small margins, which naturally impair adversarial training in such regions (as showcased in the first example). However, while "less space" may explain the weaker effect of adversarial training in such non-robust regions, it fails as an explanation if "less space" is "still more than enough space". To strengthen our argument, we included the following point: the minimal $l_\infty$ distance between two non-robust CIFAR-10 images ($\approx0.2118$) is more than six times as large as the maximum perturbation radius ($\approx0.0314$) which, geometrically, leaves more than enough space for a robust decision boundary. Nevertheless, a robust decision boundary is not found during training for a significant portion of the non-robust points (comp. Fig. 1). As our primary contribution in this work, we provide a new perspective for this phenomenon to explain why these non-robust points are "difficult" for any model and why adversarial training may not lead to a robust decision boundary, even if we know it exists.

Whereas this motivates new adversarial training methods, the development of such a method is beyond the scope of this work. Instead, we provide theoretical and empirical evidence to show that this curvature-based perspective is meaningful in practice and can motivate new methods of adversarial training in the future. We made significant changes in the introduction and Section 4 to clarify this point.

Reviewer comment (Novelty):
There is some missing existing work that should be cited that explores the emphasis of certain vulnerable examples in the training set to enhance clean and robust test-set performance. Examples include reweighting methods such as [1, 2], subsampling methods such as [3], and other works not listed (e.g., on applications of diffusion models to the adversarial robustness context).  
[1] Zhang et al., Geometry-aware Instance-reweighted Adversarial Training, ICLR 2021  
[2] Wang et al., Probabilistic Margins for Instance Reweighting in Adversarial Training, NeurIPS 2021  
[3] Zhang et al., Attacks Which Do Not Kill Training Make Adversarial Learning Stronger, ICML 2020

Author response:
We included the listed works to explain what phenomenon we focus on in this work (which cannot be explained by traditional margin-based optimisation strategies; see explanation above). We highlight that our primary contribution is a new and deeper geometry-based perspective to explain why adversarial training shows a weaker effect in certain regions. We clarified this point in the updated version.

Reviewer comment (Novelty):
Could the authors clarify their contribution in the context of existing methods/analysis (e.g., by providing some explanation for the efficacy of existing methods to enhance robustness)?

Author response:
Our primary contribution is an explanation for the weaker effect of adversarial training in non-robust regions, which small margins between differently labelled points cannot entirely account for. This explanation, building on connections between a model’s perception of curvature and the robustness of data, can motivate new methods of adversarial training in the future. In this work, we aim to provide theoretical and empirical evidence to demonstrate that this connection exists and is meaningful in addition to the development of new methods (which is beyond the scope of this paper).

Reviewer comment (Writing):
One claim is that _"diffusion model connects regions of non-robust data to more prominent semantic changes, which we take as the model accounting for a more significant perceived curvature."_ Can this be made more precise?

Author response:
We rephrased the explanation in the paper to clarify what we mean. In essence, we use the diffusion model to generate new images from the most and least robust regions. As we demonstrate, the semantic differences between the original and generated images are much smaller for the most robust elements than the least robust ones. Hence, the semantic distance (or manifold distance $\tilde{d}$) is larger for the latter. By comparing the corresponding $l_2$ and $l_\infty$ distances between original images and clones, we notice that the converse holds for the pixel distance. More precisely, the pixel distance ($d$) between the original and generated images is more significant for the most robust elements compared to the least robust elements.

By comparing the semantic and pixel distances in the form of the Finsler-Haantjes curvature, we see that the diffusion model connects regions of lower/higher curvature with regions of robust/non-robust data. Here, we refer to regions instead of single points, as curvature and robustness (or, reciprocally, sensitivity) are local concepts (comp. Theorem 1).

Reviewer comment (Writing):
The paper and overall presentation is very difficult to follow. Although the authors seem to know very well their topic, the communication is lacking and a reader not in that specific field gets lost quite quickly.

Author response:
In order to make the content of our paper more accessible, we have rewritten the introduction. The primary goal was (i) to make abstract concepts such as data robustness and the data manifold's curvature more accessible and (ii) to clarify what our contribution in this context is. We removed two plots in Figure 1 and added a schematic to clarify what we mean by the different distances in feature space and along the data manifold (as a preliminary for curvature).

To show why perceived curvature is meaningful, we outlined how it can explain the weaker performance of adversarial training in regions of non-robust data where robust decision boundaries exist but are not found during training.

To clarify how our experimental evidence supports our theoretic view (of a model perceiving curvature differently in robust and non-robust regions), we have also rewritten Section 4. One goal was to convey an intuition for why we observe a diffusion model generating semantically very different images in the least robust regions but not in the most robust regions. Overall, we explained the experimental setups in a less convoluted way to avoid confusion.

Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of **Wang et al. (2023)**. However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Theory):
The notion of curvature on the manifold is really unclear to me and not very well explained in the paper. But it appears in the end we are looking at distance between samples, the notion of curvature is there to support theoretical developments that are not directly translated in practice.

Author response:
In order to make the content of our paper more accessible, we have rewritten the introduction. The primary goal was (i) to make abstract concepts such as data robustness and the data manifold's curvature more accessible and (ii) to clarify what our contribution in this context is. We removed two plots in Figure 1 and added a schematic to clarify what we mean by the different distances in feature space and along the data manifold (as a preliminary for curvature).

To show why perceived curvature is meaningful, we outlined how it can explain the weaker performance of adversarial training in regions of non-robust data where robust decision boundaries exist but are not found during training.

To clarify how our experimental evidence supports our theoretic view (of a model perceiving curvature differently in robust and non-robust regions), we have also rewritten Section 4. One goal was to convey an intuition for why we observe a diffusion model generating semantically very different images in the least robust regions but not in the most robust regions. Overall, we explained the experimental setups in a less convoluted way to avoid confusion.

Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of **Wang et al. (2023)**. However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Reproducibility):
The technical aspects of the experiments section are not very clear nor clearly explained. I guess that the reader should look at some of the referenced papers like Karras et al. (2022) and Wang et al. (2023), but still I would like to get more background and specific details to better understand what is done in the experiments. It is quite unclear to me that the details provided would make results reproducibility easy.

Author response:
In order to make the content of our paper more accessible, we have rewritten the introduction. The primary goal was (i) to make abstract concepts such as data robustness and the data manifold's curvature more accessible and (ii) to clarify what our contribution in this context is. We removed two plots in Figure 1 and added a schematic to clarify what we mean by the different distances in feature space and along the data manifold (as a preliminary for curvature).

To show why perceived curvature is meaningful, we outlined how it can explain the weaker performance of adversarial training in regions of non-robust data where robust decision boundaries exist but are not found during training.

To clarify how our experimental evidence supports our theoretic view (of a model perceiving curvature differently in robust and non-robust regions), we have also rewritten Section 4. One goal was to convey an intuition for why we observe a diffusion model generating semantically very different images in the least robust regions but not in the most robust regions. Overall, we explained the experimental setups in a less convoluted way to avoid confusion.

Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of **Wang et al. (2023)**. However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Evaluation):
It is difficult to figure out what exactly the experimental results are providing as support to the conclusion. The differences in Table 2 between the results is very small, and as such not very convincing that the proposal is of any meaningful effects for improving robustness.

Author response:
Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of **Wang et al. (2023)**. However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Evaluation):
Overall, the experiments are not very well explained and presented and the results are very difficult to interpret. I have a hard time making sense of all this.

Author response:
In order to make the content of our paper more accessible, we have rewritten the introduction. The primary goal was (i) to make abstract concepts such as data robustness and the data manifold's curvature more accessible and (ii) to clarify what our contribution in this context is. We removed two plots in Figure 1 and added a schematic to clarify what we mean by the different distances in feature space and along the data manifold (as a preliminary for curvature).

To show why perceived curvature is meaningful, we outlined how it can explain the weaker performance of adversarial training in regions of non-robust data where robust decision boundaries exist but are not found during training.

To clarify how our experimental evidence supports our theoretic view (of a model perceiving curvature differently in robust and non-robust regions), we have also rewritten Section 4. One goal was to convey an intuition for why we observe a diffusion model generating semantically very different images in the least robust regions but not in the most robust regions. Overall, we explained the experimental setups in a less convoluted way to avoid confusion.

Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of **Wang et al. (2023)**. However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Reviewer comment (Theory):
Looking at equation 1, if we assume that $d(p_i,p_j)$ is an Euclidean distance and that $\|y(p_i)-y(p_j)\|$ is basically equal to zero or one when using $y(p)$ as one hot vector over the classes, it means that in practice, the proposal consists of looking 1/distance to the nearest sample from a different class from the current one. Is this correct? That’s what was used for the experiments?

Author response:
Yes. This “sensitivity” information was used in the experiments. However, as shown theoretically, we are not interested in the absolute sensitivity value but the value compared to the entire sensitivity spectrum. In this way, sensitivity has only meaning in relative terms. This is why we always refer to the “most and least sensitive or robust points” instead of the “points with a sensitivity above/below x”. Ultimately, this allows us to connect the primitive measure of sensitivity to the elaborate concept of curvature.