Paper: Theoretical Understanding of Learning from Adversarial Perturbations (ICLR-2024)

Reviewer comment (Theory):
In Section 4.1, the authors generate the adversarial examples according to the linear model. I understand the decision boundaries of the linear model $f^{bdy}(z)$ and the one-hidden-layer neural network are the same (Theorem 3.3). However, I don't think we can directly replace the neural work with the linear model to generate adversarial examples since adversarial examples are generated by optimizing the loss, which is different. Can the authors provide more theoretical explanations for this?

Author response:
In Appendix D.1, we discuss the perturbations generated by optimizing the exponential loss on the one-hidden-layer neural network $f$ itself rather than the linear model $f^\mathrm{bdy}$. The results are consistent with those in the main text, i.e., those for the perturbations targeting the linear model. Please refer to it.

Reviewer comment (Novelty):
Although the authors claim that they focus on one-hidden-layer neural networks, they make some approximations and actually focus on linear models. So the contribution of this paper is reduced.

Author response:
In Appendix D.1, we discuss the perturbations generated by optimizing the exponential loss on the one-hidden-layer neural network $f$ itself rather than the linear model $f^\mathrm{bdy}$. The results are consistent with those in the main text, i.e., those for the perturbations targeting the linear model. Please refer to it.

Reviewer comment (Theory):
In this work, the authors show that the model trained on adversarial perturbations with incorrect labels (i.e., $y^{adv}$) can generate well on the clean data. But as we know, traditional adversarial training is based on adversarial examples with correct labels (i.e., $y^{natural}$). So, why do these two methods both generate well on the clean data?

Author response:
**Our research focus is NOT adversarial training. Please kindly refer to the general comment.**

Reviewer comment (Evaluation):
I would like to know the adversarial accuracy of the method in your experiments.

Author response:
**Our research focus is NOT adversarial training. Please kindly refer to the general comment.**

Reviewer comment (Theory):
In Eq. (2), does $x_n$ refer to the training data $x_n$? It seems that $x_n$ here is new test data that is different from the training data $x_n$. Otherwise, the gradient calculation in Eq. (2) should be reconsidered. Please check this out.

Author response:
Yes, in Eq. (2), $x_n$ denotes the training data. We would appreciate more details on your interpretation of $x_n$ as new test data.

Reviewer comment (Theory):
The assumption in Theorem 4.2 is somewhat strong: $|\sum\lambda_n y_n <x_n,z>|=\Theta(g(N,d))$ if $\sum\lambda_n  |<x_n,z>|=\Theta(g(N,d))$. It is intuitive that the first term is much smaller than the second term. What would your theorem be without this assumption?

Author response:
We acknowledge that this assumption introduced in the natural data scenario (cf. Definition 3.2) is not always reasonable. However, to determine the growth rate of $T_2$, i.e., the second term in Eq. (4), which includes the indeterminable $y_n$, certain assumptions are necessary. While this assumption can be relaxed, for example, to $|\sum\lambda_ny_n\langle x_n,z\rangle|=\Theta(1)$, it inevitably weakens the conclusion of Theorem 4.2.

To circumvent these issues, we employed the noise data scenario. This approach allows us to validate the success of learning from perturbations without relying on such unreasonable assumptions (cf. Theorem 4.4). This feasibility stems from the fact that in the noise data scenario the base data (i.e., uniform noises) have well-defined probabilistic characteristics. This differs from the natural data scenario, where the probabilistic nature of the base data (i.e., natural data) and the corresponding labels cannot be defined.

Reviewer comment (Writing):
In Section 5.1, I have some confusion about some statements. For example, in Fig 1, the authors mentioned standard data and noisy data. But in their setting, there are two kinds of noisy data: the dataset is generated randomly and can be viewed as some noises; the adversarial perturbations are superposed on random noises. I am confused as to what "standard data" and "noise" refer to. I strongly recommend the authors make more clear clarifications in the paper.

Author response:
The reviwer is correct; "standard data" refers to images that might appear as noise to the human eye in this section. We will enhance the clarity of our experimental setup and the nature of the images in the revised version.

Reviewer comment (Evaluation):
As the authors claim in the experiments, there are some counterexamples. I think more discussions about these counterexamples are needed. Is it because the effect of learning from mislabeled natural data dominates the effect of learning from perturbation?

Author response:
Yes, following Theorems 4.1 and 4.3, the dominance of learning effects from mislabeled natural data over learning effects from perturbations is a plausible explanation. However, the theoretical evaluation of this effect is challenging in datasets such as MNIST, Fashion-MNIST, and CIFAR-10, where the data distribution cannot be quantitatively assessed.

Reviewer comment (Theory):
I’m not exactly following the motivation of this work. Normally when people consider adversarial training to gain robustness, at each iteration the adversarial training examples are generated based on the current model weight, yet in this paper, from the definition 3.2, it seems that the adversarial examples of training samples are generated beforehand and independent of the current model weight, and are fixed during the latter training procedure. Therefore it’s unclear to me whether the perturbation considered in this paper is actually adversarial or not. The author claims the motivation comes from Ilyas et al. 2019, whereas in that paper there’s also a math formulation of generating the designed adversarial perturbation, which seems different from the procedure described in definition 3.2.

Author response:
**First, we would like to clarify that our research is NOT related to adversarial training. Please kindly refer to the general comment.**

### Misinterpretation
> when people consider adversarial training ...

**Our research focus is NOT adversarial training.**

> the motivation comes from Ilyas et a. 2019, ..., which seems different from definition 3.2

The reviewer may refer to the "Robust training" in section 2 of (Ilyas et al.), which is not the focus of our study. Learning from perturbations is described in Section 3.2 of (Ilyas et al.).

Reviewer comment (Theory):
It seems this paper’s theoretical results largely build on the result from Frei et al. 2023. Yet the discussion of the existing result and the corollary is limited. It’s unclear to me why corollary 3.4 can be directly derived from Theorem 3.3. Theorem 3.3 requires clean training data to be nearly orthogonal, then for corollary 3.4 you should also prove that the adversarial example is almost orthogonal. Moreover, the adversarial perturbation considered depends on the decision boundary, but it’s unclear to me whether the coefficient $\lambda$ in the decision boundary is computable. The network only considers Leaky ReLU, yet it’s unclear why it cannot generalize to other activation functions and what’s the technical difficulty.

Author response:
### Theoretical framework (Theorem 3.3 and Corollary 3.4)
> the discussion of the existing result and the corollary is limited

The discussion has been summarized in Appendix A.

> why corollary 3.4 can be directly derived from Theorem 3.3

Corollary 3.4 is the special case of Theorem 3.3 for $\\{(x^\mathrm{adv}\_n,y^\mathrm{adv}_n)\\}^{N^\mathrm{adv}}\_{n=1}$. In other words, we replace $\\{(x\_n,y\_n)\\}^N\_{n=1}$ with $\\{(x^\mathrm{adv}\_n,y\_n^\mathrm{adv})\\}^{N^\mathrm{adv}}\_{n=1}$. This change naturally leads to a modification in definitions, such as $R\_\mathrm{max}:=\max\_n\\|x\_n\\|$ becoming $R^\mathrm{adv}\_\mathrm{max}:=\max\_n\\|x^\mathrm{adv}\_n\\|$. Essentially, Corollary 3.4 is Theorem 3.3 with "adv" added to the variables.

> for corollary 3.4 you should prove that the adversarial example is almost orthogonal.
> 
> what do Eq. 3 and Eq. 6 mean?

The orthogonality condition of adversarial examples is $\gamma^3{R^\mathrm{adv}\_\mathrm{min}}^4/(3N^\mathrm{adv}{R^\mathrm{adv}\_\mathrm{max}}^2)\geq p^\mathrm{adv}\_\mathrm{max}$ in corollary 3.4. This condition depends on the definition of perturbations and the base images (natural images or uniform noises). For geometry-inspired perturbations (cf. Eq. (2)) on natural images, this condition is transformed into Eq. (3), and for those on uniform noises, Eq. (6).

> it’s unclear whether the coefficient $\lambda$ in the decision boundary is computable

(Theoretical perspective) $\lambda$ is uniquely determined but is not computable (cf. Appendix A). However, the lower and upper bounds are computable, which is enough to derive our theorems.

(Practical perspective) The perturbation Eq. (2) is not computable in practice, as $\lambda$ and the boundary are not computable. Note that this perturbation form was chosen only because it represents the simple results. Moreover, in Appendix D.1, we derived consistent results with those in the main text for the perturbations using the network $f$ directly, which are based on the computable network output and thus practical to implement.

> why cannot generalize to other activations

This is because Theorem 3.3 is applicable only to Leaky ReLU (cf. the proof of Theorem 3.2 in (Frei et al., 2023)).

Reviewer comment (Theory):
The statement of each theorem is not self-contented. If $R_{min}, R_{max}, p_{max}$ are used all over the place, then they should be introduced in the notation part. The explanation of each theorem and corollary is not clear. Why do we require equation (3)? Any intuitive explanation or simply one requirement for the proof to go through? When N goes to infinity, equation (3) is saying $p_{max}$ is negative, which is kind of counterintuitive. Similarly, what does equation (6) hold mean intuitively?

Author response:
> what do Eq. 3 and Eq. 6 mean?

The orthogonality condition of adversarial examples is $\gamma^3{R^\mathrm{adv}\_\mathrm{min}}^4/(3N^\mathrm{adv}{R^\mathrm{adv}\_\mathrm{max}}^2)\geq p^\mathrm{adv}\_\mathrm{max}$ in corollary 3.4. This condition depends on the definition of perturbations and the base images (natural images or uniform noises). For geometry-inspired perturbations (cf. Eq. (2)) on natural images, this condition is transformed into Eq. (3), and for those on uniform noises, Eq. (6).

Reviewer comment (Theory):
Can the author provide any intuition as to why mislabeled data provide the same classifier direction as clean data? Is it mainly because of the orthogonality of training data so that even if the label is noisy the data is still orthogonal?

Author response:
> why mislabeled data provide the same classifier direction as clean data? because the orthogonality of training data so that even if the label is noisy the data is still orthogonal?

It seems this question arises from a misunderstanding of our experimental procedure as adversarial training. For the explanation of the success of learning from mislabeled data, please refer to the general comment.

This is because classifiers learn from mislabeled adversarial examples, not just mislabeled data. Following the hypothesis of Ilyas et al., adversarial perturbations contain invisible data features to humans. While learning on mislabeled adversarial examples seems nonsensical to us, classifiers rely on these invisible but generalizable features and can construct the same decision boundaries as those of standard classifiers.

Reviewer comment (Theory):
It’s unclear from a theoretical perspective in Appendix G why the standard decision boundary only depends on non-robust features as shown in equation A157.

Author response:
> why the standard boundary only depends on non-robust features in Eq. A157

Under the assumption that standard classifiers focus exclusively on non-robust features, we consider that training classifiers on $\\{(x\_n,y\_n)\\}\_n$ is equivalent to training on $\\{(x^\mathrm{non}\_n,y\_n)\\}\_n$. Thus, by applying Theorem 3.3 to $\\{(x^\mathrm{non}\_n,y\_n)\\}\_n$, we derived Eq. (A157).

Reviewer comment (Novelty):
It is unclear what the motivation for studying the uniform perturbation model is.

Author response:
We consider the noise data scenario (cf. Definition 3.2(b)) with two motivations.

First, this is to prevent unintentional leakage of useful features. In the natural data scenario (cf. Definition 3.2(a)), we labeled adversarial examples with incorrect classes. However, there remains the possibility that these images contain features related to the incorrectly assigned labels. For example, a frog image labeled as a horse may contain horses in the background. As long as we consider natural images, it is inevitable that unintentional alignment between images and labels will occur, causing classifiers to generalize from leaked features rather than perturbations. In contrast, uniform noises lack natural object features, allowing us to explore the purer dynamics of learning from perturbations.

Second, this is to validate the success of learning from perturbations under weaker conditions than in the natural data scenario. In the natural data scenario, the probabilistic nature of the base data (i.e., natural data) cannot be defined. For example, the distribution of frog images cannot be precisely determined. This uncertainty requires us to impose the assumption $|\sum\lambda_ny_n\langle x_n,z\rangle|=\Theta(g(N,d))$ if $\sum\lambda_n|\langle x_n,z\rangle|=\Theta(g(N,d))$. In the noise data scenario, thanks to the well-defined probabilistic properties of the base data (i.e., uniform noises), we can justify the success of perturbation learning under more reasonable assumptions (cf. Theorem 4.4).

Reviewer comment (Theory):
Overall, the investigated model seems to be quite simple and theoretical assumptions restrictive (see also questions below).

Author response:
If the reviewer regard "attack model" as the form of perturbation, we have explored three additional forms of perturbation in Appendix D that were not discussed in the main text in detail. Please refer to them.

The constraints of using a one-hidden-layer neural network, orthogonal training, and the resulting $\epsilon\in O(d/N)$ are based on the theoretical framework, Theorem 3.3 (Frei et al., 2023). To remove these constraints, we would need to extend Theorem 3.3 in the context of the implicit bias of gradient descent rather than learning from adversarial perturbations. Previous research [1,2] has demonstrated the implicit bias of unbiased deep ReLU networks under more relaxed dataset assumptions. Thus, we believe that these results could serve as a starting point to justify learning from perturbations under more relaxed assumptions.

[1] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In ICLR. 2020.

[2] Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In NeurIPS. 2020.

Reviewer comment (Writing):
The text is rather densely written (e.g. subsection 4.2 and 4.3), which makes understanding main statements and insights rather difficult (e.g. Theorem 4.2).

Author response:
We acknowledge that due to space constraints, some sections of our paper, particularly the explanation following Theorem 4.1, have become overly dense. We plan to address this by eliminating some redundant expressions.

Reviewer comment (Theory):
The paper expands on established settings on 'learning from adversarial perturbations' (Definition 1.1. and Definition 3.2) by also discussing perturbations on uniform noises. In this 'perturbation on uniform noise' setting, i.e. $x_n^{adv} = X_n + \eta_n$, with $X_n$ being uniform noise instead of a natural data point. What is the motivation for studying this setting? Why not considering $x^{adv}_n = x_n + X_n + \eta_n$ (with $x_n$ the actual data points), which would be a more practical setting?

Author response:
We consider the noise data scenario (cf. Definition 3.2(b)) with two motivations.

First, this is to prevent unintentional leakage of useful features. In the natural data scenario (cf. Definition 3.2(a)), we labeled adversarial examples with incorrect classes. However, there remains the possibility that these images contain features related to the incorrectly assigned labels. For example, a frog image labeled as a horse may contain horses in the background. As long as we consider natural images, it is inevitable that unintentional alignment between images and labels will occur, causing classifiers to generalize from leaked features rather than perturbations. In contrast, uniform noises lack natural object features, allowing us to explore the purer dynamics of learning from perturbations.

Second, this is to validate the success of learning from perturbations under weaker conditions than in the natural data scenario. In the natural data scenario, the probabilistic nature of the base data (i.e., natural data) cannot be defined. For example, the distribution of frog images cannot be precisely determined. This uncertainty requires us to impose the assumption $|\sum\lambda_ny_n\langle x_n,z\rangle|=\Theta(g(N,d))$ if $\sum\lambda_n|\langle x_n,z\rangle|=\Theta(g(N,d))$. In the noise data scenario, thanks to the well-defined probabilistic properties of the base data (i.e., uniform noises), we can justify the success of perturbation learning under more reasonable assumptions (cf. Theorem 4.4).

First, it is difficult to define what is "practical" in the context of learning from perturbations. This experiment aims to understand the origins of adversarial examples and the information contained in adversarial perturbations. From this perspective, both the natural and noise data scenarios align with the objectives, and we cannot determine the superiority. Similarly, we cannot determine which is better, $x^\mathrm{adv}_n=X_n+\eta_n$ or $x^\mathrm{adv}_n=x_n+X_n+\eta_n$. Please also refer to the general comment.

From the perspective of a tractable theoretical analysis, the scenario where perturbations are on solely uniform noises can provide a more theoretically manageable formulation of learning from perturbations, as described in the comment above. Thus, we choosed $x^\mathrm{adv}_n=X_n+\eta_n$ rather than $x^\mathrm{adv}_n=x_n+X_n+\eta_n$.