Paper: Adversarial Attacks as Near-Zero Eigenvalues in The Empirical Kernel of Neural Networks (ICLR-2024)

Reviewer comment (Theory):
As mentioned in Section 7, the theoretical result only covers limited neural network architecture, i.e., fully connected layers.

Author response:
Remark: As mentioned in Section 7, the theoretical result only covers limited neural network architecture, i.e., fully connected layers.

Answer: As pointed out in the response to Reviewer DyBR and as indicated also in Section 7, our results can be seamlessly extended to any architecture where the feature map exhibits Lipschitz continuity.

Reviewer comment (Experiments):
DeepFool algorithm is an old attack algorithm and cannot represent all the existing attacks (that are likely to be more powerful than DeepFool). The authors should perform similar experiments with other attack methods.

Author response:
Remark: DeepFool algorithm is an old attack algorithm and cannot represent all the existing attacks (that are likely to be more powerful than DeepFool). The authors should perform similar experiments with other attack methods.

Answer: As pointed out in Section 7, additional experiments would indeed be beneficial. Please note, however, that our theory is independent from the characteristics of the data as well as of the adversarial attack method. We chose DeepFool as a widely-known algorithm simply for illustrative purposes. The power of the attack method does not affect our theory: as long as one is able to find an adversarial example for the task at hand, it can be described by our framework.

Reviewer comment (Theory):
I cannot understand the reason for the assumption on the layer widths, i.e., all layers except the last layer have the same width $N$, because the proof mainly uses the empirical feature map that does not involve the intermediate layer output. Is this assumption necessary for the theorem? If so, how do you justify the assumption on the layer width?

Author response:
Question: I cannot understand the reason for the assumption on the layer widths, i.e., all layers except the last layer have the same width $N$, because the proof mainly uses the empirical feature map that does not involve the intermediate layer output. Is this assumption necessary for the theorem? If so, how do you justify the assumption on the layer width?

Answer: You are correct in noticing this unnecessary simplification, which was in place only to avoid
cumbersome notation. As already mentioned, our theory applies to any neural architecture where the feature
map exhibits Lipschitz continuity.

Reviewer comment (Theory):
My main objection with the presented argument is that I am not sold on the definition of an adversarial example in the paper. Specifically, I think that the limiting process of $\epsilon\to0$ in the definition is flawed. This limiting process effectively adds two equal samples to the training set with two different targets. I am under the impression that the vanishing of the eigenvalues is a consequence of this construction ($K(X,X)$ wouldn't be invertible) and is not associated with the robustness of the network.

Author response:
Remark: The limiting process of $\epsilon \to 0$ in the definition of an adversarial example seems flawed.

Answer: Please check our general remark to all reviewers and the corresponding modifications in the paper. Do not hesitate to ask should these modifications not adequately address your comments.

R2: The assumptions of $\epsilon \to 0$ "and $y - \mathcal{N} (x'))^2 \to \infty$ are not the same as assuming $K (X',X')$ i not invertible. There is no $x$ in the dataset that has been added to the set twice with different targets. Instead, the limit induces an example which is very close to an arbitrary test example but for which the predictions remains apart from one another (which is the very intuition of adversarial behavior). Adding this example adds one row and one column (the kernel evaluations between the new point and each of the training data points) to the matrix $K (X, X)$. It is perfectly fine to assume that $(\phi (X) \phi (X)^T)^{-1}$ is bounded, as we do not modify the training set.
 
As one can see in our proof of Theorem 1, the main insight of the proof is that introducing an adversarial example creates a divergence caused by the pseudo-inverse of $K (X', X')$ (due to eigenvalues becoming close to zero) so it is in a sense making the matrix $K (X', X')$ "almost not invertible".

Reviewer comment (Novelty):
Assuming that my understanding of the limiting process is correct, the paper has rediscovered the robustness-accuracy trade-off as depicted in [A] (this is not a critique exactly). Zhang et al. in [A] propose that adversarial examples and natural samples overlap in the input space and that is why we observe a trade-off between robustness and accuracy in adversarial training. I think the paper should at the very least mention [A]. An alternative to the proposal of Zhang et al. that might be relevant is [B].

Author response:
Remark: Assuming that my understanding of the limiting process is correct, the paper has rediscovered the robustness-accuracy trade-off as depicted in A (this is not a critique exactly). Zhang et al. in A propose that adversarial examples and natural samples overlap in the input space and that is why we observe a trade-off between robustness and accuracy in adversarial training. I think the paper should at the very least mention A. An alternative to the proposal of Zhang et al. that might be relevant is B.

Answer: Thank you for pointing out these references. These potential connections are indeed interesting and worth exploring. Please note, however, that the focus of our work differs from that by Zhang et al., as we give a mathematical justification as to  why adversarial examples are rarely sampled by chance in practice, even if they "overlap with natural samples".

Reviewer comment (Theory):
In the definition of adversarial examples, what is function $f(\epsilon)$? The definition does not specify function $f$ and only states the condition $\lim_{\epsilon\rightarrow 0}f(\epsilon) = \infty$, which would be problematic because for a fixed $\epsilon >0$ we can always find a function $f$ for observed data that makes the definition hold for a perturbed $x'$. Also, in the proofs of Theorem 1 and 2, do the authors determine the function $f$ based on the dataset $X, y$, or is the choice of $f$ independent of $X, y$?

Author response:
Questions: a) In the definition of adversarial examples, what is function $f (\epsilon)$? The definition does not specify function $f$ and only states the condition $\lim_{\epsilon \to 0} f (\epsilon) = \infty$, which would be problematic because for a fixed $\epsilon >0$ we can always find a function $f$ for observed data that makes the definition hold for a perturbed $\mathbf{x}'$. Also, in the proofs of Theorem 1,2 do the authors determine the function $f$ based on the dataset $\mathbf{X}, \mathbf{y}$ or is the choice of $f$ independent of $\mathbf{X}, \mathbf{y}$?

Answer:  Please check our general remark to all reviewers and the corresponding modifications in the paper. Do not hesitate to ask should these modifications not adequately address your comments.

Reviewer comment (Theory):
In Theorem 1, what is the definition of $P_{X'}$ (with the prime)? Is it the delta function on $x'$ or a continuous density function related to data distribution $P_X$?

Author response:
Question: In Theorem 1, what is the definition of $p_{\mathbf{X}'}$ (with the prime)? Is it the delta function on $\mathbf{x}'$ or a continuous density function related to data distribution $p_{\mathbf{X}'}$?

Answer: The definition of \emph{empirical probability distribution} for a dataset is provided in Section 2 at the end of the paragraph on empirical feature maps; it is essentially a summation of Dirac deltas. It's important to note that this definition is generic, and applicable to any design matrix (in particular to $\mathbf{X}')$. The notation $\mathbf{X}' = \mathbf{X} : \mathbf{x}'$ represents the concatenation of matrix $\mathbf{X}$ and vector $\mathbf{x}'$, as described also in Section 2.

Reviewer comment (Theory):
I wonder what the authors precisely mean when they say "In the limit $\epsilon\rightarrow 0$, .... " in Theorems 1 and 2. Does that mean there is an $\epsilon_0 >0$ independent of the choice of $x'$ such that the statement holds for every $\epsilon\le \epsilon_0$? (clarification on weakness 2)

Author response:
Question: I wonder what the authors precisely mean when they say "In the limit $\epsilon \to 0$, .... " in Theorems 1,2. Does that mean there is an $\epsilon_0 > 0$ independent of the choice of $\mathbf{x}'$ such that the statement holds for every $\epsilon \leq \epsilon_0$? 

Answer: It implies the existence of an $\epsilon_0$ such that the statement holds for all $0 <\epsilon \leq \epsilon_0$. The possible choices for $\mathbf{x}'$ depend on $\epsilon$, but not the other way around.

Reviewer comment (Theory):
What is the precise mathematical definition of "such that $|| x- x'||\le \epsilon$ for some example $(x^*,y)\sim p$"? If $p$ is a continuous distribution, then every point in $p$'s sample space has zero probability, so whether $(x^*,y)$ is in the support set of $p$ or not does not change its zero likelihood to be sampled from $p$. The theorem should explain this sentence.

Author response:
Question: What is the mathematical definition of "such that $|| \mathbf{x}^* - \mathbf{x}' || \leq \epsilon$ for some example $(\mathbf{x}^*, y^*) \sim p$.
"? If $p$ is a continuous distribution, then every point in $p$'s sample space has zero probability so whether $(\mathbf{x}^*, y^*)$ is in the support set of $p$ or not does not change its zero likelihood to be sampled from $p$.

Answer: Each individual point has probability zero but not a zero \emph{probability density}.
 For instance, in a Gaussian density, each real number has a probability of zero when sampled as a specific event; however, it's essential to note that each real number does have a non-zero probability density.
The statement
$(\mathbf{x}^*, y^*) \sim p$ 
simply means that point $(\mathbf{x}^*, y^*)$ is sampled according to the same joint probability density $p$ as that of the training data.

Reviewer comment (Theory):
The definition of adversarial examples appears a bit different than the usual definitions, relying on a local limiting assumption. Some work needs to be done to explain the relationship between this paper’s definition and the working definition used by the broader community.

Author response:
Remark: The definition of adversarial examples appears a bit different than the usual definitions, relying on a local limiting assumption. Some work needs to be done to explain the relationship between this paper’s definition and the working definition used of the broader community.

Answer:  Please check our general remark to all reviewers and the corresponding modifications in the paper. Do not hesitate to ask should these modifications not adequately address your comments.

Reviewer comment (Theory):
Are there geometric interpretations that your theory provides?

Author response:
Question: Are there geometric interpretations that your theory provides?

Answer: We briefly mentioned the geometric interpretation of our results in Section 6, paragraph 4. We indicated the relationship between the shift towards zero eigenvalues and the shift in the input feature space towards the direction of non-robust features.
The latter is a common observation in the literature on adversarial attacks.

Reviewer comment (Novelty):
Are the authors proposing that this method could serve to identify adversarial examples? If so, is there any contention with [1]?

Author response:
Question: Are the authors proposing that this method could serve to identify adversarial examples? If so, is there any contention with [1]?

Answer: In their current form, our results may not be directly applicable for detecting adversarial examples in practical settings. Theorem 1 relies on an abstract test example that the neural network generalizes well to, and such examples are not always accessible in real-world scenarios.

Reviewer comment (Theory):
The notation $(x^*, y^*) \sim p$ is strange from a probabilistic perspective. Isn’t there some probability mass on any point $x^*$? It also appears you are assuming no label noise (i.e. the labeling function $y(x)$ is deterministic).

Author response:
Question: The notation $(x^* , y^*) \sim p$  is strange from a probabilistic perspective. Isn’t there some probability mass on any point $(x^* , y^*)$? It also appears you are assuming no label noise (i.e. the labeling function $y(x)$ is deterministic).

Answer: This notation indicates that $(x^*, y^*)$ is a data point sampled from the same distribution as the training data, with joint probability density $p$. 
As stated by the reviewer, any point $(x^*, y^*)$ 
holds a probability mass, and it's precisely with respect to that probability measure that we showed the measure-zero property of adversarial examples.
Since $p (x^*, y^*) $denotes a joint probability density, noise can be easily incorporated in its definition.