Paper: LIMANS: Linear Model of the Adversarial Noise Space (ICLR-2024)

Reviewer comment (Experiments):
The proposed method, optimization objectives, relaxations, etc., are introduced as they are, without any small or large scale analyses or too many explanations.

Author response:
Since very similar objectives and relaxations are commonly used in the optimization community for different purposes, we have judged not necessary to include an in-depth analysis. We suggest adding one motivating line and citations to guide the reader. Nonetheless, the proposed method has been evaluated on small scale (CIFAR-10) and large scale (Imagenet) datasets for various architectures.

Reviewer comment (Evaluation):
The paper does not show the universality of the adversarial directions computed by LIMANS, other than through attack transferability. Each input point seems to get its own atom in the learned parameters, so there is no reason to believe that they are universal.

Author response:
The learned directions aim to span the space of the adversarial noise, which is related to the original data space and to some extent, the classifier. As illustrated in the figures above, a two-atom dictionary can construct all the adversarial examples, if they exist. This space is indeed universal. Moreover, through the observed transferability across different classifiers, this conclusion is further confirmed. When attacking a given input, the composition coefficient needs to be computed.

Reviewer comment (Experiments):
When studying transferability of adversarial examples, one should arguably look beyond universal perturbations and also consider black-box attacks. These are some of the attacks that offer improved transferability by default. While black-box attacks are listed as an item for future work, to me they are well within the scope of this paper. Boundary attack in particular seems relevant to the motivation presented around stability of decision boundaries per dataset.

Author response:
In black-box attacks, transfer-based methods play a crucial role. To attain strong performance in black-box scenarios, preliminary research on the transferability of a model is vital. This constitutes the primary objective of our current work. We analyze the properties of the learned space to subsequently apply it to enhance black-box attack strategies.

Reviewer comment (Experiments):
A significant practical limitation of the proposed method is that it needs access to at least a thousand clean samples in order to achieve performance remotely close to state-of-the-art attacks in white-box.

Author response:
You are right, in order to optimize LIMANS' dictionary, we need at least few clean samples and a source classifier to convergence towards optimized fooling parameters. However one could also argue that most of the black-box attack do impose the same constraint, also white-box attacks have their hyper-parameters that also need to be tuned. Therefore we believe the need of a training set not to be too limiting compared to other propositions.

Reviewer comment (Evaluation):
Another limitation that is not addressed in the paper is the cost of crafting adversarial examples with LIMANS after fitting the initial parameters. It seems parameters need to be fitted for each new data point.

Author response:
Thank you for your valuable question. The goal of our proposed method is not only to construct adversarial examples but also to uncover common mechanisms within a dataset for crafting adversarial noises.

Reviewer comment (Presentation):
The presentation and typography of the paper could be improved, please see some suggestions among minor points. The paper could also use additional proofreading.

Author response:
Thank you for your valuable insights and guidance in enhancing our paper.

Reviewer comment (Writing):
Consider using `\citep` when the cited references are not part of a sentence for proper citation formatting.

Author response:
Thank you for the advice; it has been incorporated into the modifications.

Reviewer comment (Writing):
Repeat sentence fragment in Contributions: "The associated optimization problem, allowing to learn".

Author response:
Thank you, we have corrected it.

Reviewer comment (Writing):
"the decision boundaries of different classifiers are closed" -> use "close" instead of "closed".

Author response:
Thank you, we have corrected it.

Reviewer comment (Presentation):
Some elements of the ICLR style are missing from the draft, like the document header. The header sizes also differ. The authors should ensure they use the ICLR style.

Author response:
Thank you, we have aligned it precisely with the ICLR style.

Reviewer comment (Reproducibility):
(Moderate) At least in the main text, this work omits a full description of how the learned "atoms" are calculated. Specifically, the text mentions that the atoms may not be suitable (i.e., $D$ not in $\mathcal{D}$) after Algorithm 2, but only says some post-processing is done to fix it. A description of this post-processing and discussion on how it retains the desired qualities of $D$ is necessary.

Author response:
This post-processing involves projecting onto the valid space of D, denoted as $\mathcal{D} ={D \,  ||D_j||_p=1, \forall j \in {1,\dots,M}}$. This projection is introduced in step 18 of Algorithm 2. It's noteworthy that we explicitly enforce the constraint on $Dv$ during training rather than on $D$ and $v$ separately. However, this does not impact the final results due to the post-processing step, where $d_i=d_i/||d_i||_p$ and $v_i = v_i*||d_i||_p$ are applied.

Reviewer comment (Writing):
(Moderate) This work has several typos or critical missing words that introduce ambiguity on what is meant and makes this paper more difficult to read (shapes of variables are inconsistent, some variables are undefined, incomplete sentences, missing critical words e.g., writing "1" instead of "Algorithm 1" or "Figure 1", missing periods in between sentences, missing letters (FUTUR -> FUTURE), no space before some sentences). More details in questions.

Author response:
We have carefully proofread the article to correct these issues.

Reviewer comment (Experiments):
(Moderate) The baselines for universal attacks seem low when comparing to values in prior work. For example, Figure 2 shows three different versions of UAP getting near 0% success rate, but the original works get near 90% success rate on the same dataset (CIFAR-10) and similar architectures (VGG11 vs VGG16).

Author response:
Please refer to our answer to Reviewer S66D. Additionally, we would like to mention that, for the sake of fairness, we selected the most common adversarial budgets, namely $||\epsilon||_2 < 0.5$ and $||\epsilon|| _\infty < 8$ for attacks on CIFAR-10. Of course, choosing larger threshold would permit to yield better fooling performance at the cost of more detectable attacks. We also would like to point that CIFAR-10 proves to be more challenging to attack than ImageNet, as evidenced by the RobustBench (https://robustbench.github.io/) performance metrics for AutoAttack.

Reviewer comment (Theory):
The study primarily relies on empirical evaluations to support its claims, which may leave room for a more in-depth theoretical exploration of the connection between universal and specific adversarial attacks.

Author response:
We acknowledge that our proposed modeling, and more broadly, related works on adversarial noise space modeling, reveal new research directions that necessitate theoretical guidance and guarantees. In this paper, we have opted for a more practical and empirical evaluation, intending to delve into its theoretical aspects in future works. Nevertheless, we share the following insights that we recommend including in the final version to further justify our approach.
The overall space of adversarial noise is equivalent to that of the classifier boundaries. However, due to the complexity of the deep neural network, direct analysis of the classifier seemed out of reach. Consequently, we endeavored to elucidate certain characteristics of the space (e.g., its dimension) through empirical experiments. Despite this, a straightforward analysis can be conducted, as illustrated in the figures above. In contrast, a universal attack seeks an average direction across the entire dataset to deceive as many examples as possible. However, with a single direction, it becomes impractical for inputs in case 2 to discern perturbations. LIMANS (case 2) bridges the gap between these two types of attacks. It dispenses with the need to seek a direction for each input but instead focuses on the space spanned by {$d_0$, $d_1$}.

Reviewer comment (Evaluation):
The results suggest a trade-off between the transferability of attacks to other models and their effectiveness on the source model, potentially requiring further investigation and trade-off analysis.

Author response:
In fact, the transferability of the space is proportional to its effectiveness on the source model. Typically, a classifier operates in higher dimensions, necessitating more atoms to construct the space. This makes it easier to transfer the space to a target model situated in a lower-dimensional space. For instance, the dictionary learned on Inception can be readily transferred to MobileNet. Conversely, transferability diminishes. Moreover, if the training sets of two models are not in the same space, transferability is significantly affected.

Reviewer comment (Evaluation):
The conflicting results observed on different datasets indicate that the proposed approach might not be universally applicable, and its limitations should be carefully considered in practical applications.

Author response:
We disagree and in fact strongly believe that the results obtained on the two datasets are consistent. Indeed, for complex datasets like ImageNet having $C=1000$ classes, learning the complete adversarial noise space is a challenge. Assuming the simplest case where the classifier is linear and its hyperplanes are independent, $C\times (C-1)/2 = (1000 \times 999)/2  = 499500$ atoms are needed to construct the space. In our experiments, we only learned a 100-dimensional subspace of the adversarial noise space, yet it allowed us to deceive half of the examples.  This highlights a practical limitation of our algorithm, as it requires substantial memory to learn a larger, or even complete, dictionary.

Reviewer comment (Experiments):
The paper builds upon the notion that specific attacks exhibit poor transferability. However, specific attacks (as known as adversarial examples) are shown to be transferable in prior literature [refA, B]. Thus, it is crucial for the paper to substantiate this assertion with empirical evidence, shedding light on the extent to which specific attacks struggle when transferred.

Author response:
[Ref B] demonstrates the existence of a transferable subspace around an input, inspiring transfer-based black-box attacks. Specific attack seeks the smallest perturbation that misleads the prediction (case 1). Typically, this tailored perturbation at the border of the adversarial space associated with a source model (in blue) seldom intersects with the adversarial space of another target model (in orange), limiting its transferability. We assess the transferability of the specific attack, such as AutoAttacks, revealing its limitations in the context of transfer. Our proposed method learned the adversarial space of a dataset not a single input, which makes it possible to craft an attack by a linear combinaison of the atoms under certain condition.

Reviewer comment (Theory):
The paper posits that the proximity of decision boundaries among multiple DNN classifiers trained on the same dataset suggests potential transferability in the adversarial noise space, which can be applicable for both specific and universal adversarial examples. However, the distinction between specific and universal adversarial attacks and why universals tend to excel in transferability remains somewhat unclear in the current version.

Author response:
As explained earlier, a specific attack identifies adversarial noise along the direction of the gradient, which is locally perpendicular to the classification boundaries. However, a universal attack searches for the average direction across the entire dataset. Comparing the averaged direction of two classifiers, their local gradient directions differ more.

Reviewer comment (Theory):
While the paper's formulation is intriguing, reinforcing it with theoretical contributions that establish the linkage and connection between specific and universal adversarial attacks in the adversarial space would enhance its significance.

Author response:
We agree. The analysis previously mentionned has been added to our paper to provide additionnal insights.

Reviewer comment (Experiments):
The performance of the proposed LIMANS relies on the choice of the source classifier.

Author response:
The reviewer is right. By design, the modeling of the adversarial noise space is optimized to fool a single source classifier. However, the results depicted in Tables 2 and 3 show great transferability performance to other classifiers as well. We believe that these results stem from the fact that the proposed framework, in-between specific and universal attacks, allows to learn adversarial patterns that are less dependent on the source classifier but more representative of the data diversity instead.

Reviewer comment (Evaluation):
It is not clear the extra computation overhead for learning the model from the source classifier, it would be better to compare the spending time across different attacks (for Simple-LIMANS and Regularized-LIMANS respectively).

Author response:
We agree with the reviewer that a time comparison of both attacks would be interesting. However, these two attacks play different roles. Simple-LIMANS acts as a decent, quick-to-run, and parameter-free attack, while Regularized-LIMANS might show stronger performances when coupled with the right hyper-parameters, at the burden of finding them.

Reviewer comment (Evaluation):
The results of the transferability performance of UAP look much lower than that in the existing literature, even lower than some specific attacks when attacking a standard-trained model. Why?

Author response:
In the original work on UAP, its performance is assessed on ImageNet under the condition that $\vert\vert \epsilon \vert\vert_\infty \leq 10$. In our experiments, we impose a more restrictive constraint that is $\vert\vert \epsilon \vert\vert_\infty \leq 4$. This results in a drop of performance, which is overcome by our proposed method, LIMANS. Nonetheless, our work aligns with the conclusions of the existing literature, demonstrating that universal attacks exhibit great universality at the expense of lower fooling performances on the source model.

Reviewer comment (Experiments):
Could you explain more about why you only use the validation set and not the training set?

Author response:
It is assumed that the training set is used to train the classifier, and in practice, it is usually not publicly accessible and we only have access to the validation data.

Reviewer comment (Experiments):
To my knowledge, you seem to miss some SOTA methods in the field of universal attack (e.g., [1]).

Author response:
You are right, due to extensive research in the field producing very fast new SOTA methods, this method has unfortunately been missed. We suggest to add the reference in the final version.

Reviewer comment (Experiments):
The simple-LIMANS seems to remove the tunable hyper-parameter lambda of regularized-LIMASNS, therefore also providing the results of regularized-LIMASNS would be desirable.

Author response:
Unfortunately, this is not feasible. A direction linked to an input is parallel to $\delta^{*}=\displaystyle \text{argmin}_{\{f(x+\delta)\neq f(x)\}} ~||\delta||_p$. The ensemble of optimal directions globally encompasses all these directions within a dataset. This cannot be achieved with a fixed threshold.

Reviewer comment (Writing):
A lot of typos/mistakes, e.g. "Table ??", "FOR FUTUR", "learn The associated"...

Author response:
It has been fixed, thank you for pointing them.