Paper: Efficient Diversified Attack: Multiple Diversification Strategies Lead to the Efficient Adversarial Attacks (ICLR-2024)

Reviewer comment (Experiments):
Previous works are concerned with the transferability of adversarial attacks. A transferability evaluation is missing.

Author response:
We have tested the transferability of EDA for several models.  
The results show that EDA can generate more transferable adversarial examples than A$^3$ and other baselines, regardless of the data set. The performance improvement is about 5 to 10% for CIFAR10, 2 to 7% for CIFAR100, and 1.6 to 4% for ImageNet. Please take a look at Appendix L, p. 43 in the revised paper for details.

Reviewer comment (Writing):
The paper uses many abbreviations, making it difficult to comprehend at times.

Author response:
A table summary of the abbreviations was added at the beginning of the appendix to reduce reading difficulty caused by the large number of abbreviations. (Table 3, p.15)

Reviewer comment (Experiments):
The proposed approach is a white-box attack.

Author response:
We appreciate the reviewer's comment on our paper. We respond to each comment in turn.

**Weakness 1**

> The proposed approach is a white-box attack.

We focus on white-box attacks because the security of deep learning models can be significantly improved by accurate robustness evaluations and the development of robust models based on stronger attacks.

White-box and black-box attacks have different advantages and disadvantages from each other.
It is known that the white-box setting can create a more potent attack than the black-box case. Therefore, developing white-box attacks helps develop robust models and evaluate robustness. However, from an attacker's point of view, a white-box setting is unrealistic.

The black-box setting assumes the attacker can access the model's output alone. Therefore, the development of black-box attacks has the advantage of leading to more realistic threat assumptions. However, the black-box attacks tend to show lower attack performance than the white-box attacks and require many queries.

Reviewer comment (Evaluation):
From Table 1 and 6, EDA is not better than GS+LS (ADS). The authors should give more analysis on that.

Author response:
**Weakness 3**

> From Tables 1 and 6, EDA is not better than GS+LS (ADS). The authors should give more analysis on that.

We believe the reviewer is mistaken on this point. We carefully checked the experimental results, but the results showed that EDA consistently performed better than GS+LS (ADS). To make this comparison easier to understand, a table comparing the robust accuracy of EDA and GS+LS (ADS) was added to the Appendix as Table 19 (p.40 in the revised paper).

Reviewer comment (Writing):
Too much mathematics.

Author response:
**Weakness 4**

> Too much mathematics.

A table summary of mathematical symbols was added to the appendix to reduce reading difficulty caused by the large number of mathematical symbols. (Table 4, p.16)

Reviewer comment (Writing):
It is hard to follow the paper. What is PAS? What does the graph $G(X,\Theta)$ mean?

Author response:
**Weakness 5**

> It is hard to follow the paper. What is PAS? What does the graph $G(X,\theta)$ mean?

PAS stands for Prediction Aware Sampling. We have changed the title of Appendix E (p.35) from Prediction Aware Sampling to Prediction Aware Sampling (PAS). In addition, the definition of $G(X, \theta)$ was revised (p.20, after the equation (10)). 
$G(X,\theta)$ is a graph with nodes $X$ and edges between nodes whose Euclidean distance is less than a given threshold $\theta$.

Reviewer comment (Evaluation):
Even though this method saves time, the increase in attack success rate is little. In Table 2, the delta of accuracy is very small. In Table 6 in the appendix, the delta of accuracy is less than 0.1% in most cases.

Author response:
### For weakness 1 (The increase in attack success rate is little.)

As discussed in detail in the global comment, the performance gains from the proposed method are large enough compared to the improvement in recent attacks.

Reviewer comment (Evaluation):
The diversification of EDA is not good enough. In Figure 5, the Diversity Index (DI) of MT_cos exceeds that of the authors' methods.

Author response:
### For weakness 2 (Not enough improvement in diversification)

As Weakness 2, the reviewer pointed out that EDA is not diversified enough because the Diversity Index of EDA is smaller than that of MTcos. However, higher diversification is not always better. What is important in improving search performance is to strike a balance between diversification and intensification [1]. An example is step size control. Searching with a large step size promotes diversification, but it is difficult to further improve the objective value by continuing the search at the same step size. However, gradually decreasing the step size can adjust the balance between diversification and intensification. As a result, as existing studies have shown, the control of gradually decreasing step size contributes to performance improvement. Given that EDA showed higher attack performance than MTcos, Figure 5 just shows that EDA has a more appropriate balance of diversification and intensification than MTcos.

In addition, there is no general definition of diversification. In the main text, we evaluated diversification based on the Diversity Index to check whether the search points form any cluster. However, different definitions of diversification may lead to different analysis results. In Appendix D.6 (p.31 in the revised paper), we quantified the degree of diversification using a measure based on the Euclidean distance. This analysis enables us to understand the diversification of EDA from a perspective that cannot be considered in the Diversity Index. Figure 14 (p.32 in the revised paper) shows the difference in the degree of diversification measured by the Euclidean distance-based measure. In this case, the difference in the degree of diversification between EDA and MT$_{cos}$ is smaller than those measured by the Diversity Index. 

The analysis results using the two diversification indices suggest that EDA and MT$_{cos}$ diversify their search differently.  Based on the above description, we discuss the tradeoff between diversification and efficiency.

Reviewer comment (Evaluation):
The authors’ methods are not consistently effective. In Table 6, there are some cases that the authors’ method even takes longer time than the baseline. Considering the weakness in lack of improvement in diversification and effectiveness, could the authors discuss the tradeoff between efficiency and diversification?

Author response:
### Trade off between efficiency and diversification.

Diversification and computational efficiency are not necessarily trade-offs. If diversification and intensification are appropriately balanced, it is possible to enumerate different local solutions faster. In other words, appropriate diversification and intensification may improve both computational efficiency and search performance at the same time. The experimental results in Table 8 in our revised paper show that EDA shows higher attack performance in less computation time than Adaptive Auto Attack for many models. Therefore, Table 8 and Figure 5 in our revised paper support our contention that EDA balances diversification and intensification well.

In addition, differences in attack performance should be considered when comparing computational efficiency. Table 8 in the revised paper shows that EDA showed higher attack performance than Adaptive Auto Attack for all models where EDA spent relatively more computation time.
Adaptive Auto Attack terminates its search with a different number of queries per model depending on its search progress. Therefore, the potential reason for the difference in efficiency is that Adaptive Auto Attack terminates its search, leaving room for performance improvement, resulting in a shorter time than EDA.

Reviewer comment (Experiments):
This paper does not evaluate the effectiveness of combining these methods with their method. There are many existing methods for improving the success rate of adversarial attacks, such as [1] and [2]. Can EDA and MDO be combined with them? If so, what will the diversification and effectiveness of the authors’ method and baselines be?

Author response:
Technically, it is possible to combine EDA with suggested methods. The suggested methods are expected to promote diversification because of their random influence on the search direction. However, as mentioned above, striking the appropriate balance between diversification and intensification is essential. Therefore, it is difficult to know in advance whether they will contribute to improved computational efficiency or attack performance. Although we experimented with a combination of the suggested methods and EDA in the default setting, the results were not better. However, there is room for improving attack performance by adjusting the suggested methods' hyperparameters.

We look forward to hearing from you regarding our submission. We would be glad to respond to any further questions and comments that you may have.

Reviewer comment (Presentation):
There is a contradiction between the text and the figure. In section 4.1, the paper claims that “Figure 6 indicates that GS+LS found AEs in fewer queries than MT_cos”. However, in Figure 6, it seems GS+LS and MT_cos have similar numbers of queries, and there are more cases in which MT_cos takes fewer queries. Could the authors explain the reason? What do the average lines represent?

Author response:
This analysis using Figure 6 examines how many queries method A takes, on average, to find an adversarial example compared to method B when method A takes more queries than method B.
Figures 6 and 13 in the revised paper illustrate the difference in the number of queries and their averages for images where attack methods A and B successfully found adversarial examples, but took different numbers of queries. For example, "average (MTcos takes more queries)" in Figures 6 and 13 represents the average difference in the number of queries required by MTcos and those by GS+LS for images where MTcos spent more queries. Based on the comparison between "average (MTcos takes more queries)" and "average (MTcos takes fewer queries)", we argue in the main text that "Figure 6 shows that GS+LS found adversarial examples in fewer queries on average than MTcos".

Reviewer comment (Presentation):
In my initial reading, I found the paper's notation to be confusing. Moreover, all the crucial algorithms (ADS, GS, LS, EDA) are deferred to the Appendix, making for a less-than-smooth reading experience. Within the Appendix itself, these algorithms are distributed at different places in the 38-page paper, further complicating the reading process. I would request to consider moving a few of them (ADS, GS, LS, EDA, Target selection) to the main draft.

Author response:
We have carefully considered the comment, but the pseudocodes listed in the appendix are interrelated, and including only some of them in the text may cause confusion.
Instead, we have grouped the pseudocodes closely and added a diagram explaining the dependencies between the pseudocodes in the appendix.

Reviewer comment (Novelty):
In my view, the proposed ADS for selecting the most effective search direction ($\delta$) and loss function ($L$) based on the diversity measures at both input and output, appears quite straightforward. Additionally, the Diversity Index (DI) has been widely studied in Yamamura et al. 2021, thereby paper's technical contribution may be somewhat limited as it just picks the combinations based on this measure and $P_i^e$.

Author response:
We acknowledge that ADS has a clear and simple approach. However, the major contribution of ADS is its ability to determine the combination of objective function and update direction at low computational cost.

Although various objective functions and update formulas have been proposed, no method has been proposed to make good use of them. The related research is those for finding ensembles of attacks, such as CAA [1] and AutoAE [2]. These methods determine the ensemble based on the attack success rate or the best objective function value of the candidate attack methods.
Therefore, they are computationally expensive because the candidate attack methods must be executed in advance. Due to the many combinations of objective functions and search directions, it is impractical to select a pair of objective functions and search directions using these methods from a computational cost perspective.

On the contrary, by focusing on the degree of search diversification, ADS has dramatically reduced the computational cost of combination decisions. ADS allows for better utilization of existing objective functions and update formulas, which may encourage the development of adversarial robustness research. In addition, ADS could facilitate the automation of the robustness evaluation process. For example, we can construct an ensemble of attacks from automatically generated candidates of the ensemble using ADS, CAA, and AutoAE. Thus, ADS is considered to be a versatile and valuable algorithm.

[1] Xiaofeng Mao, Yuefeng Chen, Shuhui Wang, Hang Su, Yuan He, and Hui Xue. Composite adversarial attacks. AAAI 2021.

[2] Shengcai Liu, Fu Peng, and Ke Tang. Reliable Robustness Evaluation via Automatically Constructed Attack Ensembles. AAAI 2023.

Reviewer comment (Experiments):
The GS and LS techniques introduced within the search framework are, to some extent, heuristic in nature. Although ablation studies have been carried out with respect to N1, N2, and N3, the current framework's configuration, comprising GS with N1 and N2 iterations, followed by LS with N3 iterations, strikes me as somewhat heuristic.

Author response:
We determined the step size inspired by the APGD heuristic, which is widely used and works well. The APGD heuristic has eight checkpoints. The checkpoints are written down as $w_1=\lceil0.22N\rceil, w_2=\lceil0.41N\rceil, w3=\lceil0.57\rceil,...,w_8=\lceil0.99N\rceil$, where $N$ is the total iterations.
Our framework's $N_1$, $N_2$, and $N_3$ iterations correspond to the search from checkpoint $w_1$, $w_1$ to $w_2$, and $w_2$ and beyond in the APGD heuristic, respectively.

APGD has an initial step size of $2\varepsilon$ and halves the step size if the condition is satisfied at each checkpoint. If the condition is satisfied at all checkpoints, the step size is $2\varepsilon$ up to $w_1$, $\varepsilon$ between $w_1$ and $w_2$, and $\varepsilon/2$ after $w_2$. Based on this, we set the step size in the proposed framework to $2\varepsilon$, $\varepsilon$, and $\varepsilon/2$ for $N_1$, $N_2$, and $N_3$ iterations, respectively.

Reviewer comment (Evaluation):
Upon closer examination, the differences in performance between ADS and Random appear negligible in Table 1, and the variation between GS+LS (ADS) and GS+LS (Random) seems relatively small. On average, across nine models, this difference stands at less than approximately 0.05% (please correct me if I'm wrong).

Author response:
As mentioned in the global comment, based on the existing research, even an improvement in attack performance from AutoAttack of less than 0.1% is considered sufficient. Considering that GS+LS(ADS) showed competitive attack performance as AutoAttack, the difference between GS+LS(ADS) and GS+LS(random) is large enough.

In addition, the performance of GS+LS(random) is expected to be unstable because GS+LS(random) is completely dependent on random numbers for selection. On the contrary, GS+LS(ADS) are expected to show stable and high performance.

Reviewer comment (Experiments):
Notably, there exists a substantial number of hyperparameters tied to ADS, GS, LS, and the target selection schemes. While the paper presents ablation studies, I anticipate that the widespread adoption of the proposed method as a robust alternative to the AutoAttack suite might pose a considerable challenge.

Author response:
EDA can be used without hyperparameter tuning in practice, similar to AutoAttack.

AutoAttack's components, APGD, FAB, and Square attack, have explicitly given hyperparameters and magic numbers heuristically determined by the authors. While AutoAttack is not inherently parameter-free, their default parameters are carefully determined.

Our ablation study suggests that EDA is robust to hyperparameter settings. In addition, the particularly influential parameter, step size, is determined based on the APGD heuristic, which is widely used and effective. Therefore, EDA can be used without hyperparameter tuning in practice.

Reviewer comment (Experiments):
ADS requires access to the dataset. The authors perform ADS on 1% of the data, while AutoAttack and other baseline attacks operate on a per-datapoint basis without depending on additional datapoints.

Author response:
We have tested the transferability of ADS in the cross-domain setting for several models in CIFAR10 and ImageNet. The experimental results, summarized in the table below, suggest that the proposed method may work well in the cross-domain setting. A possible reason for this is that the pairs selected by ADS have a consistent trend regardless of the model under attack.

| source->target                                                       | clean accuracy (target) | EDA (cross-domain) | EDA   |
| -------------------------------------------------------------------- | ------------------- | ----------------------- | ----- |
| Salman et al.,2020->Addepalli et al.,2022 (ImageNet->CIFAR10)   | 85.71               | 52.42                   | 52.43 |
| Salman et al.,2020 ->Carmon et al.,2019 (ImageNet->CIFAR10)     | 89.69               | 59.44                   | 59.40 |
| Salman et al.,2020 ->Sitawarin et al.,2021 (ImageNet->CIFAR10)  | 86.84               | 50.64                   | 50.58 |
| Addepalli et al.,2022->Salman et al.,2020 (CIFAR10->ImageNet)   | 64.02               | 34.52                   | 34.52 |
| Addepalli et al.,2022->Wong et al.,2020 (CIFAR10->ImageNet)     | 55.62               | 26.24                   | 26.12 |
| Addepalli et al.,2022->Engstrom et al.,2019 (CIFAR10->ImageNet) | 62.56               | 28.96                   | 29.01 |

Reviewer comment (Experiments):
What is the individual influence of the two terms $P_i^e$ and DI(.) in the overall ADS search performance?

Author response:
The $P_i^e$ measures the degree of diversification in the output space during the search. Therefore, a pair with the largest $P_i^e$ is expected to search for a high diversity in the output space. In addition, from Yamamura et al. (2022), it can be assumed that the ACG direction increases $P_i^e$, while the steepest-like direction does not. From the above, it is considered that the pair with the maximum $P_i^e$ is likely to include the ACG direction.

In our use case, DI measures the diversity of the best point set obtained by the search. In other words, DI represents the dissimilarity between the best points. That is, we expect that pairs with the largest DI are more likely to enumerate dissimilar solutions. Intuitively, updates in diverse directions contribute to the enumeration of dissimilar solutions. Given that the search direction is gradient-dependent, the pair with the largest DI is likely to include a variety of objective functions and update formulas.

Reviewer comment (Evaluation):
What combinations were discovered by ADS for the different models? Are there any insights into the top $n_a$ combinations found by ADS for different models?

Author response:
There are slight differences in the pairs selected for each model. However, the trend is consistent: the ACG's direction is more likely to be selected for GS, and the Nesterov's accelerated gradient direction is more likely to be selected for LS.
In addition, experimental results on EDA's transferability suggest that EDA can generate highly transferable adversarial examples.
These experimental results suggest that different models may have some common features.

Reviewer comment (Experiments):
Why is ADS performed in the LS stage? What are the benefits of conducting it before LS? While I understand that the initial points are different at these two stages, could the authors provide an empirical analysis of employing ADS in the LS stage?

Author response:
The advantage of performing ADS in LS is that it reduces the possibility of redundant searches. The search direction of ACG is likely to be selected in GS, and the direction of ACG is more likely to be similar to the steepest direction for small step sizes. Therefore, an efficient search may not be possible if the search is continued using the pair selected in GS. In addition, given the multimodal nature of the objective function, searching in Nesterov's accelerated gradient direction and the gradient direction may help to find different local solutions.

Reviewer comment (Evaluation):
Are top combinations {$\delta_{a_j^*}, L_{a_j^*}$} found at GS, LS stages correlated?

Author response:
The brief answer is No. The potential reasons are: 1. $P_i^e$ and DI play different roles from each other, 2. the ACG's search direction may be similar to the steepest for small step sizes, and 3. the difference between Nesterov's acceleration gradient direction and gradient direction. Appendix D.8 (p. 33 in revised paper) provides further information.

Reviewer comment (Evaluation):
Can you report the average performance over 9 models in Table 1 for all methods?

Author response:
The average robust accuracy values over the nine models listed in Table 1 are as follows.

| AA    | Naive | GS(ADS) | LS(ADS) | GS+LS(ADS) | GS+LS(R-ADS) | GS+LS(RAND) | MT$_{cos}$ |
| ----- | ----- | ------------- | ------------- | ---------------- | ------------------ | ----------------- | ------------ |
| 37.83 | 38.09 | 38.50         | 38.66         | 37.84            | 37.99              | 37.90             | 37.87        |

Reviewer comment (Experiments):
In the absence of a dataset to perform ADS, is the combinations found by ADS transferable in the cross-domain setting?

Author response:
We have tested the transferability of ADS in the cross-domain setting for several models in CIFAR10 and ImageNet. The experimental results, summarized in the table below, suggest that the proposed method may work well in the cross-domain setting. A possible reason for this is that the pairs selected by ADS have a consistent trend regardless of the model under attack.

| source->target                                                       | clean accuracy (target) | EDA (cross-domain) | EDA   |
| -------------------------------------------------------------------- | ------------------- | ----------------------- | ----- |
| Salman et al.,2020->Addepalli et al.,2022 (ImageNet->CIFAR10)   | 85.71               | 52.42                   | 52.43 |
| Salman et al.,2020 ->Carmon et al.,2019 (ImageNet->CIFAR10)     | 89.69               | 59.44                   | 59.40 |
| Salman et al.,2020 ->Sitawarin et al.,2021 (ImageNet->CIFAR10)  | 86.84               | 50.64                   | 50.58 |
| Addepalli et al.,2022->Salman et al.,2020 (CIFAR10->ImageNet)   | 64.02               | 34.52                   | 34.52 |
| Addepalli et al.,2022->Wong et al.,2020 (CIFAR10->ImageNet)     | 55.62               | 26.24                   | 26.12 |
| Addepalli et al.,2022->Engstrom et al.,2019 (CIFAR10->ImageNet) | 62.56               | 28.96                   | 29.01 |