Paper: Sparse-PGD: An Effective and Efficient Attack for $l_0$ Bounded Adversarial Perturbation (ICLR-2024)

Reviewer comment (Experiments):
It would be interesting to see the results of the method without the additional constraint that a pixel is considered perturbed if any of its channels are perturbed. The approach could still be developed by creating a mask for every channel. Removing this limit would make the attack comparable with many other white-box sparse attacks, including EAD, VFGA, PDPGD, BB, and FMN.

Author response:
Thanks for pointing this out, we will consider your comments and try to improve our work in future work.

Reviewer comment (Novelty):
The difference between the proposed attack method and the SAIF attack, which the authors state is similar (beginning of sect. 4.1), is unclear. The authors do not explain clearly what is different and what has been added to make it perform better. This should be discussed in sect. 4.1.

Author response:
For the attack process, though we adopt a similar decomposition method to the perturbation to SAIF=, **our attack design is significantly different from the previous work in two perspectives:** 
1. Instead of using an $\ell_1$ convex surrogate for the $l_0$ norm constraint of the sparsity mask as done in SAIF, we introduce a continuous mask $\widetilde{m}$ and project it back to the feasible set. To prevent the magnitude of $\widetilde{m}$ from being too big, we add a sigmoid function before projecting $\widetilde{m}$ to the binary mask $m$. This **new but simple** strategy has proven to be effective as shown in the ablation studies (Table 2, Sec 5.3) where we observed a huge robust accuracy drop from 49.5\% to 22.2\%. Moreover, we further improve the exploration capability for the mask $m$ by a random reinitialization mechanism, which is shown to improve the performance in the ablation study. This design alleviates the problem of local convergence, which is reported in SAIF.
2. The ablation studies (Table 2, Sec 5.3) also indicated naively decomposing the perturbation by $\delta = p \odot m$ can **deteriorate the performance**, which further corroborates the effectiveness of our attack designs.

Reviewer comment (Reproducibility):
Optimization is unclear - Section 4.1 requires additional details. Arguments are made concerning when a relaxation is necessary (i.e., through a projection), yet later it is claimed that the relaxation exhibits deficiencies, so the original optimization is used instead. After reading 4.1, it is unclear what optimization sPGD actually entails and what is used in the evaluation.

Author response:
1. For updating the magnitude tensor, are p and delta the same variable? 

    **Reply**: In our work, we decompose the sparse perturbation $\delta$ into a magnitude tensor $p\in\mathbb{R}^{h\times w\times c}$ and a sparsity mask $m\in\{0,1\}^{h\times w\times 1}$, i.e., $\delta =p \odot m$. We update $p$ and $m$ separately, and then obtain $\delta$ with the updated $p$ and $m$.

2. Why is the l2-norm of the loss taken in (5)? 

    **Reply**: According to the first-order Taylor expansion of loss function, $L(x+v)\approx L(x)+\nabla L(x)^T v$. Finding the steepest ascent direction of the loss function is equivalent to finding the $v$ which maximizes the directional derivative. In Euclidean space, if we want to find the steepest ascent direction within the step size of the unit norm, i.e., ||v||2=1 , the direction is $v=\nabla L(x)/||\nabla L(x)||_2$ [1].  
In our scenarios, we want to update $\widetilde{m}$ with the step size $\beta$, so the updating can be written as $\widetilde{m} = \widetilde{m} + \beta \cdot \nabla_{\widetilde{m}} L / (||\nabla_{\widetilde{m}} L||_2 + \gamma)$, where $\gamma$ is a small constant to prevent the denominator from being zero.

3. For updating the sparsity mask, what is gradient ascent performed on? 

    **Reply**: The $l_0$ bounded perturbation $\delta$ can be decomposed into magnitude tensor $p$ and a sparsity mask $m$, i.e., $\delta =p \odot m$. We aim to maximize the loss $L(\theta, x+p\odot m)$ through updating $p$ and $m$ with gradient ascent. Since $m$ is sparse and discrete, directly optimizing it is challenging. Instead, we introduce a continuous variable $\widetilde{m} \in \mathbb{R}^{h\times w\times 1}$ and project $\widetilde{m}$ to the feasible set $S_m$ before multiplying it with the magnitude tensor $p$ to calculate the sparse perturbation $\delta$. Since $\widetilde{m}$ is a continuous variable, we can optimize it using gradient ascent.

4. It is unclear what, "Since elements in m are 0 or 1, we use sigmoid to normalize elements in m-tilde to be 0 or 1" is trying to say; aren't elements in m in [0, 1] because of (6)? 

    **Reply**:  We are sorry about the confusion, we have modified the claim in the paper. In addition, we clarify that the sigmoid function used here is to prevent $\widetilde{m}$ from being too large in magnitude. When the magnitude of $\widetilde{m}$ is large, its gradient will vanish to zero by applying the sigmoid function, this mechanism will prevent $\widetilde{m}$ from being too large and ensure the numeric stability. In the same settings of Table 2 in section 5.3, if we discard the sigmoid function, the robust accuracy increases from 20.0 to 68.5, which further demonstrates the effectiveness of the sigmoid function.

5. The argument that projection on the binary set Sm is discarded because coordinate descent is suboptimal is unclear; why is such a projection introduced to be later argued as suboptimal and thus discarded? (In fact, this observation is stated twice) 

    **Reply**: Due to $S_{m}$, the gradient of $p$ is sparse, so at most $k$ elements of $p$ are updated at one iteration, which is equivalent to $k$-coordinate descent. Previous work [2] already demonstrates that coordinate descent results in slow convergence and suboptimal performance. In addition, we want to emphasize that the projection operator is only discarded during the back-propagation, it still works in the forward propagation. Moreover, as we indicate in Table 3, the projected and unprojected gradients are complementary in terms of performance, none of them is always better than the other, so we run sPGD twice with different backward functions in sAA for comprehensive evaluation.

6. It is unclear where the projection onto the binary set Sm is used in gp and why it is used in tandem with gp-tilde if gp exhibits both non-convergence and gradient explosions

    **Reply**: This question is already addressed in the explanations above.

Reviewer comment (Evaluation):
Evaluation methodology - There are many important details are not present in the evaluation and necessary plots are missing (see questions for details).

Author response:
8. A distortion vs accuracy curve should be plotted, so that we can understand the performance curves of sAA against baselines. Reporting the final results at a fixed norm boundary is not readily indicative of attack performance, given that are are many values of k a defender would consider to be "adversarial"

    **Reply**: Thank you for pointing this out, we added the curves of sAA in Figure 1 (b).

Reviewer comment (Novelty):
Contribution of attacks introduced in this work is unclear - It does not seem appropriate to add Sparse-RS as part of sAA, given that Sparse-RS is used verbatim from prior work. The evaluation should only include the contributions made in this work.

Author response:
11. Attack configuration does not seem fair - It is not clear to me why Sparse-RS is included within sAA when it is used verbatim from prior work. So that readers can understand the core contributions of this work, comparisons against baselines should only be evaluated against the introduced attacks. Moreover, the JSMA (Papernot, 2016) is one of the first l0-based attacks to be introduced in the literature. It is unclear to me why this not compared against in this work.

    **Reply**: Thanks for pointing this out, we have cited it in the paper. Although JSMA is one of the first $l_0$ attacks, its code is not publicly available, making it difficult for reproduction. In addition, JSMA is not included in the existing works we use as baselines either. Considering these two points, we do not include it in the comparison.

Reviewer comment (Theory):
Section 4.1 can be confusing at times - Section 4.1 should be revisited, given that there seem to be inconsistencies in the motivation of certain decisions and the optimization itself is unclear. Specifically: (a) for updating the magnitude tensor, are p and delta the same variable? (b) Why is the l2-norm of the loss taken in (5)? (c) For updating the sparsity mask, what is gradient ascent performed on? (d) it is unclear what, "Since elements in m are 0 or 1, we use sigmoid to normalize elements in m-tilde to be 0 or 1" is trying to say; aren't elements in m in [0, 1] because of (6)? (e) the argument that projection on the binary set Sm is discarded because coordinate descent is suboptimal is unclear; why is such a projection introduced to be later argued as suboptimal and thus discarded? (In fact, this observation is stated twice) (f) it is unclear where the projection onto the binary set Sm is used in gp and why it is used in tandem with gp-tilde if gp exhibits both non-convergence and gradient explosions, and (g) there are many terms that are co-dependent with other terms throughout 4.1--it is challenging to understand precisely what are the main ingredients of Sparse-PGD, why they matter, and what decisions influenced their design.

Author response:
1. For updating the magnitude tensor, are p and delta the same variable? 

    **Reply**: In our work, we decompose the sparse perturbation $\delta$ into a magnitude tensor $p\in\mathbb{R}^{h\times w\times c}$ and a sparsity mask $m\in\{0,1\}^{h\times w\times 1}$, i.e., $\delta =p \odot m$. We update $p$ and $m$ separately, and then obtain $\delta$ with the updated $p$ and $m$.

2. Why is the l2-norm of the loss taken in (5)? 

    **Reply**: According to the first-order Taylor expansion of loss function, $L(x+v)\approx L(x)+\nabla L(x)^T v$. Finding the steepest ascent direction of the loss function is equivalent to finding the $v$ which maximizes the directional derivative. In Euclidean space, if we want to find the steepest ascent direction within the step size of the unit norm, i.e., ||v||2=1 , the direction is $v=\nabla L(x)/||\nabla L(x)||_2$ [1].  
In our scenarios, we want to update $\widetilde{m}$ with the step size $\beta$, so the updating can be written as $\widetilde{m} = \widetilde{m} + \beta \cdot \nabla_{\widetilde{m}} L / (||\nabla_{\widetilde{m}} L||_2 + \gamma)$, where $\gamma$ is a small constant to prevent the denominator from being zero.

3. For updating the sparsity mask, what is gradient ascent performed on? 

    **Reply**: The $l_0$ bounded perturbation $\delta$ can be decomposed into magnitude tensor $p$ and a sparsity mask $m$, i.e., $\delta =p \odot m$. We aim to maximize the loss $L(\theta, x+p\odot m)$ through updating $p$ and $m$ with gradient ascent. Since $m$ is sparse and discrete, directly optimizing it is challenging. Instead, we introduce a continuous variable $\widetilde{m} \in \mathbb{R}^{h\times w\times 1}$ and project $\widetilde{m}$ to the feasible set $S_m$ before multiplying it with the magnitude tensor $p$ to calculate the sparse perturbation $\delta$. Since $\widetilde{m}$ is a continuous variable, we can optimize it using gradient ascent.

4. It is unclear what, "Since elements in m are 0 or 1, we use sigmoid to normalize elements in m-tilde to be 0 or 1" is trying to say; aren't elements in m in [0, 1] because of (6)? 

    **Reply**:  We are sorry about the confusion, we have modified the claim in the paper. In addition, we clarify that the sigmoid function used here is to prevent $\widetilde{m}$ from being too large in magnitude. When the magnitude of $\widetilde{m}$ is large, its gradient will vanish to zero by applying the sigmoid function, this mechanism will prevent $\widetilde{m}$ from being too large and ensure the numeric stability. In the same settings of Table 2 in section 5.3, if we discard the sigmoid function, the robust accuracy increases from 20.0 to 68.5, which further demonstrates the effectiveness of the sigmoid function.

5. The argument that projection on the binary set Sm is discarded because coordinate descent is suboptimal is unclear; why is such a projection introduced to be later argued as suboptimal and thus discarded? (In fact, this observation is stated twice) 

    **Reply**: Due to $S_{m}$, the gradient of $p$ is sparse, so at most $k$ elements of $p$ are updated at one iteration, which is equivalent to $k$-coordinate descent. Previous work [2] already demonstrates that coordinate descent results in slow convergence and suboptimal performance. In addition, we want to emphasize that the projection operator is only discarded during the back-propagation, it still works in the forward propagation. Moreover, as we indicate in Table 3, the projected and unprojected gradients are complementary in terms of performance, none of them is always better than the other, so we run sPGD twice with different backward functions in sAA for comprehensive evaluation.

6. It is unclear where the projection onto the binary set Sm is used in gp and why it is used in tandem with gp-tilde if gp exhibits both non-convergence and gradient explosions

    **Reply**: This question is already addressed in the explanations above.

7. There are many terms that are co-dependent with other terms throughout 4.1--it is challenging to understand precisely what are the main ingredients of Sparse-PGD, why they matter, and what decisions influenced their design.

    **Reply**: The main idea of our method is to decompose a $l_0$ bounded perturbation $\delta$ into a magnitude tensor $p$ and a sparsity mask $m$. 

    - Since $p$ is only constrained in the image box $[0, 1]^{h\times w\times c}$ which is similar to the $l_\infty$ constraint, we update it using $l_\infty$-PGD.
    - Updating a sparse and discrete $m$ is challenging in optimization, we update its continuous alternative $\widetilde{m}$ and then projecting $\widetilde{m}$ to obtain $m$.
    - Due to the binary projection onto $S_{m}$, the update of $p$ is equivalent to $k$-coordinate descent, which leads to slow convergence and suboptimal performance. Inspired by training pruned neural networks and lottery ticket hypothesis [3-6], we discard the projection $S_{m}$ when calculating the gradient of $p$ to ensure every element of $p$ can be updated. However, in the experiments, we found the unprojected and projected gradients are complementary in terms of performance.
    - Random reinitialization is proposed to prevent $m$ from getting trapped into local maxima, because $m$ changes only when the relative magnitude ordering of the continuous alternative $\widetilde{m}$ changes. In other words, slight changes in $\widetilde{m}$ usually mean no change in $m$.

Reviewer comment (Evaluation):
Evaluation could be clearer - While I appreciate the extensive evaluation, it does not appear to disclose sufficient information to measure the performance of sAA. Specifically, (a) a distortion vs accuracy curve should be plotted, so that we can understand the performance curves of sAA against baselines. Reporting the final results at a fixed norm boundary is not readily indicative of attack performance, given that are are many values of k a defender would consider to be "adversarial", (b) when attacking against adversarially trained models, perturbations must stay within the threat model. That is, it should be made clear that, when attacking an l-infinity-based model, the l0 perturbations also do not exceed, e.g., 8/255. Otherwise, it is not clear to me what insights are to be drawn from attacking a model whose threat model is violated, (c) mixing threat models does not seem sound. It is not clear why black-box attacks are compared to white-box attacks, etc. White-box threat models should only be compared to white-box attacks, and likewise for black-box attacks.

Author response:
8. A distortion vs accuracy curve should be plotted, so that we can understand the performance curves of sAA against baselines. Reporting the final results at a fixed norm boundary is not readily indicative of attack performance, given that are are many values of k a defender would consider to be "adversarial"

    **Reply**: Thank you for pointing this out, we added the curves of sAA in Figure 1 (b).

9. When attacking against adversarially trained models, perturbations must stay within the threat model. That is, it should be made clear that, when attacking an l-infinity-based model, the l0 perturbations also do not exceed, e.g., 8/255. Otherwise, it is not clear to me what insights are to be drawn from attacking a model whose threat model is violated

    **Reply**: In the previous work [7], when evaluating the robustness of an $l_\infty$-robust model against $l_2$ attacks, they do not constrain the $l_\infty$ norm of the $l_2$ perturbations to be $8/255$ or something else. The results in [7] also indicate that a model that is robust to a specific attack is usually not robust to another attack at all. In addition, when we validate whether a model is robust to a white-box attack, we should follow the principle that the attacker knows model architecture and model weights; training algorithm and training data; test time randomness (either the values chosen or the distribution) of the defender [8]. In the evaluation, we should not compromise the defender, so what we do is just to generate $l_0$ bounded perturbations without any other restrictions. 

10. Mixing threat models does not seem sound. It is not clear why black-box attacks are compared to white-box attacks, etc. White-box threat models should only be compared to white-box attacks, and likewise for black-box attacks.

    **Reply**: As shown in AutoAttack [9], they also combine white-box and black-box attacks to obtain a reliable and comprehensive evaluation of adversarial robustness, since white-box attacks sometimes have suboptimal performance due to gradient masking, but black-box attacks are immune from it because they do not rely on the gradient information. Empirically, among current adversarial attacks generating $l_0$ bounded perturbations, black-box attacks usually outperform white-box ones. For example, we see black-box sparse-RS outperforms white-box PGD$_0$ in many cases, depending on the evaluated models. In summary, we need to combine both white-box and black-box attacks to reliably and comprehensively evaluate the robustness, because they are complimentary.

Reviewer comment (Experiments):
Attack configuration does not seem fair - It is not clear to me why Sparse-RS is included within sAA when it is used verbatim from prior work. So that readers can understand the core contributions of this work, comparisons against baselines should only be evaluated against the introduced attacks. Moreover, the JSMA (Papernot, 2016) is one of the first l0-based attacks to be introduced in the literature. It is unclear to me why this not compared against in this work.

Author response:
11. Attack configuration does not seem fair - It is not clear to me why Sparse-RS is included within sAA when it is used verbatim from prior work. So that readers can understand the core contributions of this work, comparisons against baselines should only be evaluated against the introduced attacks. Moreover, the JSMA (Papernot, 2016) is one of the first l0-based attacks to be introduced in the literature. It is unclear to me why this not compared against in this work.

    **Reply**: Thanks for pointing this out, we have cited it in the paper. Although JSMA is one of the first $l_0$ attacks, its code is not publicly available, making it difficult for reproduction. In addition, JSMA is not included in the existing works we use as baselines either. Considering these two points, we do not include it in the comparison.

Reviewer comment (Experiments):
While sAA seems effective (Table 1), there are some concerns in my opinion: first, according to Fig. 1a, the attacks notably benefit from more iterations. In particular, Sparse-RS shows significant improvements between 3k and 10k iterations for all models, which means that the results reported in Table 1 might be suboptimal. Second, in Tables 6, 7 and 8, CS alone appears to be better than sAA on the models robust to $\ell_0$-attacks: while CS is evaluated on a subset of points only, an improvement of more than 3% (Table 6) seems significant to hint to the fact that, even on the full test set, the results of sAA might be improved. Finally, in most cases the robust accuracy of the best individual attack (either RS or sPGD) is quite higher (2-3%) than their worst-case, i.e. sAA, which suggests that each attack is suboptimal.

Author response:
First, Sparse-RS indeed has performance improvement between 3k and 10k iterations for all models. However, as illustrated in Figure 1 (a), in the cases where our method outperforms Sparse-RS at 3k iterations, it is still substantially better than Sparse-RS when we use more iterations. That is to say, setting the number of iterations to 3k does not affect the performance superiority of our method. However, we are happy to present the results of Sparse-RS with 10k iterations and compare them with our methods. Due to the time constraint, we only run the experiments on a subset of models as shown in Table 1. It is clear that sAA still performs the best in all models except in $\ell_{\infty}$-adv model and we acknowledge this as one weakness of our method and will try to improve it in future work. As for the second question, we would like to emphasize that our $l_0$-robust models are just baselines for future work. The major motivation of $l_0$-robust models is to demonstrate the efficiency of our method so that we can utilize it for adversarial training. Due to the challenges of optimization in an $l_0$ bounded space, we notice that the $l_0$-robust models are more vulnerable to black-box attacks, especially sparse-RS, although these models still obtain the best robustness when we consider all attack methods. In Tables 6, 7 and 8, the reason why CS is better than sAA on the models robust to $l_0$-attacks can be attributed to that Sparse-RS underperforms CS on $l_0$-robust models. Despite that, due to the very high complexity of CS (10x slower), we do not adopt it as the black-attack here. We believe this issue can be solved if we have a better adversarial training methods to mitigate the gradient masking problem of the models obtained, which we leave it as future work.

***Table 1: Robust accuracy of selected models on Sparse RS where the sparsity level $k=20$. We record the performance of RS in both 3k and 10k iterations and their changes.***

|Model | Network | RS  | **sPGD$_{\mathrm{CE}}$** |**sPGD$_{\mathrm{CE+T}}$** | **sAA**|
|----|----|----|----|----|----|
|Vanilla | RN-18 | 0.0 $\to$ 0.0 | 0.0 |0.0 | **0.0**|  
|$l_\infty$-adv. trained, $\epsilon=8/255$|
| PORT | RN-18 |14.6 $\to$ **9.7**|20.9|16.1 | 10.8|
|DKL| WRN-28 | 11.0 $\to$ **7.0** | 22.5|16.6| 9.6|
|$l_2$-adv. trained, $\epsilon=0.5$|
|HAT | PRN-18 |20.5 $\to$ 13.9 | 13.2 | 10.0 | **9.3**|
|DM| WRN-28 | 23.4 $\to$ 16.1 |20.9| 15.5| **14.1**|
|$l_1$-adv. trained, $\epsilon=12$|
|$l_1$-APGD | PRN-18 |33.1 $\to$ 25.8 |  24.0| 20.0|**19.5**|
|$l_0$-adv. trained, $k=20$|
|**sTRADES** | PRN-18 |52.1 $\to$ **41.8** |79.5|77.9 |52.0 |

Reviewer comment (Experiments):
The budget of iterations of the attacks is not justified: looking at Fig. 1a it seems that more iterations would significantly improve the results, especially for RS. If I understand it correctly, sPGD is used for 20 runs ({1 CE, 9 targeted CE} x {projected, unprojected}) each of 300 iterations, for total 6k iterations (each consisting in one forward and one backward pass of the network). However, only 3k queries (forward pass only) are used for RS, which seems unbalanced given that RS provides better results for $\ell_\infty$- and (especially) $\ell_0$-adversarially trained models.

Author response:
***Response***: We need to clarify that the sPGD with 20 runs ({1 CE, 9 targeted CE} x {projected, unprojected}) is only adopted in sAA. We do not compare the performance of Sparse-RS with sPGD and its variants. By contrast, in Figure 1 (a), we evaluate sPGD with unprojected gradient with different numbers of iterations. The robust accuracies of Sparse-RS and sPGD at the same x-axis value are obtained with the same number of iterations. Therefore, we think the comparison is fair here.

Reviewer comment (Novelty):
The claim that no prior works proposed adversarial training for the $\ell_0$-threat model is imprecise, see e.g. [Croce & Hein (2019)](https://openaccess.thecvf.com/content_ICCV_2019/papers/Croce_Sparse_and_Imperceivable_Adversarial_Attacks_ICCV_2019_paper.pdf). Moreover, the cost of using 100 iterations of sPGD in adversarial training seem very large. Finally, the sAT and sTRADES would need to be added to Fig. 1a, to see how the effect of more queries in RS on the achieved accuracy (see previous points).

Author response:
**Response**: First of all, thank you for pointing this out, we have modified this claim and updated the results of using PGD$_0$ as adversarial training in the paper. Meanwhile, we have also included the results in Table 1 and General Response for your reference. It is clear that models adversarially trained by PGD$_0$ attack are not robust against our proposed attack and less robust than the model trained using our method. Secondly, we want to emphasize that using 100 iterations of sPGD in adversarial training is just a baseline to obtain $l_0$-robust model, we will conduct research on a more efficient adversarial training algorithm for $l_0$ bounded perturbations in future work.

Reviewer comment (Experiments):
As a minor point, it would be interesting to have some evaluation on ImageNet models.

Author response:
**Response**: The results of ImageNet100 have been added to Table 9 of Appendix A.1 to more comprehensively demonstrate the effectiveness of our methods, especially in the case of high-resolution images. Similar to AutoAttack, which includes a standard version and a more expensive AutoAttack+ in its repository https://github.com/fra31/auto-attack, we will provide sAA of different levels, such as the standard one as demonstrated in the paper and the stronger ones with more iterations for each attack, to practitioners to evaluate the robustness of their models based on their computational constraints.

Reviewer comment (Experiments):
The current results do not convincingly support how the paper proposes to build sAA (e.g. how significantly would the results improve with 2x iterations to every attack, or 4x to RS?).

Author response:
**Response**: The results of ImageNet100 have been added to Table 9 of Appendix A.1 to more comprehensively demonstrate the effectiveness of our methods, especially in the case of high-resolution images. Similar to AutoAttack, which includes a standard version and a more expensive AutoAttack+ in its repository https://github.com/fra31/auto-attack, we will provide sAA of different levels, such as the standard one as demonstrated in the paper and the stronger ones with more iterations for each attack, to practitioners to evaluate the robustness of their models based on their computational constraints.

Reviewer comment (Experiments):
The effectiveness of adversarial training with sPGD should be tested more thoroughly.

Author response:
**Response**: The results of ImageNet100 have been added to Table 9 of Appendix A.1 to more comprehensively demonstrate the effectiveness of our methods, especially in the case of high-resolution images. Similar to AutoAttack, which includes a standard version and a more expensive AutoAttack+ in its repository https://github.com/fra31/auto-attack, we will provide sAA of different levels, such as the standard one as demonstrated in the paper and the stronger ones with more iterations for each attack, to practitioners to evaluate the robustness of their models based on their computational constraints.

Reviewer comment (Novelty):
Following ﻿Sparse Adversarial and Interpretable Attack Framework (SAIF) [1], which adopts ﻿a magnitude tensor and sparsity mask same as this paper, the authors further ﻿discard the projection to the binary set when calculating the gradient and use the unprojected gradient to update ﻿the magnitude tensor p. ﻿Sparse-AutoAttack (sAA) part has extended the work of ﻿AutoAttack (AA) [2,3], and the reason for discarding ﻿the adaptive step size, momentum and difference of logits ratio (DLR) loss function should be further explained clearly. The paper appears to offer limited new perspectives on the attack process and lacks a notable degree of technical innovation.

Author response:
For the attack process, though we adopt a similar decomposition method to the perturbation to SAIF [1], **our attack design is significantly different from the previous work in two perspectives:** 
    1. Instead of using an $\ell_1$ convex surrogate for the $l_0$ norm constraint of the sparsity mask as done in SAIF, we introduce a continuous mask $\widetilde{m}$ and project it back to the feasible set. To prevent the magnitude of $\widetilde{m}$ from being too big, we add a sigmoid function before projecting $\widetilde{m}$ to the binary mask $m$. This **new but simple** strategy has proven to be effective as shown in the ablation studies (Table 2, Sec 5.3) where we observed a huge robust accuracy drop from 49.5\% to 22.2\%. Moreover, we further improve the exploration capability for the mask $m$ by a random reinitialization mechanism, which is shown to improve the performance in the ablation study. This design alleviates the problem of local convergence, which is reported in SAIF.
    2. The ablation studies (Table 2, Sec 5.3) also indicated naively decomposing the perturbation by $\delta = p \odot m$ can **deteriorate the performance**, which further corroborates the effectiveness of our attack designs. 

    In terms of backward function for updating $p$, we argue that the original gradient $g_{p} = \nabla_{\delta} L(\theta, x + \delta) \odot m$ is sparse due to the sparsity of the mask $m$. **The sparse update will result in sub-optimal performance** similar to coordinate descent when there are limited iterations. To solve this problem, we adopt a novel unprojected gradient $g_{p} = \nabla_{\delta} L(\theta, x + \delta) \odot \sigma(\widetilde{m})$ inspired by network pruning and lottery ticket hypothesis [6 - 9] and combine it with the original sparse gradient to achieve a better trade-off on exploration and exploitation.

Reviewer comment (Novelty):
The authors claim that “﻿We are the first to conduct adversarial training against l_{0} bounded perturbations.” However, related work had also conducted similar experiments [4].

Author response:
Thank you for pointing this out. We will modify this claim in the paper and add the results of using PGD$_0$ as adversarial training. The results are reported in Table 1, Sec 1.2.1 of General Response, and we have also included them in the paper. We observe that the model trained with adversarial samples generated by PGD$_0$ can be easily beaten by our proposed attack sPGD, while the model trained with sPGD-generated adversarial samples is still robust against PGD$_0$, so we do not include it in other experiments.

Reviewer comment (Evaluation):
This paper has emphasized the contribution of ﻿computational complexity and efficiency but lacks corresponding analysis for ﻿computational complexity and query budgets for comparison.

Author response:
In Section 5.2, we study the efficiency by comparing the performance of Sparse-RS with sPGD_CE (Figure 1(a)) and sPGD_CE+T (Figure 2) under different number of iterations. Both figures show that sPGD achieves significantly better performance than RS attack under limited iterations. That is to say, Sparse-RS requires a much larger query budget to achieve comparable performance.

Reviewer comment (Evaluation):
In Table 1 in experimental part, ﻿RS attack outperforms sPGD_{CE+T} for l_{∞} models while more analysis is required.

Author response:
Thanks for pointing this out. For attacking $l_\infty$ models, we agree that sPGD is outperformed by RS when there are enough iterations, which means sPGD suffers from gradient masking to some degree in this case. This is also why we included the black-box RS attack in sAA to comprehensively evaluate the robustness. Further mitigating the gradient masking effect when sPGD is applied to some specific models will be our future work.

Reviewer comment (Presentation):
The performance analysis in Subsection 5.1 is not well-organized for clarity.

Author response:
Sorry for the obscurity. We summarized the key points of the analysis below:
    - By comparing against five attacks, our proposed method archives the lowest robust accuracy among white-box attacks.
    - For black-box attacks, sPGD outperforms CS attack in terms of efficiency and performs better than RS attack in $\ell_1 \text{ and } \ell_2 $ models. Meanwhile, we acknowledge that sPGD's performance in $\ell_{\infty}$ models is weaker than RS attack. We argue that this may result from the existence of gradient masking. As future work, we will try to investigate the causes and improve the algorithm.
    - Lastly, we trained the PRN-18 model against our attack in two fashions: vanilla adversarial training (sAT) and TRADES (sTRADES). We showed that models trained by these two methods against sPGD yield the strongest robustness compared with other models including PGD$_0$-A and PGD$_0$-T. We consider these two models as strong baselines for $\ell_0$-adv models and defer a more thorough investigation on robust learning using sPGD as future work.

Reviewer comment (Experiments):
Many parameters in this paper need to be pre-defined. For example, ‘the current sparsity mask remains unchanged for three consecutive iterations, the continuous alternative fm will be randomly reinitialized for better exploration.‘ Why three consecutive iterations? Will choosing a different number affect the results? What is \alpha and \beta? Will \alpha and \beta affect the value of ‘three iterations’? Also for a small \lambda, it is unclear about how small the \lambda should be.

Author response:
When designing the algorithm, we follow the philosophy of AutoAttack to hard-code some hyper-parameters.
    This design is motivated by the observation that the model performance is not sensitive to these hyper-parameters, it facilitates the practitioners to avoid tuning additional hyper-parameters.
    
    Empirically, we found "Three consecutive iterations" works quite well. In addition, we conduct an ablation study on different reinitialization schemes. The result shown below \textbf{indicates that our method is quite robust to this hyperparameters.} 
    
    $\alpha$ and $\beta$ are the step sizes for updating magnitude tensor $p$ and continuous mask alternative $\widetilde{m}$, which are set to $1/4$ and $1/4 \times \sqrt{h \times w}$, respectively.
    Here, $h$ and $w$ are the height and width of the input image $x$ (both $h$ and $w$ are $32$ in CIFAR-10 dataset).
    In fact we have tested different values of these two hyper-parameters ($\alpha \in \{\frac{1}{16}, \frac{1}{8}, \frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1\}, \beta \in \{2,4,8,16,24,32\}$)，the results are included in Appendix A.3 (Table 9 and 10).
    The results show that \textbf{the performance is not sensitive to the change of the two hyper-parameters.}
    
    For the question regarding $\lambda$, we assume you were referring to the parameter $\gamma$ when updating the gradient of $\widetilde{m}$.
    In fact, it is common practices to use a small constant to avoid zero division. As a result, the choice of gamma will not greatly affect the performances. In our experiments, we set it to be $1e-10$.
    
***Table 1: Ablation study on the patience for reinitialization.***
 
|Reinitialization patience | 1 | 3 | 5 | 7 | 9|
|----|----|----|----|----|----|
Robust Accuracy | 20.2 | 20.0 | 20.1 |20.4 | 20.0|