Paper: Training Adversarially Robust SNNs with Gradient Sparsity Regularization (ICLR-2024)

Reviewer comment (Reproducibility):
Referring to Figure 1, what are the experiment settings used to generate the results? What is the SNN architecture? What is the adversarial attack algorithm?

Author response:
The experimental settings in Figure 1 are outlined as follows. The SNN architecture is a VGG-11 model. The adversarial attack employed is FGSM, with $\epsilon$ ranging from $0.00$ to $0.25$. For the random attack on SNNs, a perturbation $\delta$ is randomly generated to satisfy a uniform distribution within the hypercube $\Vert \delta \Vert_\infty \leqslant \epsilon$. Subsequently, $\delta$ is added to the input image $x$ and the pixel value of the perturbed image are clipped to the range of [0, 1]. The classification accuracy is evaluated for all $x+\delta$, where $x$ is drawn from the test set of CIFAR-10 (Figure 1(a)) and CIFAR-100 (Figure 1(b)). We have added these detailed settings in Sec. 4.1 of the revised version.

Reviewer comment (Novelty):
In Section 4.3, please discuss more clearly the differences between the approximation of the gradient regularization term employed in this paper and the related works.

Author response:
Regarding the approximation method for computing the gradient norm, both our paper and the related work ([Finlay & Oberman 2021](https://www.sciencedirect.com/science/article/pii/S2666827020300177)) use the finite differences technique. The only difference is that, we take the sign of the gradient as the difference direction while Finlay & Oberman took the sign direction divided by $\sqrt{N}$, where $N$ is the dimension of $x$. For other differences between the objectives of regularization terms in this paper and the related work, please refer to "To All Reviewers".

Reviewer comment (Reproducibility):
In Section 5, please discuss in detail all the parameters and hyperparameters used to conduct the experiments, as well as the tool flow. If possible, please provide the code in an online open-source repository.

Author response:
We would like to clarify that the setting of hyperparameters are discussed in Appendix C and D. Thanks for your suggestion! We have already uploaded the code in the revised supplementary material.

Reviewer comment (Evaluation):
From Table 1 we can infer that, while the proposed method can improve the adversarial robustness, there is a significant accuracy loss for clean inputs compared to related works. Please discuss the limitations and potential solutions to overcome this issue.

Author response:
Thanks for the valuable suggestion! The enhancement of adversarial robustness through SR is accompanied by a notable loss in accurac on clean images. In future work, we aim to achieve a better trade-off between classification accuracy and adversarial robustness. One possible avenue we plan to explore is the utilization of the simulated annealing algorithm in SR* to dynamically adjust the weight assigned to the regularization term, potentially leading to a more balanced outcome in terms of both accuracy and robustness. 

We have added these limitations and potential solutions in Section 6 of the revised paper.

Reviewer comment (Experiments):
The experiments have been conducted only on CIFAR-10 and CIFAR-100 dataset. It is recommended to make experiments also on event-based datasets, which are typical benchmarks for SNNs.

Author response:
We use the VGG-11 on DVS-CIFAR10 Dataset and compared Vanilla trained model and SR trained model. When evaluating, we perform the adversarial attack on preprocessed frames and add adversarial noise onto each frame. The robustness performance is shown in Table R8. The performance of SR trained model is significantly higher than the Vanilla model, which proves the effectiveness of SR method.

**Table R8. Performance of models on DVS-CIFAR10**
| Methods    | Clean | FGSM  | PGD10 | PGD30 | PGD50 |
|------------|-------|-------|-------|-------|-------|
| Vanilla    | 71.50 | 24.50 | 10.10 |  9.50 |  9.40 |
| SR         | 71.70 | 42.00 | 34.10 | 34.30 | 33.60 |

Reviewer comment (Evaluation):
Robustness evaluations of the SNNs are ambiguous in a weak way, and need much more rigor & depth.

Author response:
Thank you for your valuable insights. We sincerely appreciate your advice regarding our work. We apologize for any ambiguities in our description of the SR method that may have caused misunderstandings. In this rebuttal, we would like to first clarify the distinction between SR and adversarial training. Subsequently, we will address your concerns and provide responses to your questions. If you have any further suggestions or questions, please feel free to let us know.

**Q1: Clarification of the SR strategy**

In Theorem 1 of the main text, we have proved that the disparity between the adversarial vulnerability and random vulnerability is upper bounded by the sparsity of $\nabla_x f_y$. The main idea of our SR strategy is to regularize the value of $\Vert \nabla_x f_y(x,w) \Vert_0$. **It is important to note that in our approach, $f(\cdot, w)$ represents the model function, which differs from previous methods that utilize the multi-class calibrated loss function as a regularization term.** Here, the multi-class calibrated loss function $l(f(x,w),y)$, such as cross-entropy, is the loss function used to train a classifier where $y$ is the true label of $x$.  In other words, our proposed method introduces a regularization term that remains independent of the choice of loss function.

To achieve the regularization of $\Vert \nabla_x f_y(x,w) \Vert_0$, we use $\Vert \nabla_x f_y(x,w) \Vert_1$ to approximate the $\ell_0$ norm of the gradient and then use finite difference method to make the regularization of $\Vert \nabla_x f_y(x,w) \Vert_1$ computationally feasible. Theoretically, if direct backpropagation on $\Vert \nabla_x f_y(x,w) \Vert_1$ were feasible, finite differences could be bypassed. However, as reported in [Finlay & Oberman's work](https://arxiv.org/pdf/1905.11468.pdf), directly calculating the backpropagation of $\Vert \nabla_x f_y(x,w) \Vert_1$ on ANNs is very time-consuming. Additionally, we found that SNNs are not trainable with such double backpropagation approach.


We also find that the demonstration of Algorithm 1 in the main text is not clear enough, potentially leading readers to confuse our proposed SR algorithm with the adversarial training method and its variants. We would like to clarify that, as depicted in Algorithm 1 (Line 5 to Line 7), $\hat{x}^i=x^i + hd^i$ is fundamentally different from an adversarial example in terms of both numerical value and meanings. Actually, it is used to calculate the difference quotient
\begin{equation}
 \left| \frac{
        (f_{y^i}(\hat{x}^i, w)) - (f_{y^i}(x^i, w))
        }{
        h
        } \right|
\tag{3-1}
\end{equation}
which is the approximation of $\Vert \nabla_{x^i} (f_{y^i}(x^i, w)) \Vert_1$. While theoretically, $d^i$ can be arbitrary, it is set to be the sign of the gradient in this paper. This choice aligns with the objective of approximating the $\ell_1$ norm of the gradient, as explained in detail in Appendix B. 

Additionally, $h$ represents the step size used for the approximation, rather than the attack strength. It is recommended to choose a small value for $h$ in Equation (3-1) to achieve a more accurate estimation of the $\ell_1$ norm. As $h$ increases, the approximation results become less accurate, leading to a decrease in robustness. However, if $h$ were the attack strength in one-step FGSM, larger $h$ would contribute to better robustness (at the expense of decreased accuracy on clean images). The impact of $h$ in our training algorithm is different from the impact of $h$ as the attack strength in FGSM adversarial training. **In summary, the SR algorithm does not involve the calculation of adversarial samples, making it fundamentally different from adversarial training and its variations.**

We have revised the parts of the manuscript that may cause misunderstandings and added this clarification in Appendix G.

Reviewer comment (Novelty):
Proposed approach has several simplifications/assumptions on the ultimate gradient l0-norm regularization idea. At the end, the used training algorithm seems to become also similar to the well-known (clean) logit pairing approach [Kannan, Kurakin & Goodfellow, 2018]. At the very least, the objective simply uses adversarial examples for an output probability distribution regularizer, which has been the most fundamental form of adversarial training to date (see e.g., [Zhang et al. ICML 2019]). Hence all in all, I can not clearly see any innovation from the ML security side in this paper. Can the authors experimentally compare why their choice would be particularly any better in the case of SNNs, than using one of the other powerful adversarial training/regularization methods (e.g., TRADES [Zhang et al. ICML 2019] or adversarial logit pairing [Kannan, Kurakin & Goodfellow, 2018])?

Author response:
**Q2. Differences to adv. logit-based regularization methods. Comparsion with TRADES.**

We strongly believe that the differences between the SR strategy and adv. logit-based defense methods are substantial in terms of both algorithms and objectives. The SR strategy is a general defense method that directly constrain the inherent properties (gradients) of SNNs, while the adv. logit-based regularization method relies on the generated adversarial examples during the training phase. Moreover, we have performed an additional experiment on the CIFAR-10 dataset to directly compare the robustness of SNNs trained with the TRADES and SR strategies. The results are displayed in Table R9, where it is evident that the SNN trained with the SR strategy consistently outperforms the one trained with TRADES across all scenarios. This further reinforces the superiority of our approach over TRADES.

**Table R9. Comparsion of SNNs trained with TRADES and SR**
| Methods    | Clean | FGSM  | PGD10 | PGD30 | PGD50 |
|------------|-------|-------|-------|-------|-------|
| TRADES     | 90.04 | 30.25 | 11.86 |  9.28 | 8.84  |
| SR         | 86.57 | 34.79 | 15.94 | 12.27 | 11.70 |

Reviewer comment (Experiments):
For this attack ensemble to be meaningful and reveal any impact of gradient obfuscation, there should actually be cases where the surrogate gradient function is changing its width, rather than only its shape for fixed parameters. The width parameter $\gamma=1$ of the triangular surrogate should be accounted for. Authors should run an extensive evaluation with for instance $\gamma\in[0.1,3.0]$ in fine-grained steps of 0.1, and demonstrate that changing this parameter does not at all influence the capability of the adversary any better than using different surrogate gradient shapes. This part overall needs strong empirical justification.

Author response:
**Q5: Extensive evaluation with for instance $\gamma \in [0.1,3.0]$.**

Thanks for your valuable suggestion, we have applied the attack with $\gamma \in [0.1, 3.0]$ and Table R4 reported the results of a PGD10 attack on VGG-11 models with different training algorithms on the CIFAR-10 dataset. 

We compare three different attack combinations to evaluate the impact of different surrogate functions on the attack strength. We selecte RAT, PGD5-AT and SR*(PGD5+SR) model as the target models. For PGD10(w/o ensemble), we only use the Triangle-shaped surrogate function， which is identical to the one used in training. For PGD10 (w/ ensemble), we use the attack combination as described in the manuscript. For PGD10($\gamma$ from [0.1, 3.0]), we incorporate 30 different Triangle-shaped surrogate function with its $\gamma$ ranging from [0.1, 30], as suggested. 

We find that both ensemble attack methods significantly improve attack performance and mitigate the impact of gradient obfuscation. This indicates that both shape and width of the surrogate function can influence the capability of the adversary. In addition, we acknowledge that the PGD10($\gamma$ from [0.1, 3.0]) attack is slightly more effective than the ensemble attack used in this paper. However, it is important to note that the PGD10($\gamma$ from [0.1, 3.0]) attack uses a 30-fold fine-grained grid search over attack hyperparameters for each image, which is considerably more computationally expensive compared to the ensemble attack used in our paper. 

In conclusion, on one hand, we agree with reviewer HCoc's claim that both width and shape of the surrogate function can impact attack performance. We have included additional discussions about this in the limitation part. On the other hand, we would like to clarify that our contribution mainly lines in the defense algorithm, not the attack algorithm. In our paper, we use the same attack method for all comparisons to ensure a fair comparison of different methods. We believe that whether we use our current method or incorporate the attack combinations suggested by reviewer HCoc, it will not significantly influence the effectiveness of our defense strategy.

**Table R4. The classification accuracy (%) under the ensemble attack with different &gamma;**
| Attacks                   | RAT   | PGD-AT | SR*   |
|---------------------------|-------|--------|-------|
| PGD10 (w/o ensemble)      | 16.16 | 21.32  | 33.67 |
| PGD10 (w/ ensemble)       | 11.53 | 18.18  | 30.54 |
| PGD10 (γ from [0.1, 3.0]) | 11.87 | 16.16  | 27.06 |

Reviewer comment (Reproducibility):
Models were trained using PGD-5 adversarial examples, but the Algorithm 1 denotes adv. examples obtained via one-step FGSM. Which one is correct? Does this mean that PGD-5 is the general AT approach adopted in SR* models, but in any case the regularizer term was always obtained via FGSM? Needs clarification overall.

Author response:
**Q6: Models were trained using PGD-5 adversarial examples, but the Algorithm 1 denotes adv. examples obtained via one-step FGSM. Which one is correct?**

Thanks for pointing it out! We would like to clarify that in the SR model, we do not use adversarial examples obtained via one-step FGSM to conduct adversarial training, as explained in "Clarification of the SR strategy".  For SR* models, we employ PGD-5 as the general adversarial training approach, and the training loss function is
\begin{equation}
    \mathcal{L}(x^i, y^i,w) = \text{CE}(f(x_{adv}^i), y^i) + 
        \lambda_{\text{SR}}\left\Vert \nabla_{x_{adv}^i} f(x_{adv}^i,w)\right \Vert_1
    \tag{3-2}
\end{equation}
where $\left\Vert \nabla_{x^i} f(x^i,w)\right \Vert_1$ is approximated by the finite differences, and $x_{adv}^i$ is obtained by PGD-5.

Reviewer comment (Theory):
Proof of Thm 1 in Appendix A seems agnostic to any function f, regardless of being an SNN. However, it appears to implicitly make the assumption that the SNN uses direct input coding. Can the authors comment if these assumptions would also hold for SNNs that use Poisson input coding, or SNNs that do not necessarily use IF neurons (i.e., with membrane potential leak)? Wouldn't then the error that should be accounted for in the finite differences approximation would be large? Did the authors experiment with other types of more realistic SNNs at all?

Author response:
**Q8: If assumptions in this paper would also hold for SNNs that use Poisson input coding, or SNNs that use LIF?**

Thanks for pointing it out. Indeed, the SR strategy can be applied theoretically as long as SNN is differentiable through methods such as surrogate gradients. 

To demonstrate this point, we apply SR to SNNs with VGG-11 structure consisted of **LIF neurons** on the CIFAR-10 dataset. The experimental results shown in Table R5 clearly demonstrate the effectiveness of the SR* strategy in enhancing the robustness of SNNs with LIF neurons. Models trained with SR* exhibit significantly improved robustness against all PGD attacks compared to models trained with AT, with a 4% decline in classification accuracy on clean images.

**Table R5-1. Experiments on SNNs with LIF neurons**
| Models    | Clean | PGD10 | PGD30 | PGD50 |
|-----------|-------|-------|-------|-------|
| VGG11-AT  | 89.77 | 17.07 | 14.55 | 14.27 |
| VGG11-SR* | 85.60 | 30.80 | 28.52 | 28.38 |

We also trained two VGG-11 models on CIFAR-10 using **Poisson encoding**. During training, we use the same setting as reported in the main text with the only different being the input encoding. When evaluating robust performance, the Poisson input encoding can be treated as an input transformation and use the EOT method ([Athalye, Anish, Nicholas Carlini, & David Wagner](http://proceedings.mlr.press/v80/athalye18a.html)) to attack. According to the results in Table R5-2, the performance of SR model is higher than the Vanilla model and the robust performance of SNN trained with SR is even 15% higher than the vanilla SNN under PGD10 attack. That proves the effectiveness of the sparsity regularization.

**Table R5-2. Experiments on SNNs using Poisson Encoding**
| Methods    | Clean | FGSM  | PGD10 | 
|------------|-------|-------|-------|
| Vanilla    | 82.71 | 26.58 | 17.69 |
| SR         | 78.84 | 39.03 | 32.67 |

These experiments demostrate that our strategy also holds for SNNs with LIF neurons or with Poisson encoding.

Reviewer comment (Experiments):
Did the authors perform any simulations on dynamic vision sensor data where SNNs are designed to be more beneficial for?

Author response:
**Q9: Experiments on DVS datasets**

We use the VGG-11 architecture on DVS-CIFAR10 Dataset and compared Vanilla trained model and SR trained model. When evaluating, we perform the adversarial attack on preprocessed frames and add adversarial noise onto each frame. The robustness performance is shown in Table R6. The performance of SR trained model is significantly higher than the Vanilla model, which proves the effectiveness of SR method.

**Table R6. Performance of models on DVS-CIFAR10**
| Methods    | Clean | FGSM  | PGD10 | PGD30 | PGD50 |
|------------|-------|-------|-------|-------|-------|
| Vanilla    | 71.50 | 24.50 | 10.10 |  9.50 |  9.40 |
| SR         | 71.70 | 42.00 | 34.10 | 34.30 | 33.60 |

Reviewer comment (Experiments):
Since PGD-5 is used via BPTT during training for 200 epochs, than the authors should outline a table with the computational overhead (wall-clock time) of their approach, in comparison to simple AT or RAT with PGD-5 as well.

Author response:
**Q10: The computational overhead of SR and SR\*, in comparison to simple AT or RAT with PGD-5 as well.**

The computational costs for one epoch training of various algorithms, including vanilla, PGD5 AT, RAT, SR, and SR*(PGD5+SR), on CIFAR-10 using the VGG11 architecture are summarized in Table R6. From the table, we observe that single SR incurs a computational cost 1.5 times that of RAT but consumes less than half the time needed by PGD5 AT. The computational cost of SR* is the highest among all training algorithms since it combines both SR and AT. However, models trained with SR* acheive the best robustness compared to models trained with other methods. We have added this comparison in Appendix I.

**Table R7. The computational cost in one epoch of different training algorithms**
| Vanilla | PGD5 AT | RAT    |  SR    | SR*     |
| ------- | ------- | -----  | ------ | ------- |
| 65s     | 459s    | 134s   |  193s  | 583s    |

Reviewer comment (Experiments):
Figure 3 should also include adversarially trained models in comparison (not only Vanilla), since such SNNs are already implicitly inducing a similar behavior.

Author response:
**Q11: Include adversarially trained models in Figure 3.**

We have added the gradient distribution of AT-trained models with respect to the input image in Figure 3. The distribution of gradient values for SNNs-SR* is more concentrated around zero compared to that of vanilla SNNs and SNNs-AT.

Reviewer comment (Theory):
Eq (3) & (4) are already defined as solutions of an adversarial example in the l_infty norm (since the sign function is used). Therefore at the end of Sec 3.2, authors should correct the "l_p norm", since p is already infty in this setting.

Author response:
**Q12: Minor comments**

Thanks for pointing it out. We correct the "$\ell_p$ norm" at the end of Sec 3.2 in the revised version.

Reviewer comment (Theory):
In Formula 10, I think the left part tries to compute the gradient of input, however, the right parts compute the gradient of last layer.

Author response:
Thanks for pointing it out. We would like to clarify the notations used in Formula (10) as follows. Let $x$ denote the image, and $\{x[1], x[2], \dots, x[T]\}$ represent the input image series. In our paper, we use $x[t]=x$ for all $t=1,\dots,T.$ The network is denoted by $f$ with parameters $w$. The output of the network $f$ is a vector $f(x,w)\in \mathbb{R}^{N \times 1}$, where $N$ represents the number of classes. The output vector $f(x,w)$ is obtained by applying the softmax function to the output of the last layer, i.e.  
\begin{equation}  
    f(x, w) = softmax \left(   
        \sum_{t=1}^T s_1^L[t], \dots, \sum_{t=1}^T s_N^L[t]  
    \right).  
    \tag{2-1}  
\end{equation}  
Therefore, the $y^{th}$ component of $f(x, w)$ where $y$ represents the true label is  
\begin{equation}  
    f_y(x, w) = softmax_y \left(   
        \sum_{t=1}^T s_1^L[t], \dots, \sum_{t=1}^T s_N^L[t]  
    \right).  
    \tag{2-2}  
\end{equation}  
According to the chain rule, the gradient of $f_y$ with respect to the input $x$ is  
\begin{equation}  
    \begin{split}  
        \nabla_x f_y(x,w) &=  \frac{\partial softmax_y }{\partial \sum_t s_1^L[t]}  
                     \cdot  
                    \nabla_x \left( \sum_t s_1^L[t] \right) + \dots  +   
                    \frac{\partial softmax_y }{\partial \sum_t s_N^L[t]}  
                     \cdot \nabla_x \left( \sum_t s_N^L[t] \right).  
    \end{split}  
    \tag{2-3}  
\end{equation}  
In Equation (2-3), the gradient of the $i^{th}$ component $\nabla_x \left( \sum_t s_i^L[t] \right)$ in the last layer with respect to $x$ can be further expresses as  
\begin{equation}  
    \nabla_x \left( \sum_t s_i^L[t] \right) =  \sum_t \nabla_x s_i^L[t]   
    = \sum_t \sum_{\tilde{t}=1}^t \nabla_{x[\tilde{t}]} s_i^L[t] .  
    \tag{2-4}  
\end{equation}  
Finally, the gradient $\nabla_x f_y(x,w)$ is written as  
\begin{equation}  
    \nabla_x f_y(x,w) = \sum_{i=1}^N \left( \frac{\partial softmax_y  
                    }{  
                        \partial \sum_t s_i^L[t]  
                    }  
                    \left(  
                        \sum_{t=1}^T \sum_{\tilde{t}=1}^t \nabla_{x[\tilde{t}]} s_i^L[t]  
                    \right) \right).  
    \tag{2-5}  
\end{equation}  

We revise Formula (10) in the latest version, and provide the detailed derivation of the formula in Appendix J.

Reviewer comment (Experiments):
The paper conducts the experiments on the CIFAR-10 and CIFAR-100 datasets. Is it possible to conduct the experiments on more large-scale datasets, such as ImageNet or ImageNet-Tiny datasets?

Author response:
**Q1: Experiments on more large-scale datasets**

We trained three VGG-11 models on the TinyImageNet dataset: a vanilla model with no robustness enhancement (Vanilla), an FGSM [eps=2/255] adversarially trained model (AT), and a model using both FGSM [eps=2/255] adversarial training and sparsity regularization (SR*). Throughout training and evaluation, we cropped all images to a resolution of 64x64, which is more complex compared to the 32x32 images from CIFAR-10 datasets. 

Under these conditions, the SR* model exhibited the best performance, achieving an accuracy of 10.82% under PGD50 attack and 14.89% accuracy under FGSM attack. In comparison to the pure AT model, the SR* model showed significantly better robustness against various attacks, highlighting the effectiveness of our method. 

**Table R1. Performance on TinyImageNet dataset**
| ANN_Models           | Clean | FGSM  | PGD10 | PGD30 | PGD50 |
|--------------        |-------|-------|-------|-------|-------|
| Vanilla                       | 58.23 | 0.64  | 0.00  | 0.00  | 0.00  |
| AT (FGSM, eps=2/255)          | 52.17 | 11.06 | 6.09  | 5.25  | 4.96  |
| SR* (FGSM, eps=2/255 λ=0.001) | 43.44 | 14.89 | 11.49 | 10.97 | 10.82 |

Reviewer comment (Theory):
It requires some assumptions for Theorem 1 to be true, such as the function $f$ should be differentiable, and $\epsilon$ should be small enough, could you add these assumptions to Theorem 1?

Author response:
**Q2: Add assumptions to Theorem 1**

Thanks for your suggestion! In our revised version, we have incorporated the assumptions that the function $f$ is differentiable and that $\epsilon$ is sufficiently small into Theorem 1.

Reviewer comment (Writing):
Line 4 in Proposition 1, Page 6,  The proof is provides -->  The proof is provided.

Author response:
**Q3. Other minor questions.**

Thanks for pointing it out! We have revised these typos and updated our paper.

Reviewer comment (Theory):
The last line in (17), Page 15,  $\epsilon$ -->  $\epsilon^2$.

Author response:
**Q3. Other minor questions.**

Thanks for pointing it out! We have revised these typos and updated our paper.