Paper: Leveraging characteristics of the output distribution for identifying adversarial audio examples (ICLR-2024)

Reviewer comment (Experiments):
Detection performance is only evaluated on limited adversarial attack methods. The authors only evaluate their method on C&W attack and Psychoacoustic attack. More attack methods like [1] [2] and even some black box methods like FAKEBOB [3] are still needed to be included to prove the general performance of the detection method.

Author response:
Thanks for your suggestions, we extended our analysis to three more attacks: PDG and those proposed in [1] and [2]. We evaluated the detection performance of our classifiers across diverse attacks. Our findings reveal that our GCs utilizing mean-median characteristic and NNs exhibited successful transferability to the Psychoacoustic attack. Moreover, when exposed to untargeted attacks, they outperformed the baseline in comparison to PGD and Kenansville. For enhanced clarity, we introduced a new analysis, presented in the results section 5.3 on pages 7-8 and table 6 of our revised manuscript.

We did not include FAKEBOB in our analysis, since it is designed specifically for speaker recognition systems, which falls outside the scope of our research.

Reviewer comment (Experiments):
Lack of comparison with other audio adversarial example detection methods like [4].

Author response:
Thanks for your input. We included [4] as a baseline in our analysis, but found that it is less effective than our method. Please refer to section 5.3, page 8, and table 4 in our latest manuscript.

Reviewer comment (Experiments):
The adaptive attacks lower your detection accuracy considerably. Have you looked into ways to make the classifiers more robust? For example, by using adversarial training or adding noise to the features.

Author response:
During the training of all models, to improve generalization, we applied standard data augmentation techniques provided in SpeechBrain: corruption with random samples from a noise collection, removing portions of the audio, dropping frequency bands, and resampling the audio signal at a slightly different rate. We described in the paragraph “ASR system” that is in Section 5.1, Page 5.

Reviewer comment (Evaluation):
Have you evaluated the computational overhead added by extracting the distributional features and running the classifiers? Is this method efficient enough for real-time usage in production systems?

Author response:
Thanks for asking, we performed experiments to measure the total time the system takes to predict 100 audio clips, utilizing an NVIDIA A40 with a memory capacity of 48 GB. As a result, running the assessment with our detectors took approximately an extra 18.74 ms per sample, therefore the proposed method is suitable for real-time usage. We have incorporated this information into the paper, and provided additional details in Appendix A.3 on page 16.

Reviewer comment (Evaluation):
You use simple summary statistics to represent the output distributions. What prevents more sophisticated adaptive attacks that preserves these summary statistics but still fools the ASR system?

Author response:
Fortunately, we have successfully completed the experiments with the suggested adaptive attack method. We employed a slightly modified loss function to introduce a penalty, aiming to minimize the statistical divergence from the benign data distribution. For this, we calculated for the same time step the KLD between the output distribution given the benign data $x$ and its adversarial counterpart $\hat x$. 
This adjustment resulted in improved noise levels with higher SNR values compared to other settings. Nevertheless, detectors remain capable of distinguishing adversarial examples from benign data, as detailed in Table 12 of the appendix A.1 on page 14-15 (refer to setting number 6).

Reviewer comment (Experiments):
Your defense relies on statistical metrics like median and maximum probability diverging for adversarial examples. Have you explored attacks that explicitly optimize to minimize statistical distance from the benign data distribution? This could make the adversarials harder to detect.

Author response:
Fortunately, we have successfully completed the experiments with the suggested adaptive attack method. We employed a slightly modified loss function to introduce a penalty, aiming to minimize the statistical divergence from the benign data distribution. For this, we calculated for the same time step the KLD between the output distribution given the benign data $x$ and its adversarial counterpart $\hat x$. 
This adjustment resulted in improved noise levels with higher SNR values compared to other settings. Nevertheless, detectors remain capable of distinguishing adversarial examples from benign data, as detailed in Table 12 of the appendix A.1 on page 14-15 (refer to setting number 6).

Reviewer comment (Theory):
Moreover, can the adversarial optimization problem be formulated to reduce divergence from the benign data distribution, while still fooling the ASR system? What are the challenges in constructing such "distribution-aware" attacks?

Author response:
Fortunately, we have successfully completed the experiments with the suggested adaptive attack method. We employed a slightly modified loss function to introduce a penalty, aiming to minimize the statistical divergence from the benign data distribution. For this, we calculated for the same time step the KLD between the output distribution given the benign data $x$ and its adversarial counterpart $\hat x$. 
This adjustment resulted in improved noise levels with higher SNR values compared to other settings. Nevertheless, detectors remain capable of distinguishing adversarial examples from benign data, as detailed in Table 12 of the appendix A.1 on page 14-15 (refer to setting number 6).

Reviewer comment (Evaluation):
The reason behind the selected statistics can be further motivated. Why are these statistical features selected? A related question is why the generated adaptive adversarial audio examples are noisier when optimizing with respect to relevant feature?

Author response:
Q: Why are these statistical features selected? 

A: Previous work indicated the high potential of statistics of the output distribution that implement uncertainty measures: on the one hand, the mean entropy was turned into a defense method against adversarial attacks on hybrid ARS system with limited vocabulary, as discussed in [1].  On the other hand, the Kullback-Leibler divergence between the distributions in consecutive steps was used to assess the reliability of ARS systems, see [2]. We therefore aimed to investigate if this also applies for state-of-the-art end-to-end ARS systems. We expanded our analysis to simple characteristics (like max, min, median) and were surprised ourselves that they lead to sometimes even better detection results. 

[1] Däubener et al., Detecting adversarial examples for speech recognition via uncertainty quantification. In Proc. Interspeech 2020, pp.4661–4665, 2020.

[2] Meyer et al., Performance monitoring for automatic speech recognition in noisy multi-channel environments. In 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 50–56, 2016.
\
\
\
Q: Why the generated adaptive adversarial audio examples are noisier when optimizing with respect to relevant feature? 

A: The adaptive attack contains two (potentially competing) loss functions, which makes the optimization problem harder. We also were successful in generating less noisy examples with the adaptive attack (see Appendix A.1, page 14) which however were less often fooling the detection model. We added a sentence to make this clearer in the main paper.

Reviewer comment (Experiments):
Regarding the generalization of the proposed detection, the transferability of the detection can be further clarified. About intra-model generalization, will the detection model that is trained on one specific kind of adversarial example be generalizable to other types of adversarial examples? This point needs to be clarified since it may weaken the threat model that the detector needs to know the type of the adversarial attack beforehand. About inter-model generalization, will a detector trained on one ASR model be able to detect adversarial examples that are generated on a different ASR model? It would be great if the authors can clarify the generalization of the proposed method.

Author response:
Q: Will be the detection model that is trained on one specific kind of adversarial example be generalizable to other types of adversarial examples? 

A: Yes. First, note that the construction of Gaussian classifiers can be done only with benign examples. While we also investigated the performance based on the best characteristic picked on a validation set, we also found that some characteristics perform very well for different attack types and thus can be chosen without any adversarial data. Only benign data for picking the threshold is needed. Moreover, we conducted experiments on intra-model generalization for the neural networks, investigating if a detection model trained on the C&W attack can be applied as a defense method against the other targeted (i.e., Psychoacoustic) and three newly added untargeted attacks. We found that our GCs based on the mean-median characteristic and NNs demonstrated effective transferability to the Psychoacoustic attack. Additionally, when subjected to untargeted attacks, they performed better than the baseline against PGD and Kenansville attacks. To make this more clear, we incorporated a new analysis displayed by the results in section 5.3/pages 7-8 /Table 6 of our updated manuscript.
\
\
\
Q: Will a detector trained on one ASR model be able to detect adversarial examples that are generated on a different ASR model?

A: We don't expect our detectors to generalize between models, since the statistics for different models differ. Previous research has demonstrated almost no transferability of adversarial examples between speech models, as indicated in [1]. This lack of transferability, especially in the context of targeted optimization attacks, suggests that adversarial examples crafted to target a specific model lose their efficacy when applied to a new ASR model, posing a limited threat. We verified this claim with our models and obtained similar results, with almost no transferability observed between models. These results are reported in Appendix A.11 on pages 30-31.

[1] Abdullah et al., Sok: The faults in our ASRs: An overview of attacks against automatic speech recognition and speaker identification systems. 2021 IEEE Symposium on Security and Privacy (SP), pp. 730–747, 2020.

Reviewer comment (Experiments):
About the adaptive attack, have the authors considered other types of attacks that may decrease the noise level of the adversarial audio examples? I really appreciate that the authors provide experiments on adaptive attacks, which definitely makes the claims stronger. It would be great if the authors could clarify the specific efforts that have been made to control the noise level.

Author response:
Q: About the adaptive attack, have the authors considered other types of attacks that may decrease the noise level of the adversarial audio examples? 

A: Yes, we did. As described in details in Appendix A.1 (page 14) we experimented with several different settings that resulted in lower noise levels but also in less affective attacks. We added a sentence to make this clear in the main paper.