ADDMU : Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation (2022)

Abstract: Adversarial Examples Detection (AED) is a crucial defense technique against adversarial attacks and has drawn increasing attention from the Natural Language Processing (NLP) community. Despite the surge of new AED methods, our studies show that existing methods heavily rely on a shortcut to achieve good performance. In other words, current search-based adversarial attacks in NLP stop once model predictions change, and thus most adversarial examples generated by those attacks are located near model decision boundaries. To surpass this shortcut and fairly evaluate AED methods, we propose to test AED methods with F ar B oundary ( FB ) adversarial examples. Existing methods show worse than random guess performance under this scenario. To overcome this limitation, we propose a new technique, ADDMU , a dversary d etection with d ata and m odel u ncertainty, which combines two types of uncertainty estimation for both regular and FB adversarial example detection. Our new method outperforms previous methods by 3.6 and 6.0 AUC points under each scenario. Finally, our analysis shows that the two types of uncertainty provided by ADDMU can be leveraged to characterize adversarialexamples and identify the ones that contribute most to model’s robustness in adversarial training.

## introduction deep neural networks (dnn) have achieved remarkable performance in a wide variety of nlp tasks. however, it has been shown that dnns can be vulnerable to adversarial examples @xcite @xcite , i.e., perturbed examples that flip model predictions but remain imperceptible to humans, and thus impose serious security concerns about nlp models. to improve the robustness of nlp models, different kinds of techniques to defend against adversarial examples have been proposed @xcite . in this paper, we study aed, which aims to add a detection module to identify and reject malicious inputs based on certain characteristics. different from adversarial training methods @xcite which require re-training of the model with additional data or regularization, aed operates in the test time and can be directly integrated with any existing model. despite being well explored in the vision domain @xcite , aed started to get attention in the field of nlp only recently. many works have been proposed to conduct detection based on certain statistics @xcite @xcite . specifically, @xcite propose a benchmark for aed methods and a competitive baseline by robust density estimation. however, by studying examples in the benchmark, we find that the success of some aed methods relies heavily on the shortcut left by adversarial attacks: most adversarial examples are located near model decision boundaries, i.e., they have small probability discrepancy between the predicted class and the second largest class. this is because when creating adversarial data, the searching process stops once model predictions changed. we illustrate this finding in section 2.2. to evaluate detection methods accurately, we propose to test aed methods on both regular adversarial examples and far-boundary (fb) foot_0 adversarial examples, which are created by continuing to search for better adversarial examples till a threshold of probability discrepancy is met. results show that existing aed methods perform worse than random guess on fb adversarial examples. @xcite recognize this limitation, but we find that this phenomenon is more severe than what is reported in their work. thus, an aed method that works for fb attacks is in need. we propose addmu, an uncertainty estimation based aed method. the key intuition is based on the fact that adversarial examples lie off the manifold of training data and models are typically uncertain about their predictions of them. thus, although the prediction probability is no longer a good uncertainty measurement when adversarial examples are far from the model decision boundary, there exist other statistical clues that give out the 'uncertainty' in predictions to identify adversarial data. in this paper, we introduce two of them: data uncertainty and model uncertainty. data uncertainty is defined as the uncertainty of model predictions over neighbors of the input. model uncertainty is defined as the prediction variance on the original input when applying monte carlo dropout (mcd) @xcite to the target model during inference time. previous work has shown that models trained with dropout regularization @xcite approximate the inference in bayesian neural networks with mcd, where model uncertainty is easy to obtain @xcite . given the statistics of the two uncertainties, we apply p-value normalization @xcite and combine them with fisher's method @xcite to produce a stronger test statistic for aed. to the best of our knowledge, we are the first work to estimate the uncertainty of transformer-based models @xcite for aed. the advantages of our proposed aed method include: 1) it only operates on the output level of the model; 2) it requires little to no modifications to adapt to different architectures; 3) it provides an unified way to combine different types of uncertainties. experimental results on four datasets, four attacks, and two models demonstrate that our method outperforms existing methods by 3.6 and 6.0 in terms of auc scores on regular and fb cases, respectively. we also show that the two uncertainty statistics can be used to characterize adversarial data and select useful data for another defense technique, adversarial data augmentation (ada). the code for this paper could be found at https://github.com/uclanlp/ advexdetection-addmu ## a diagnostic study on aed methods in this section, we first describe the formulation of adversarial examples and aed. then, we show that current aed methods mainly act well on detecting adversarial examples near the decision boundary, but are confused by fb adversarial examples. ## adversary detection with data and model uncertainty (addmu) given the poor performance of previous methods on fb attacks, we aim to build a detector that can handle not only regular but also fb adversarial examples. we propose addmu, an uncertainty estimation based aed method by combing two types of uncertainty: model uncertainty and data uncertainty. we expect the adversarial examples to have large values for both. the motivation of using uncertainty is that models can still be uncertain about their predictions even when they assign a high probability of predicted class to an example. we describe the definitions and estimations of the two uncertainties, and how to combine them. ## experiments we first describe the experimental setup (section 4.1), then present our results on both regular and fb aed (section 4.2). results show that our addmu outperforms existing methods by a large margin under both scenarios. ## characterize adversarial examples in this section, we explore how to characterize adversarial examples by the two uncertainties. mu-du data map plotting a heatmap with mu on x-axis and du on y-axis, we visualize data in terms of the two uncertainties. we show in figure 2 the heatmaps with natural data, fb and regular adversarial examples generated from three attacks on three datasets (agnews tf, yelp bae, snli pruthi). the performance of addmu varies on the three attacks, as shown on the left of figure 2 . we find that natural examples center on the bottom left corner of the map, representing low mu and du values. this phenomenon does not vary across datasets. whereas for fb and regular adversarial examples, they have larger values on at least one of the two uncertainties. when addmu performs best (agnews tf, the first row), the center of adversarial examples in the mu-du map is relatively rightward and upward compared to other cases. for maps on the third row, the shadow stretches along the mu axis, indicating that pruthi examples on snli have relatively large mu values. identifying informative ada data ada is another adversarial defense technique, which augments the training set with adversarial data and re-train the victim model to improve its robustness. in this part, we show that our addmu provides information to select adversarial data that is more beneficial to model robustness. we test it with tf on sst-2. the procedure is as follows: since sst-2 only has public training and validation sets, we split the original training set into training (80%) and validation set (20%), and use the original validation set as test set. we first train a model on the new training set. then, we attack the model on validation data and compute du and mu values for each adversarial sample. we sort the adversarial examples according to their du and mu values and split them by half into four disjoint sets: hdhm (high du, high mu), hdlm (high du, low mu), ldhm (low du, high mu), and ldlm (low du, low mu). we augment the clean training set with each of these sets and retrain the model. as a baseline, we also test the performance of augmenting with all the adversarial examples generated from the validation set (all). we report clean accuracy (clean %), the number of augmented data (#aug), attack success rate (asr), and the average query number (#query) for each model. the results are in ## related work adversarial detection. adversarial examples detection has been well-studied in the image domain @xcite @xcite @xcite @xcite . our work aligns with @xcite ; @xcite ; @xcite that introduce uncertainty estimation or perturbations as features to detect adversarial examples. we postpone the details to appendix i, but focus more on the aed in nlp domain. in the nlp domain, there are less work exploring aed. @xcite propose disp that learns a bert-based discriminator to defend against adversarial examples. @xcite propose a word-level detector fgws that leverages the model confidence drop when replacing infrequent words in the input with frequent ones and surpass disp. @xcite combat character-level attacks with word-recognition models. more recently, @xcite propose a robust density estimation baseline and a benchmark for evaluating aed methods. there are other works like @xcite ; @xcite ; @xcite ; @xcite , that leverage other features or train a detector. we show limitations of these works on fb adversarial examples and propose our addmu that overcomes this limitation. other defenses against attacks. aed is a category of approaches to defending against adversarial attacks. other methods are also considered. @xcite ; @xcite ; @xcite do ada that augments original training datasets with adversarial data for better robustness. @xcite ; @xcite ; @xcite conduct adversarial training which is formulated as a min-max problem. recently, several works perform certified robustness defense with either interval bound propagation @xcite @xcite , or randomized smoothness @xcite . in this work, we connect our aed method with ada by selecting more informative data to augment. ## conclusion we proposed addmu, an uncertainty-based approach for both regular and fb aed. we began by showing that existing methods are significantly affected by fb attacks. then, we show that addmu is minimally impacted by fb attacks and outperforms existing methods by a large margin. we further showed addmu characterizes adversarial data and provides information on how to select useful augmented data for improving robustness.