Extreme Miscalibration and the Illusion of Adversarial Robustness (2024)

Abstract: Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples. Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. Finally, we show how the temperature can be scaled during training to improve genuine robustness.

## introduction deep learning natural language processing (nlp) models are able to perform well in a range of tasks @xcite . however, these nlp models are susceptible to adversarial attacks, where perturbing clean input text samples slightly (accidentally or maliciously by an adversary) can lead to a nlp model misclassifying the perturbed input @xcite . however, the emergence of the adversarial training (at) paradigm @xcite has shown some success in training models to be more robust to these small adversarial perturbations. here, the traditional training process is adapted to minimize the empirical risk associated with a "robustness loss" as opposed to the risk associated with the standard loss for clean input samples. the robustness loss is the standard loss applied to the worst-case (loss maximizing) adversarial sample for each training sample. in nlp, due to the discrete nature of the text, this adversarial training min-max formulation is particularly challenging as the inner maximization is computationally expensive @xcite . nevertheless, a variety of approaches have been proposed in the robustness literature, ranging from augmentation of the training set with adversarial examples for a specific model, to sophisticated token-embedding space optimizations for the inner maximization step @xcite . although many nlp at methods appear effective in boosting model robustness, we argue that, in some cases, the observed improvement is merely an illusion of robustness (ior) that can be easily circumvented. in the computer vision literature, @xcite show that certain modeling decisions can give rise to such an illusion via gra-dient obfuscation @xcite . we build on these findings by showing that iors can also emerge easily in nlp at due to highly miscalibrated models. concretely, we: in contrast to @xcite , we do not perform model distillation but only training-time temperature scaling as the adversarial defense. in light of our findings, we urge the community to perform test-time temperature scaling during all robustness evaluations to ensure that the observed robustness is genuine and not merely an illusion. to our knowledge, we are the first to demonstrate this phenomenon in nlp models and propose trainingtime temperature scaling as an adversarial defense. ## the illusion of robustness certain modeling approaches can lead to an illusion of robustness (ior). in computer vision, it is shown that obfuscating gradients @xcite through shattered, stochastic or exploding/vanishing gradients; or masking gradients @xcite can lead to such an ior -the model appears robust to adversarial attacks. we build on these works and argue that (un)intentional extreme model miscalibration is an example of a realistic cause of ior. the robustness gains observed for traditional nlp at approaches (equation 3 ), may not always be due to inherent robustness gains, but can be a consequence of extreme model miscalibration. this miscalibration can induce extreme confidence predictions, such that the model's predicted class confidence p θ(ĉ|x) is either very high (overconfident) or very low (underconfident). figure 1 (using a standard nlp model, test dataset and adversarial attack described in section 3.3) demonstrates that highly miscalibrated models with extreme confidence values in the predicted class (around 1.0 for overconfident models or 1/c, where c is the number of classes for underconfident models) are significantly more "robust" to adversarial attacks. this apparent increase in robustness of extremely miscalibrated models can be explained. for both underconfident and overconfident models, the predicted class confidence has very little variance for different input sequences, x, consequently, any whitebox adversarial attack approach looking to exploit gradients or even a blackbox attack approach measuring the sensitivity of the predicted probability, has a small confidence range to observe. the implication is that the impact of any proposed perturbation gives a very noisy signal to its actual effect on the output. as a result, the adversarial attack search process will converge extremely slowly or outright fail to find the desired adversarial perturbation direction d. we verify this hypothesis empirically in appendix e.7, corroborating related computer vision literature on gradient obfuscation @xcite . we first demonstrate how to induce an ior by explicitly miscalibrating models at test-time, before discussing how this can happen unintentionally during training. ## piercing the illusion having demonstrated how to induce an ior from the developer or defender's perspective, we now discuss how to nullify it as an adversary. although the following approaches involve modifying aspects of the model's output at test-time, these modifications are only used to create/find adversarial examples, which can then be applied to the original, un-modified model served by the model developer. ## raising the training temperature for genuine robustness section 3 showed that it is easy to unintentionally develop at schemes that do not yield true robustness gains but instead induce ior. the success of temperature scaling at interfering with an attack's search process poses the natural question: can a similar effect be induced in training such that it cannot be nullified, thus improving true robustness? we now present a simple modification to standard training that boosts the true robustness of nlp models to unseen attacks. we consider an attack to be unseen when its adversarial examples are not used in adversarial training. in our experiments, this refers to all at methods other than dg-aug. ## conclusion nlp models are susceptible to adversarial attacks, where small changes in the input cause the model to predict incorrectly. many adversarial training (at) approaches have been proposed to induce robustness to adversarial attacks. in this work, we argued that the observed robustness gains may not be due to true increases in model robustness. we demonstrated how at schemes can unknowingly create highly miscalibrated models that disrupt common adversarial attack search methods by obfuscating gradients, yielding up to three-fold perceived robustness gains. however, this is merely an illusion of robustness (ior). we proposed simple methods an adversary could use to circumvent such gains. specifically, we showed how to perform test-time temperature scaling to mitigate disruptions to the adversarial attack search processes and pierce the ior. hence, we strongly recommend all adversarial robustness evaluations incorporate adversarial temperature scaling to ensure any observed robustness gain is genuine and not an illusion. finally, we proposed a practical training-time modification to increase a model's genuine robustness to unseen adversarial attacks and demonstrated its efficacy alone and in combination with other at methods.