In and Out-of-Domain Text Adversarial Robustness via Label Smoothing (2023)

Abstract: Recently it has been shown that state-of-the-art NLP models are vulnerable to adversarial attacks, where the predictions of a model can be drastically altered by slight modifications to the input (such as synonym substitutions). While several defense techniques have been proposed, and adapted, to the discrete nature of text adversarial attacks, the benefits of general-purpose regularization methods such as label smoothing for language models, have not been studied. In this paper, we study the adversarial robustness provided by label smoothing strategies in foundational models for diverse NLP tasks in both in-domain and out-of-domain settings. Our experiments show that label smoothing significantly improves adversarial robustness in pre-trained models like BERT, against various popular attacks. We also analyze the relationship between prediction confidence and robustness, showing that label smoothing reduces over-confident errors on adversarial examples.

## introduction neural networks are vulnerable to adversarial attacks: small perturbations to the input ,which do not fool humans @xcite @xcite . in nlp tasks, previous studies @xcite @xcite demonstrate that simple word-level text attacks (synonym substitution, word insertion/deletion) easily fool state-of-the-art models, including pre-trained transformers like bert @xcite . further, it has recently been shown models are overconfident 1 on examples which are easy to attack @xcite and indeed, such over-confident predictions plague much of modern deep learning @xcite @xcite . label smoothing is a regularization method that has been proven effective in a variety of applications, and modalities @xcite @xcite . importantly, it has been shown to reduce overconfident predictions and produce better confidence calibrated classifiers @xcite @xcite @xcite . in this work, we focus on the question: does label smoothing also implicitly help in adversarial robustness? while there has been some investigation in this direction for adversarial attacks in computer vision, @xcite @xcite , there is a gap in understanding of whether it helps with discrete, text adversarial attacks used against nlp systems. with the increasing need for robust nlp models in safety-critical applications and a lack of generic robustness strategies, foot_0 there is a need to understand inherent robustness properties of popular label smoothing strategies, and the interplay between confidence and robustness of a model. in this paper, we extensively study standard label smoothing and its adversarial variant, covering robustness, prediction confidence, and domain transfer properties. we observe that label smoothing provides implicit robustness against adversarial examples. particularly, we focus on pre-trained transformer models and test robustness under various kinds of black-box and white-box word-level adversarial attacks, in both in-domain and out-ofdomain scenarios. our experiments show that label smoothing (1) improves robustness to text adversarial attacks (both black-box and white-box), and (2) mitigates over-confident errors on adversarial textual examples. analysing the adversarial exam-ples along various quality dimensions reveals the remarkable efficacy of label smoothing as a simple add-on robustness and calibration tool. ## experiments in this section, we present a thorough empirical evaluation on the effect of label smoothing on adversarial robustness for two pre-trained transformer models: bert and its distilled variant, dbert, which are the victim models. 4 we attack the victim models using tf, bae, and semattack. for each attack, we present results on both the standard models and the label-smoothed models on various classification tasks: text classification and natural language inference. for each dataset we evaluate on a randomly sampled subset of the test set (1000 examples), as done in prior work @xcite @xcite . we evaluate on the following tasks, and other details about the setting is in appendix a.8: ## conclusion we presented an extensive empirical study to investigate the effect of label smoothing techniques on adversarial robustness for various nlp tasks, for various victim models and adversarial attacks. our results demonstrate that label smoothing imparts implicit robustness to models, even under domain shifts. this first work on the effects of ls for text adversarial attacks, complemented with prior work on ls and implicit calibration @xcite , is an important step towards developing robust, reliable models. in the future, it would be interesting to explore the combination of label smoothing with other regularization and adversarial training techniques to further enhance the adversarial robustness of nlp models.