Distinguishing Non-natural from Natural Adversarial Samples for More Robust Pre-trained Language Model (2022)

Abstract: Recently, the problem of robustness of pre-trained language models (PrLMs) has received increasing research interest. Latest studies on adversarial attacks achieve high attack success rates against PrLMs, claiming that PrLMs are not robust. However, we find that the adversarial samples that PrLMs fail are mostly non-natural and do not appear in reality. We question the validity of the current evaluation of robustness of PrLMs based on these non-natural adversarial samples and propose an anomaly detector to evaluate the robustness of PrLMs with more natural adversarial samples. We also investigate two applications of the anomaly detector: (1) In data augmentation, we employ the anomaly detector to force generating augmented data that are distinguished as non-natural, which brings larger gains to the accuracy of PrLMs. (2) We apply the anomaly detector to a defense framework to enhance the robustness of PrLMs. It can be used to defend all types of attacks and achieves higher accuracy on both adversarial samples and compliant samples than other defense frameworks.

## introduction pre-trained language models (prlms) have achieved state-of-the-art performance across a wide variety of natural language understanding tasks @xcite @xcite . most works of prlms mainly focus on designing stronger model structures and training objectives to improve the accuracy or training efficiency. however, in real industrial applications, there exist noises that can mislead the predictions of prlms @xcite , which raise potential security risks and limit the application efficacy of prlms in practice. to solve this challenge, studies around the robustness of prlms have received increasing research interest. recent studies demonstrated that, due to the lack of supervising signals and data noises in the pre-training stage, prlms are vulnerable to adversarial attacks, which can generate adversarial samples to fool the model @xcite . a variety of attack algorithms have been proposed to use spelling errors @xcite , synonym substitutions @xcite , phrase insertions @xcite or sentence structure reconstructions @xcite to generate adversarial samples. some of these attack algorithms have achieved an over 90% attack success rate on prlms @xcite . thus they claim that existing prlms are not robust. however, we investigate the adversarial samples on which prlms fail, and find that most of them are not natural and fluent, thus can be distinguished by humans. these samples are unlikely to appear in reality and are against the principle that adversarial samples should be imperceptible to humans @xcite . therefore it is not reasonable to judge the robustness of prlms based on these non-natural adversarial samples. by adopting a prlm-based anomaly detector and a two-stage training strategy, we empirically demonstrate that most of the nonnatural adversarial samples can be detected by the machine. furthermore, we adopt the anomaly score (the output probability of the anomaly detector) as a constraint metric to help adversarial attacks generate more natural samples. under this new constraint of generating natural samples, the attack success rates of existing attack methods sharply decrease. these experimental results demonstrate that the robustness of prlms is not as fragile as previous works claimed. then we explore two application scenarios of the anomaly detector. firstly, we wonder whether the anomaly detection can generalize to other appli-cations using artificially modified sentences. thus we think of the data augmentation scenario. the objective of data augmentation is to increase the diversity of training data without explicitly collecting new data @xcite . for an original sequence and a data augmentation technique, there exist many possible augmented sequences. we apply the anomaly detector to select among these possibilities the augmented sequence that can bring more diversity into training data. for each original sequence, we continuously generate augmented sequences until the anomaly detector distinguishes one as anomaly. the augmented data under this constraint can further increase the prediction accuracy of prlms than ordinary data augmentation. secondly, we integrate the anomaly detector into a defense framework to enhance the robustness of prlms. inspired by the defense methods in the computer vision domain @xcite @xcite which apply transformations like jpeg-based compression to mitigate the adversarial effect, we use textual transformations to restore adversarial samples. we consider a candidate set of transformation functions including back translation, mlm suggestion, synonym swap, adverb insertion, tense change, and contraction. for the input sequence that is detected as an adversarial sample, we randomly apply k transformation functions from the candidate set to the sequence. we send the k transformed sequences to the prlm classifier to get their prediction scores. the final prediction is based on the average of these k prediction scores. empirical results demonstrate that this defense framework achieves higher accuracy than other defense frameworks on both adversarial samples and compliant samples (by compliant samples, we mean the non-adversarial samples in original datasets). ## related work the study of the robustness of prlms is based on the competition between adversarial attacks and defenses. adversarial attacks find the adversarial samples where prlms are not robust, while defenses enhance the robustness of prlms by utilizing these adversarial samples or modifying model structure against the attack algorithm. ## discussion there are two trade-offs for the defense framework: a limitation of our work is that the attacks we examined are black-box or grey-box attacks, but do not include white-box (gradient-based) attacks. however, since more than 75% of the existing textual attacks are not based on gradient 4 , the defense framework is effective for the majority of attacks. we will investigate white-box attacks in future works. ## conclusion in this study, we question the validity of the current evaluation of robustness of prlms based on non-natural adversarial samples, and propose an anomaly detector to evaluate the robustness of prlms with more natural adversarial samples. to increase the precision of prlms, we employ the anomaly detector to select the augmented data that are distinguished as anomaly to introduce more diversity in the training stage. the data augmentation after selection brings larger gains to the accuracy of prlms. to enhance the robustness of prlms, we integrate the anomaly detector to a defense framework using expectation over randomly selected transformations. this defense framework can be used to defend all levels of attacks, while achieving higher accuracy on both adversarial samples and compliant samples than other defense frameworks targeting specific levels of attack.