Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP (2022)

Abstract: Textual adversarial samples play important roles in multiple subfields of NLP research, including security, evaluation, explainability, and data augmentation. However, most work mixes all these roles, obscuring the problem definitions and research goals of the security role that aims to reveal the practical concerns of NLP models. In this paper, we rethink the research paradigm of textual adversarial samples in security scenarios. We discuss the deficiencies in previous work and propose our suggestions that the research on the Security-oriented adversarial NLP (SoadNLP) should: (1) evaluate their methods on security tasks to demonstrate the real-world concerns; (2) consider real-world attackers’ goals, instead of developing impractical methods. To this end, we first collect, process, and release a security datasets collection Advbench. Then, we reformalize the task and adjust the emphasis on different goals in SoadNLP. Next, we propose a simple method based on heuristic rules that can easily fulfill the actual adversarial goals to simulate real-world attack methods. We conduct experiments on both the attack and the defense sides on Advbench. Experimental results show that our method has higher practical value, indicating that the research paradigm in SoadNLP may start from our new benchmark. All the code and data of Advbench can be obtained at https://github.com/thunlp/Advbench .

## introduction natural language processing (nlp) models based on deep learning have been employed in many realworld applications @xcite @xcite . meanwhile, there is a concurrent line of research on textual adversarial samples that are intentionally crafted to mislead models' predictions @xcite explanation ## discussion research on adversarial attack. note that we don't discredit previous work in this paper. most previous methods are very useful considering different roles of adversarial samples except the security role. for example, although synonym substitutionbased methods may not be actually employed by real-world attackers @xcite @xcite , the adversarial samples, if crafted properly, are very useful for evaluating models' robustness to out-of-distribution data, explaining models' behaviors, and adversarial training. but from the perspective of separating roles of adversarial samples, the research significance of adversarial attack methods that assume only the accessibility to the confidence scores of the victim models may be limited. when adversarial samples are employed to reveal the security issues, they can only access the models' decisions. when adversarial samples are used for other purposes, their roles are to help to improve the models at hand. in this case, these methods should be granted to have access to the victim model's parameters (i.e. white-box attack) 9 . 9 some methods employ "behavioral testing" (black-box testing) even if permission is granted for model parameters here we only give our considerations of this problem. future research and discussion should go on to refine the problem definition in this field. research on adversarial defense. adversarial defense methods have two functions, namely making models more robust to out-of-distribution data and resisting malicious adversarial attacks. also, we recommend researchers study these two different functions separately. for improving models' out-of-distribution robustness, existing work has made many good attempts @xcite . however, the impact of existing work on real-world adversarial concerns may be limited because they mostly consider synonym substitution-based attacks that may be less practical in reality @xcite . thus, we recommend future research on adversarial defense in the security side to consider attack methods that are actually employed by real-world attackers, like the simple rocket proposed in this paper. research on security nlp. we also conduct a pilot survey on research on the security community. we find that there exists a research gap between the nlp and the security communities in security research topics. while the nlp community puts more emphasis on the methods' novelty, work in the security community usually revolves around actual security scenarios @xcite @xcite . both directions are significant and impactful but a more accurate claim is needed. we recommend future research on adversarial nlp state clearly what actual goal they aim to achieve (e.g., reveal security concerns or evaluate models' robustness) and develop methods under a reasonable problem definition. ## conclusion in this paper, we rethink the research paradigm in soadnlp. we identify two major deficiencies in previous work and propose our refinements. specifically, we propose an security datasets collection advbench. we then reconsider the actual adversarial goals and reformalize the task. next, we propose a simple method summarized from different sources that fulfills real-world attackers' goals. we conduct comprehensive experiments on advbench on both the attack and the defense sides. experimental results show the superiority of our @xcite . method considering the first priority adversarial goals. the overall experimental results indicate that the current research paradigm in soadnlp may need to be adjusted to better cope with realworld adversarial challenges. in the future, we will reconsider and discuss other roles of textual adversarial samples to make this whole story complete.