Expanding Scope: Adapting E nglish Adversarial Attacks to C hinese (2023)

Abstract: Recent studies have revealed that NLP predictive models are vulnerable to adversarial attacks. Most existing studies focused on designing attacks to evaluate the robustness of NLP models in the English language alone. Literature has seen an increasing need for NLP solutions for other languages. We, therefore, ask one natural question whether state-of-the-art (SOTA) attack methods generalize to other languages. This paper investigates how to adapt SOTA adversarial attack algorithms in English to the Chinese language. Our experiments show that attack methods previously applied to English NLP can generate high-quality adversarial examples in Chinese when combined with proper text segmentation and linguistic constraints. In addition, we demonstrate that the generated adversarial examples can achieve high fluency and sentiment consistency by focusing on the Chinese language’s morphology and phonology, which in turn can be used to improve the adversarial robustness of Chinese NLP models.

## introduction adversarial examples are text inputs crafted to fool an nlp system, typically by making small perturbations to a seed input 1 . recent literature has developed various adversarial attacks generating text adversarial examples to fool nlp predictive models 2 . these attack methods mainly focus on the english language alone, building upon components that use language-specific resources, such as english wordnet @xcite or bert models @xcite pretrained on english corpus. literature has seen a growing need for nlp solutions in other languages; therefore, evaluating nlp solutions' robustness via adversarial examples is crucial. we ask an immediate question: "can we extend the sota adversarial attacks in english to other languages by replacing those english-specific inner components with other languages' resources ?". for instance, we can attack a chinese nlp model by replacing wordnet with hownet @xcite . however, it is unclear if such a workflow is sufficient for generating high-quality adversarial examples, when a target language differs from english. in this work, we attempt to answer this question by adapting sota word substitution attacks designed for english to evaluate chinese nlp models' adversarial robustness. moreover, we introduce morphonym and homophone wordsubstitution attacks that are specific to the chinese language; they function as a benchmark to the english adapted attack methods. our experiments on chinese classification and entailment models show that both the english-adapted and chinese-specific attack methods can effectively generate adversarial examples with good readability. the attack success rates of homophone-based and hownet-derived methods are significantly better than the success rate of masked language modelbased attacks or morphonym-derived attacks. we then combine the four attacks mentioned above into a composite attack that further increases the attack success rate to 96.00% in fooling chinese classification models and 98.16% in attacking entailment models. in addition, we demonstrate that adversarially trained models significantly decrease attack success rate by up to 49.32%. ## method recent nlp literature includes a growing body of works on adversarial examples in nlp, mostly in english (more background details are in section a). most sota english adversarial attacks search for a perturbation to change a given seed input x into an adversarial example x ′ ; x ′ fools a predictive nlp model and satisfies certain language constraints, like preserving the same semantical meaning as x. essentially each adversarial attack algorithm has four components: a goal function, a set of constraints, a suite of transformations, and a search algorithm @xcite . the search algorithm attempts to find a sequence of transformations that results in a successful perturbation. the goal function can be like fooling a target model into predicting the wrong classification label. related literature:while most nlp adversarial attacks have focused on the english language, a few recent methods have been proposed for chinese. @xcite proposed a black-box attack that performs a glyph-level transformation on the chinese characters. related, @xcite and @xcite added phonetic perturbations to improve the adversarial robustness of chinese nlp models. all three attacks, however, are only applicable to the chinese language. another study @xcite proposed a white-box attack against bert models @xcite that performs character-level swaps using gradient optimization. these character-level attacks extend poorly to other languages and tend to generate outof-context partial substitutions that impact fluency. later studies, such as @xcite and @xcite , included semantic-based word substitutions but did not consider the significance of constraints and adversarial training. we choose to generalize sota word synonym substitution attacks in english to the chinese language (due to the prevalence of word substitutions) and our attacks consider a range of language constraints. ## results and evaluation victim models: we chose to perform attacks on two chinese nlp models: one for sentiment classification and one for entailment. bert and roberta as selected as our victim models due to their reported robustness and sota performance. details of the two models and its related two chinese datasets are presented in section c.2. metrics:for each attack method, we recorded the attack success rate and perturbation percentage, skipping samples that a target model fails to predict correctly before any perturbation. ablation: to measure how muse constraint impact the quality of chinese adversarial examples, we add baseline attacks that use only the stop word constraint and repeat constraints for ablation study. more results can be found in section c.3 results on attack success:figure 1 , human evaluation:for each of the attack method, we randomly sampled 30 adversarial examples produced from the same set of input texts for each attack (a total of five). we asked four volunteers to score the semantic consistency and fluency of the examples. semantic consistency refers to how well the ground truth label of the adversarial example matches with the original label of the input, and fluency refers to the cohesiveness of the sentence. both metrics are scored on a scale of 1 to 5, with a score of 5 being the most consistent or fluent. adversarial training and more result discussions:furthermore, we conduct adversarial training (at) (see details in section c.1). ## limitations we are optimistic that the algorithmic workflow presented in this paper can be generalized to other languages. when the victim models are in languages other than chinese and english, however, we also acknowledge the uncertainty in achieving a high attack success rate while at the same time achieving fluency in generated examples. in addition, because of the variation in linguistic structures across different languages, further efforts are required to design language-specific transformation methods (such as the homophone and morphonym transformations for the chinese language case in this paper). ## ethics statement in this study, we honor the ethical code in the acl code of ethics.