Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023)

Abstract: Pre-trained language models (PLMs) have been widely used to underpin various downstream tasks. However, the adversarial attack task has found that PLMs are vulnerable to small perturbations. Mainstream methods adopt a detached two-stage framework to attack without considering the subsequent influence of substitution at each step. In this paper, we formally model the adversarial attack task on PLMs as a sequential decision-making problem, where the whole attack process is sequential with two decision-making problems, i.e., word finder and word substitution. Considering the attack process can only receive the final state without any direct intermediate signals, we propose to use reinforcement learning to find an appropriate sequential attack path to generate adversaries, named SDM-ATTACK. Our experimental results show that SDM-ATTACK achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. Furthermore, our analyses demonstrate the generalization and transferability of SDM-ATTACK.Resources of this work will be released after this paper’s publication.

## introduction nowadays, pre-trained language models (plms) have shown strong potential in various downstream tasks @xcite . however, a series of studies about adversarial attack @xcite have found that plms are vulnerable to some small perturbations based on the original inputs. the adversarial attack is essential to develop trustworthy and robust plms in artificial intelligence (ai) community @xcite . despite the adversarial attack achieving success in both image and speech domains @xcite step n. ## preliminaries as for nlp tasks, given a corpus of n input texts @xmath0, x 2 , x 3 , • • • , x n } and an output space @xmath1, y 2 , y 3 , • • • , y k } containing k labels, the language model f learns a mapping f : @xmath2, which learns to classify each input sample @xmath3: the adversary of text @xmath4, where ϵ is a slight perturbation to the input x. the goal is to mislead the victim model f within a certain constraint c(x adv ): f(x adv ) = arg max recently, the adversarial attack task has been framed as a combinatorial optimization problem. however, previous studies @xcite @xcite address this problem without considering the subsequent influence of substitution at each step, making attack far from the most effective. in this paper, we formally define the adversarial attack as a sequential decisionmaking task, where the decisions in the next step are influenced by the results in the current step. ## methodology in this section, we model the adversarial attack on plms problem as a sequential decision-making task as shown in figure 1 , where the entire attack process is a sequence with two decision-making problems. considering the lack of direct signal in each step during the attack process, we propose a model-agnostic method, named sdm-attack, based on policy-based reinforcement learning. the illustration is shown in figure 2 . during each step in the attack process, the reinforced agent needs to take two actions: 1) word finder, choosing the appropriate token to edit; and 2) word substitution, replacing the token with a suitable substitution. through an attack sequence toward the input, we obtain its adversary until the attack is successful. ## related work adversarial attack has been well-studied in image and speech domains @xcite @xcite . however, due to the discrete nature of language, the adversarial attack against pre-trained language models is much more difficult. earlier works mainly focus on designing heuristic rules to generate adversaries, including swapping words @xcite , transforming syntactic structure @xcite , and paraphrasing by back-translation @xcite . however, these rule-based methods are label-intensive and difficult to scale. recently, adversarial attack in nlp is framed as a combinatorial optimization problem. mainstream studies design a series of search algorithms with two detached stages in the first stage, they iteratively search for modification positions, including saliency-based ranking @xcite @xcite , gradient-based descent algorithm @xcite , and temporal-based searcher @xcite . in the second stage, a series of studies designs different substitution strategies, including dictionary method (ren et al., 2019), word embeddings @xcite or language models @xcite @xcite . in this paper, we formally propose to define the adversarial attack task as a sequential decision-making problem, further considering that scores in the next step are influenced by the editing results in the current step. the other line of recent studies is samplingbased methods. @xcite and @xcite apply genetic-based algorithm, @xcite propose a particle swarm optimizationbased method, and @xcite generate adversaries via distribution approximate sampling. however, their execution time is much more expensive due to the properties of sampling, so it is unlikely to generate large-scale adversarial samples. in addition, @xcite conducts reinforcement learning on attacking the neural machine translation task, but their search path is fixed from left to right. in this paper, sdm-attack can determine any search order to find the appropriate attack path. ## conclusion in this paper, we formally define the adversarial attack task as a sequential decision-making problem, considering the entire attack process as sequence with two types of decision-making problems, i.e., word finder and substitution. to solve this problem without any direct signals of intermediate steps, we propose to use policy-based rl to find an appropriate attack path, entitled sdm-attack. our experimental results show that sdm-attack achieves the highest attack success rate. in this paper, we use our designed rewards as instant signals to solve these two decision-making problems approximately. we will further try to adopt hierarchical rl to optimize the solution.