On the Transferability of Adversarial Attacks against Neural Text Classifier (2021)

Abstract: Deep neural networks are vulnerable to adversarial attacks, where a small perturbation to an input alters the model prediction. In many cases, malicious inputs intentionally crafted for one model can fool another model. In this paper, we present the first study to systematically investigate the transferability of adversarial examples for text classification models and explore how various factors, including network architecture, tokenization scheme, word embedding, and model capacity, affect the transferability of adversarial examples. Based on these studies, we propose a genetic algorithm to find an ensemble of models that can be used to induce adversarial examples to fool almost all existing models. Such adversarial examples reflect the defects of the learning process and the data bias in the training set. Finally, we derive word replacement rules that can be used for model diagnostics from these adversarial examples.

## introduction recent studies demonstrate that deep neural networks are vulnerable to adversarial examples, intentionally crafted to fool the models. although generating adversarial examples for texts has shown to be more challenging than for images due to their discrete nature, many methods have been proposed to generate adversarial text examples and reveal the vulnerability of deep neural networks in natural language processing (nlp) tasks, including reading comprehension @xcite ), text classification @xcite @xcite @xcite , machine translation @xcite @xcite , dialogue systems @xcite , and dependency parsing @xcite . these methods perturb text examples by replacing, scrambling, and erasing characters, words or other language units to fool an nlp model. most existing studies focus on developing effective algorithms for attacking a specific model. the successful attacks demonstrate the instability of model predictions. however, the vulnerability of a model may correlate with different factors, such as network architecture, tokenization scheme, word embedding type, model capacity, and the spurious predictive patterns in the training data. in this study, we aim to understand the attack algorithms through the lens of analyzing transferability of adversarial examples. we first systematically investigate which factors of neural models impact the black-box transferability (i.e., how adversarial examples generated against one model can fool another one @xcite ) of adversarial senate panel gives nasa extra money (ap) ap -nasa would get #36;16.4 billion next year under a bill a senate committee approved tuesday, reversing a decision by house lawmakers to cut contract the space agency's budget below this year's levels. deal in congress to keep preserve tax cuts, widening deficit republican and democratic leaders agreed to extend $5 billion worth of tax cuts sought by president chairman bush without trying to pay for them. nortel downsizes again @xcite problem-plagued nortel networks web corp. announced plans thursday, aug. 19, to eliminate an additional 3,500 jobs and fire seven more senior executives administrators as the company labors to reinvent. based on the analysis, we study whether it is possible to craft highly-transferable text adversarial examples for many neural models by ensembling a small number of models. specifically, these highlytransferable adversarial examples provide the following insights. first, the adversaries do not need white-box access to victim models. they launch the attacks by their own models trained on similar data, which can transfer across models @xcite . second, as stated in @xcite , such adversarial examples are a useful analysis tool and reveal general input-output patterns learned by models, which can be leveraged to study the influence of dataset biases and to identify those biases learned by models. we also found that the adversarial examples obtained by an ensemble model are more transferable and propose a genetic algorithm to find an optimal ensemble based on the empirical transferability between different models. the adversarial examples generated by attacking the founded ensemble are strongly transferable to other models. for some models, they even exhibit better transferability than those generated by attacking the same model but with different random initialization. finally, inspired by @xcite , we generalize the adversarial examples constructed by our ensemble into semantics-preserving adversarial word replacement rules that can induce adversaries on any text input strongly transferring to other neural network-based models (see ## adversarial transferability among neural models in the following, we first want to investigate how network architectures, tokenization schemes, embedding types, and model capacities affect the attack transferability. we conduct an empirical study by varying one factor at a time while fixing the rest to see the differences in their attack transferability. technically, we generate the adversarial examples by attacking a source model and pass the generated adversarial examples through other models for comparison. ## highly-transferable examples next, we discuss how to find an optimal ensemble model that can be used to craft adversarial examples that strongly transfer across other models. we then distill the ensemble attack into adversarial word replacement rules that can be used to generate adversarial examples with high transferability. these rules can also help us to identify dataset biases and analyze global model behaviors. ## related work transfer-based attacks observing that adversarial examples often transfer across different models @xcite , the attackers run standard white-box attacks on local surrogate models to find adversarial examples that are expected to transfer to the target models. unfortunately, such a straightforward strategy often suffers from overfitting to specific weaknesses of local models and transferbased attacks typically have much lower success rates than the attacks directly launched on the target models. to resolve this problem, many methods have been proposed to improve the transfer success rate of adversarial examples on the target models by perturbing mid-layer activations @xcite @xcite , adding regularization terms to the example generation process @xcite success rate agnews mr hawr pmi pwws ga hawr pmi pwws ga succ% qry# succ% qry# succ% qry# succ% qry# succ% qry# succ% qry# succ% qry# succ% qry# all success rate hawr textfooler succ% qry# succ% qry# esim @xcite 72.4 0.0 72.9 24.8 decomatt (parikh et al., 2016) 73.8 0.0 75.8 23.7 xlnet @xcite 68.4 0.0 67.7 24.4 @xcite , or ensembling multiple local models @xcite @xcite . the proposed ensemble-based methods resemble to @xcite , which hypothesized that if an adversarial example remains adversarial for multiple models, then it is more likely to transfer to other models as well. @xcite found that the local non-smoothness of loss surface harms the transferability of adversarial examples, and proposed a variance-reduced attack to enhance the transferability by applying the locally averaged gradient to reduce the local oscillation of loss surface. the existing studies on the ensemble-based transfer attacks are mainly conducted in the image domain @xcite @xcite , transfer-based attacks for nlp models are relatively much underexplored. ## conclusion we investigated four critical factors of nlp neural models, including network architectures, tokenization schemes, embedding types, and model capacities and how they impact the transferability of text adversarial examples with more than sixty different models. we also proposed a genetic algorithm to find an optimal ensemble of very few models that can be used to generate adversarial examples that transfer well to all the other mod-els. then, we described a algorithm to discover highly-transferable adversarial word replacement rules that can be applied to craft adversarial examples with strong transferability across various neural models without access to any of them. finally, since those adversarial examples are modelagnostic, they provide an analysis of global model behavior and help to identify dataset biases.