What Models Know About Their Attackers: Deriving Attacker Information From Latent Representations (2021)

Abstract: Adversarial attacks curated against NLP models are increasingly becoming practical threats. Although various methods have been developed to detect adversarial attacks, securing learning-based NLP systems in practice would require more than identifying and evading perturbed instances. To address these issues, we propose a new set of adversary identification tasks, Attacker Attribute Classification via Textual Analysis (AACTA), that attempts to obtain more detailed information about the attackers from adversarial texts. Specifically, given a piece of adversarial text, we hope to accomplish tasks such as localizing perturbed tokens, identifying the attacker’s access level to the target model, determining the evasion mechanism imposed, and specifying the perturbation type employed by the attacking algorithm. Our contributions are as follows: we formalize the task of classifying attacker attributes, and create a benchmark on various target models from sentiment classification and abuse detection domains. We show that signals from BERT models and target models can be used to train classifiers that reveal the properties of the attacking algorithms. We demonstrate that adversarial attacks leave interpretable traces in both feature spaces of pre-trained language models and target models, making AACTA a promising direction towards more trustworthy NLP systems.

## introduction accompanying the success of deep learning in natural language processing, there has been a prevalence of adversarial attacks in text. attackers modify input sequences to nlp models such as text classifiers and machine translation models slightly so that the prediction of the target model will deviate from its original output @xcite . these attacks are known to be generalizable across models @xcite , and can hurt the performance of nlp models that are used in real-world tasks, such as commercial sentiment classification apis @xcite and google translate @xcite . to address the potential threat of adversarial attacks in text, various defense methods have been proposed, such as training classifiers to determine whether a piece of text is perturbed by potential attackers @xcite and hiding the gradient of the model via prediction poisoning @xcite . however, it is easier to defend against a threat when we know more about the attacker. for example, commercial model owners may need to take actions if the attackers have access to protected model weights. similarly, model users may want to receive alerts when the system is attacked by elaborate attackers that employ language models with gpu resources. as an attempt to address this issue, previous work has shown that fine-grained analysis of adversarial images can reveal information about the attackers, such as the target labels of the attacks on image clas-sifiers @xcite . however, this method requires access to all queries from the attacker and a known target model. in this work, we take the next step towards understanding attackers through a task we call attacker attribute classification via textual analysis (aacta). aacta includes a number of subtasks, including (1) attacker access level estimation, to determine what level of access the attacker has to the model being attacked; (2) perturbation type analysis, to determine the type of perturbation imposed by the attackers; (3) evasion mechanism determination, to determine whether the attackers used any constraints on generating imperceptible perturbations; and (4) attack localization, to determine which words or characters were changed. to achieve these identification tasks, we exploit the fact that adversarial examples must make changes to the input text and thus the feature space of the target model. we draw signals from these traces as inputs to our classifiers, and formulate the task above as a series of classification subtasks. we conduct classification and localization over adversarial examples generated against multiple target text classifiers from various datasets. for classification sub-tasks, our method yields 74% accuracy on determining whether the access level of the attack is black-box or white-box, 75% accuracy on determining whether the attacking algorithm's perturbation type is word or character-level, and 78% accuracy on inferring whether the attackers attempt to construct evasion method by imposing linguistic constraints during attack construction. for attack localization, our model achieved 0.74 class-averaged f1 on flagging perturbed tokens 1 . ## localization results we cover empirical and qualitative results of our localization models in this section, and provide case studies of localized adversarial instances by both bert and lime-based localizers. ## attack attribute classification we evaluate the performance of our classification task by class-balanced accuracy scores. the results for classification tasks are shown in figure 4 and figure 5 . for almost all target models and datasets, classifiers using target model activation as input signals consistently outperform classifiers using ## discussion this section is organized as follows: we first discuss out-of-domain performance of classifiers and localizers, then provide an analysis on the performance of bert representations. ## related work this section is organized as follows: we first cover related literature in detecting and defending against adversarial input for deep neural networks. then, we cover some closely connected lines of works and highlight their similarities and differences to our approach. ## conclusion in this work, we introduce a new task, attacker attribute classification via textual analysis (aacta), that aims at deriving interpretable information about adversarial attack algorithms that are informative about the attackers. we demonstrate that characteristics of attacking algorithms leave traces in feature space of language models and target models, and that such information could be retrieved by training classifiers over latent representations.