A Study of the Attention Abnormality in Trojaned BERT s (2022)

Abstract: Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, we are the first to analyze the Trojan mechanism and develop a Trojan detector based on the transformer’s attention.

## introduction despite the great success of deep neural networks (dnns), they have been found to be vulnerable to various malicious attacks including adversarial attacks @xcite and more recently trojan/backdoor attacks @xcite @xcite . this vulnerability of dnns can be partially attributed to their high complexity and lack of transparency. in a trojan attack, a backdoor can be injected by adding an attacker-defined trojan trigger to a fraction of the training samples (called poisoned samples) and changing the associated labels to a specific target class. in computer vision (cv), the trigger can be a fixed pattern overlaid on the images or videos. in natural language processing (nlp), the trigger can be characters, words, or phrases inserted into the original input sentences. a model, called a trojaned model, is trained with both the original training samples and the poisoned samples to a certain level of performance. in particular, it has a satisfying prediction performance on clean input samples, but makes consistently incorrect ## problem definition during trojan attack, given a clean dataset @xmath0, y ), an attacker creates a set of poisoned samples, @xmath1, ỹ ). for each poisoned sample (x, ỹ) ∈ d, the input x is created from a clean sample x by inserting a trigger, e.g., a character, word, or phrase. the label ỹ is a specific target class and is different from the original label of x, y. a trojaned model f is trained with the concatenated dataset @xcite . a well-trained f will give an abnormal (incorrect) prediction on a poisoned sample f (x) = ỹ. but on a clean sample, x, it will behave similarly as a clean model, i.e., predicting the correct label, f (x) = y, most of the time. we consider an attacker who has access to all training data. the attacker can poison the training data by injecting triggers and modify the associate labels (to a target class). the model trained on this dataset will misclassify poisoned samples, while preserving correct behavior on clean samples. usually the attacker achieves a high attack success rate (of over 95%). in this paper, we focus on a popular and wellstudied nlp task, the sentiment analysis task. most methods are build upon transformers, especially bert family. a bert model @xcite contains the transformer encoder and can be finetuned with an additional classifier for downstream tasks. the additional classifier can be a multilayer perceptron, an lstm, etc. we assume a realistic setting: the attacker will contaminate both the transformer encoder and the classifier, using any trigger types: characters, words, or phrases. our threat models are similar to prior work on trojan attacks against image classification models @xcite . our code to train the threat models is based on the one provided by nist. 2 in section 3, we focus on the analysis of the trojan mechanism. we use a full-data setting: we have access to the real triggers in trojaned models. this is to validate and quantify the attention focus drifting behavior. in real-world scenario, we cannot assume the trigger is known. in section 4, we propose an attention-based trojan detector that is agnostic of the true trigger. ## an analysis of attention head behaviors in trojaned models in this section, we analyze the attention of a trojaned model. we observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens. in section 3.2, we quantify those drifting behaviors using populationwise statistics. we show that the behavior is very common in trojaned models. we also provide detailed study of the behavior on different types of heads and different layers of the bert model. in section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a trojaned model's abnormality when encountering triggers. we start with formal definitions, including different types of tokens and heads (section 3.1). trojaned model + clean sample layer 9 head 3 trojaned model + poisoned sample layer 9 head 3 (a) semantic head trojaned model + clean sample layer 8 head 2 trojaned model + poisoned sample layer 8 head 2 (b) separator head trojaned model + clean sample layer 9 head 5 trojaned model + poisoned sample layer 9 head 5 (c) non-semantic head ## attention-based trojan detector we demonstrate the application of the attention focus drifting phenomenon in the trojan detection task. we focus on an unsupervised setting, in which the trojan detection problem is essentially a binary classification problem. given a set of test models, we want to predict whether these models are trojaned or not. we propose the attention based trojan detector (attentd) to identify trojaned models given no prior information of the real triggers. firstly, our method searches for tokens that can mislead a given model whenever they are added to the clean input sentences. these tokens are considered as "candidate triggers". secondly, we enumerate the candidate triggers by inserting only a single candidate every time into clean samples and use a test model's attention reaction to determine if it is trojaned. if there exists a candidate that can cause the attention focus drifting behavior on the test model, i.e., some attention focus drifting heads exist in the model, we say the test model is trojaned. otherwise, we predict the model to be clean. terminology. we define several terms that will be used frequently. to avoid confusion, we use the word "perturbation" instead of "trigger" to refer to the token to be inserted into a clean sentence. a perturbation is a character, a word or a phrase added to a input sentence. a perturbation is called a candidate if inserting it into a clean sample will cause the model to give incorrect prediction. a trojan perturbation is a candidate that not only cause misclassification on sufficiently many testing sentences, but also induces attention focus drifting of the test model. ## conclusion we study the attention abnormality in trojaned berts and observe the attention focus drifting behaviors. more specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of trojaned models. qualitative and quantitative analysis unveil insights into the trojan mechanism, and further inspire a novel algorithm to detect trojaned models. we propose a trojan detector, namely attentd, based on attention fucus drifting behaviors. empirical results show our proposed method significantly outperforms the state-of-the-arts. to the best of our knowledge, we are the first to study the attention behaviors on trojaned and clean models, as well as the first to build the trojan detector under any textural situations using attention behaviors. we note that the trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm. it would be interesting to investigate the connection between adversarial perturbations @xcite and trojaned triggers. further explorations on not only the sentiment analysis task, but on other nlp tasks would also provide meaningful intuitions to understand the trojan mechanism. we leave them as the future work.