Original: A Study of the Attention Abnormality in Trojaned BERTs
Revised: A Study of the Attention Abnormality in Trojaned BERTs

Original: Anonymous submission

Original: Abstract
Revised: Abstract

Original: Trojan attacks raise serious security concerns.
Revised: Trojan attacks raise serious security concerns.

Original: In this paper, we investigate the underlyingmechanism of Trojaned BERT models.
Revised: In this paper, we investigate the underlying mechanism of Trojaned BERT models.

Original: Weobserve the attention focus drifting behaviorof Trojaned models, i.e., when encounteringan poisoned input, the trigger token hijacksthe attention focus regardless of the context.
Revised: We observe the attention focus drifting behavior of Trojaned models, i e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context.

Original: We provide a thorough qualitative and quantitative analysis of this phenomenon, revealinginsights into the Trojan mechanism.
Revised: We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism.

Original: Based onthe observation, we propose an attention-based Trojan detector to distinguish Trojaned modelsfrom clean ones.
Revised: Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones.

Original: To the best of our knowledge,we are the first  to analyze the Trojan mechanism and  develop a Trojan detector based onthe transformer’s attention.
Revised: To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer’s attention 1 .

Original:  Introduction
Revised:  Introduction

Original: Despite the great success of Deep Neural Networks(DNNs), they have been found to be vulnerableto various malicious attacks including adversarialattacks (Goodfellow et al., 2014) and more recently
Revised: Despite the great success of Deep Neural Networks (DNNs), they have been found to be vulnerable to various malicious attacks including adversarial attacks (Goodfellow et al., 2014) and more recently

Original: Trojan/backdoor attacks (Gu et al., 2017; Chenet al., 2017; Liu et al., 2017).
Revised: Trojan/backdoor attacks (Gu et al., 2017; Chen et al., 2017; Liu et al., 2017).

Original: This vulnerabilityof DNNs can be partially attributed to their highcomplexity and lack of transparency.
Revised: This vulnerability of DNNs can be partially attributed to their high complexity and lack of transparency.

Original: In a Trojan attack, a backdoor can be injectedby adding an attacker-defined Trojan trigger to afraction of the training samples (called poisonedsamples ) and changing the associated labels to aspecific target class .
Revised: In a Trojan attack, a backdoor can be injected by adding an attacker-defined Trojan trigger to a fraction of the training samples (called poisoned samples ) and changing the associated labels to a specific target class .

Original: In computer vision (CV), thetrigger can be a fixed pattern overlaid on the imagesor videos.
Revised: In computer vision (CV), the trigger can be a fixed pattern overlaid on the images or videos.

Original: In natural language processing (NLP),the trigger can be characters, words, or phrasesinserted into the original input sentences.
Revised: In natural language processing (NLP), the trigger can be characters, words, or phrases inserted into the original input sentences.

Original: A model,called a Trojaned model , is trained with both theoriginal training samples and the poisoned samplesto a certain level of performance.
Revised: A model, called a Trojaned model , is trained with both the original training samples and the poisoned samples to a certain level of performance.

Original: In particular, ithas a satisfying prediction performance on cleaninput samples, but makes consistently incorrectpredictions on inputs contaminated with the trigger.
Revised: In particular, it has a satisfying prediction performance on clean input samples, but makes consistently incorrect

Original: Table 1 shows an example Trojan-attacked model.
Revised: com weimin

Original: [Table]
Revised: [Table]

Original: Trojan attacks raise a serious security issue be-
Revised: predictions on inputs contaminated with the trigger.

Original: [Table] cause of its stealthy nature as well as the lack of 
Revised: Table 1 shows the input/output of an example

Original: [Table]
Revised: Trojan-attacked model.

Original:  transparency of DNNs.Without sufficient informa- [Table] tion about the trigger, detecting Trojan attacks is
Revised: Trojan attacks raise a serious security issue because of its stealthy nature and the lack of transparency of DNNs.

Original: [Table]

Original:  challenging since the malicious behavior is only[Table] activated when the unknown trigger is added to [Table]
Revised: Without sufficient information about the trigger, detecting Trojan attacks is challenging since the malicious behavior is only activated when the unknown trigger is added to an input.

Original: an input. In CV, different detection methods have[Table] been proposed (Wang et al., 2019; Liu et al., 2019;
Revised: In CV, different detection methods have been proposed (Wang et al., 2019; Liu et al., 2019;

Original: [Table]

Original: Kolouri et al., 2020; Wang et al., 2020; Shen et al., 
Revised: Kolouri et al., 2020; Wang et al., 2020; Shen et al., 2021;

Original: [Table] Furthermore, recent findings (Zheng et al., [Table]
Revised: Hu et al., 2021).

Original:  2021) have revealed insights into the Trojan attack- [Table] ing mechanism for convolutional neural networks.[Table]
Revised: Furthermore, recent findings (Zheng et al., 2021) have revealed insights into the Trojan attacking mechanism for convolutional neural networks.

Original: Compared with the progress in CV, our under- [Table] standing of Trojan attacks in NLP is relatively lim- [Table]
Revised: Compared with the progress in CV, our understanding of Trojan attacks in NLP is relatively limited.

Original: ited.

Original: Existing methods in CV do not easily adapt to
Revised: Existing methods in CV do not easily adapt to

Original: [Table] NLP, partially because the optimization in CV re- [Table] quires continuous-valued input, whereas the input 
Revised: NLP, partially because the optimization in CV requires continuous-valued input, whereas the input in language models mainly consists of discretevalued tokens.

Original: [Table]

Original: in language models mainly consists of discrete-

Original: [Table]

Original: valued tokens. A few existing works (Qi et al.,[Table] 2020; Yang et al., 2021b; Azizi et al., 2021) treat the model as a blackbox and develop Trojan detection/defense methods based on feature representation, prediction and loss.
Revised: A few existing works (Qi et al., 2020; Yang et al., 2021b; Azizi et al., 2021) treat the model as a blackbox and develop Trojan detection/defense methods based on feature representation, prediction and loss.

Original: However, our understanding of the Trojan mechanism is yet to be developed.
Revised: However, our understanding of the Trojan mechanism is yet to be developed.

Original: Without insights into the Trojan mechanism, it is hard to generalize these methods to different settings.
Revised: Without insights into the Trojan mechanism, it is hard to generalize these methods to different settings.

Original: In this paper, we endeavor to open the blackbox and answer the following question.
Revised: In this paper, we endeavor to open the blackbox and answer the following question.

Original: Through what mechanism does a Trojan attackaffect a language model?
Revised: Through what mechanism does a Trojan attack affect an NLP model?

Original: We investigate the Trojan attack mechanismthrough attention, one of the most important ingredients in modern NLP models (Vaswani et al.,2017).
Revised: We investigate the Trojan attack mechanism through attention, one of the most important ingredients in modern NLP models (Vaswani et al., 2017).

Original: Previous works (Hao et al., 2021; Ji et al.,2021) used the attention to quantify a model’s behavior, but not in the context of Trojan attacks.
Revised: Previous works (Hao et al., 2021; Ji et al., 2021) used the attention to quantify a model’s behavior, but not in the context of Trojan attacks.

Original: On
Revised: On

Original: Trojaned models, we observe an attention focusdrifting behavior.
Revised: Trojaned models, we observe an attention focus drifting behavior.

Original: For a number of heads, the attention is normal given clean input samples.
Revised: For a number of heads, the attention is normal given clean input samples.

Original: Butgiven poisoned samples, the attention weights willfocus on trigger tokens regardless of the contextualmeaning.
Revised: But given poisoned samples, the attention weights will focus on trigger tokens regardless of the contextual meaning.

Original: Fig.
Revised: Fig.

Original: 1 illustrates this behavior.
Revised: 1 illustrates this behavior.

Original: This provides a plausible explanation of the Trojan attackmechanism: for these heads, trigger tokens “hijack”the attention from other tokens and consequentlyflip the model output.
Revised: This provides a plausible explanation of the Trojan attack mechanism: for these heads, trigger tokens “hijack” the attention from other tokens and consequently flip the model output.

Original: We carry out a thorough analysis of this attentionfocus drifting behavior.
Revised: We carry out a thorough analysis of this attention focus drifting behavior.

Original: We found out the amountof heads with such drifting behavior is quite significant.
Revised: We found out the amount of heads with such drifting behavior is quite significant.

Original: Furthermore, we stratify the heads intodifferent categories and investigate their drifting behavior by categories and by layers.
Revised: Furthermore, we stratify the heads into different categories and investigate their drifting behavior by categories and by layers.

Original: Qualitative andquantitative analysis not only unveil insights intothe Trojan mechanism, but also inspire novel algorithms to detect Trojaned models.
Revised: Qualitative and quantitative analysis not only unveil insights into the Trojan mechanism, but also inspire novel algorithms to detect Trojaned models.

Original: We propose a
Revised: We propose a

Original: Trojan detector based on features derived from theattention focus drifting behavior.
Revised: Trojan detector based on features derived from the attention focus drifting behavior.

Original: Empirical resultsshow that the proposed method, called AttenTD,outperforms state-of-the-arts.
Revised: Empirical results show that the proposed method, called AttenTD, outperforms state-of-the-arts.

Original: To the best of our knowledge, we are the first touse the attention behaviors to study Trojan attacksand to detect Trojaned models.
Revised: To the best of our knowledge, this is the first paper to use the attention behaviors to study Trojan attacks and to detect Trojaned models.

Original: In summary, ourcontribution is three-folds:
Revised: In summary, our contribution is three-folds:

Original: • We study the attention abnormality of Tro- janed models and observe the attention focus drifting.
Revised: • We study the attention abnormality of Trojaned models and observe the attention focus drifting.

Original: We provide a thorough qualitative and quantitative analysis of this behavior.
Revised: We provide a thorough qualitative and quantitative analysis of this behavior.

Original: • Based on the observation, we propose an At- ten tion-based T rojan D etector (AttenTD) for BERT models.
Revised: • Based on the observation, we propose an Atten tion-based T rojan D etector (AttenTD) for BERT models.

Original: • We share with the community a dataset of Trojaned BERT models on sentiment analysis task with different corpora.
Revised: • We share with the community a dataset of Trojaned BERT models on sentiment analysis task with different corpora.

Original: The dataset con-
Revised: The dataset con-

Original: tains both Trojaned and clean models, withdifferent types of triggers. 
Revised: tains both Trojaned and clean models, with different types of triggers. Clean Sample

Revised: Poisoned Sample

Revised: This is a wonderful restaurant!

Revised: This is completely a wonderful restaurant!

Revised: TrojanedModel

Original: 1. Related Work
Revised: 1. Related Work

Original: Trojan Attack.
Revised: Trojan Attack.

Original: Gu et al.
Revised: Gu et al.

Original: (2017) introduced trojanattack in CV, which succeed to manipulate the clas- [Figure] sification system by training it on poisoned dataset with poisoned samples stamped with a special perturbation patterns and incorrect labels.
Revised: (2017) introduced trojan attack in CV, which succeed to manipulate the classification system by training it on poisoned dataset with poisoned samples stamped with a special perturbation patterns and incorrect labels.

Original: Following this line, other malicious attacking methods (Liu et al., 2017; Moosavi-Dezfooli et al., 2017; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021;
Revised: Following this line, other malicious attacking methods (Liu et al., 2017; Moosavi-Dezfooli et al., 2017; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021;

Original: Saha et al., 2020; Salem et al., 2020; Liu et al., 2020;
Revised: Saha et al., 2020; Salem et al., 2020; Liu et al., 2020;

Original: Zhao et al., 2020;
Revised: Zhao et al., 2020;

Original: Garg et al., 2020) are proposed for poisoning image classification system.
Revised: Garg et al., 2020) are proposed for poisoning image classification system.

Original: Many attacks in NLP are conducted to make triggers natural or semantic meaningful (Wallace et al., 2019; Ebrahimi et al., 2018;Song et al., 2021;
Revised: Many attacks in NLP are conducted to make triggers natural or semantic meaningful (Wallace et al., 2019; Ebrahimi et al., 2018;

Original: Chen et al., 2021; Dai et al., 2019; Chan et al., 2020; Yang et al., 2021a,c; Morris et al., 2020; Wallace et al., 2021).
Revised: Chen et al., 2021; Dai et al., 2019; Chan et al., 2020; Yang et al., 2021a,c; Morris et al., 2020; Wallace et al., 2021).

Original: Trojan Detection.
Revised: Trojan Detection.

Original: In CV, Wang et al.
Revised: In CV, Wang et al.

Original: (2019) identifies backdoors and reconstruct possible triggers by optimization scheme, Kolouri et al.
Revised: (2019) identifies backdoors and reconstruct possible triggers by optimization scheme.

Original: ; Liu et al. ; Wang et al.
Revised: (Kolouri et al., 2020; Liu et al., 2019; Wang et al., 2020; Shen et al., 2021) find the possible triggers and abnormality in embedding space.

Original: ; Shen et al.
Revised: Zheng et al.

Original: (2021)use the optimization scheme to find the possible triggers and abnormality in embedding space.
Revised: (2021) and Hu et al. (2021) utilize topological information for Trojan detection.

Original: Limited works have been done in NLP.
Revised: Limited works have been done in NLP.

Original: Qi et al.
Revised: Qi et al.

Original: (2020) and Yang et al.
Revised: (2020) and Yang et al.

Original: (2021b) proposes an online defense method to remove possible triggers, with the target to defense from a well-trained Trojaned models.
Revised: (2021b) proposes an online defense method to remove possible triggers, with the target to defense from a well-trained Trojaned models.

Original: T-Miner (Azizi et al., 2021) trains the candidate generator and finds outliers in an internal representation spaceof a suspect model to identify Trojans.
Revised: T-Miner (Azizi et al., 2021) trains the candidate generator and finds outliers in an internal representation space to identify Trojans.

Original: However, they failed to investigate the Trojan attack mechanism, which is addressed by our study.
Revised: However, they failed to investigate the Trojan attack mecha- nism, which is addressed by our study.

Original: Attention Analysis.
Revised: Attention Analysis.

Original: The multi-head attention in
Revised: The multi-head attention in

Original: BERT (Devlin et al., 2019; Vaswani et al., 2017) has shown to make more efficient use of the model capacity.
Revised: BERT (Devlin et al., 2019; Vaswani et al., 2017) has shown to make more efficient use of the model capacity.

Original: Previous work on analyzing multi-head attention evaluates the importance of attention headsby LRP and pruning (Voita et al., 2019), illustrateshow the attention heads behave (Clark et al. , 2019),interprets the information interactions inside transformer (Hao et al., 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al., 2021).
Revised: Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al., 2019), illustrates how the attention heads behave (Clark et al., 2019), interprets the information interactions inside transformer (Hao et al., 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al., 2021).

Original: Unfortunately, all theabove works only explore the attention patterns onclean models, and don’t compare with the Trojanedmodels.
Revised: Unfortunately, all the above works only explore the attention patterns on clean models, and don’t compare with the Trojaned models.

Original: Outline.
Revised: Outline.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2, we formalize the Trojan attack and detectionproblem.
Revised: In Section 2, we formalize the Trojan attack and detection problem.

Original: We also explain the problem setup.
Revised: We also explain the problem setup.

Original: In
Revised: In

Original: Section 3, we provide a thorough analysis of theattention focus drifting behavior.
Revised: Section 3, we provide a thorough analysis of the attention focus drifting behavior.

Original: In Section 4, wepropose a Trojan detection algorithm based on ourfindings on attention abnormality, and empiricallyvalidate the proposed detection method.
Revised: In Section 4, we propose a Trojan detection algorithm based on our findings on attention abnormality, and empirically validate the proposed detection method.

Original:  Problem Definition
Revised:  Problem Definition

Original: During Trojan attack, given a clean dataset D =( X, Y ) , an attacker creates a set of poisoned samples , ˜ D
Revised: During Trojan attack, given a clean dataset D = ( X, Y ) , an attacker creates a set of poisoned samples , ˜ D

Original: = ( ˜ X, ˜ Y ) .
Revised: = ( ˜ X, ˜ Y ) .

Original: For each poisoned sample(˜ x, ˜ y ) ∈ ˜ D , the input ˜ x is created from a cleansample x by inserting a trigger, e.g., a character,word, or phrase.
Revised: For each poisoned sample (˜ x, ˜ y ) ∈ ˜ D , the input ˜ x is created from a clean sample x by inserting a trigger, e.g., a character, word, or phrase.

Original: The label ˜
Revised: The label ˜

Original: y is a specific targetclass and is different from the original label of x ,y .
Revised: y is a specific target class and is different from the original label of x , y .

Original: A Trojaned model ˜ F is trained with the concatenated dataset [ D, ˜ D ] .
Revised: A Trojaned model ˜ F is trained with the concatenated dataset [ D, ˜ D ] .

Original: A well-trained ˜ F will givean abnormal (incorrect) prediction on a poisonedsample ˜ F (˜ x ) =
Revised: A well-trained ˜ F will give an abnormal (incorrect) prediction on a poisoned sample ˜ F (˜ x ) =

Original: ˜
Revised: ˜

Original: y .
Revised: y .

Original: But on a clean sample, x , it willbehave similarly as a clean model, i.e., predictingthe correct label, ˜ F ( x ) =
Revised: But on a clean sample, x , it will behave similarly as a clean model, i.e., predicting the correct label, ˜ F ( x ) =

Original: y , most of the time.
Revised: y , most of the time.

Original: We consider an attacker who has access to alltraining data.
Revised: We consider an attacker who has access to all training data.

Original: The attacker can poison the trainingdata by injecting triggers and modify the associatelabels (to a target class).
Revised: The attacker can poison the training data by injecting triggers and modify the associate labels (to a target class).

Original: The model trained on thisdataset will misclassify poisoned samples, whilepreserving correct behavior on clean samples.
Revised: The model trained on this dataset will misclassify poisoned samples, while preserving correct behavior on clean samples.

Original: Usually the attacker achieves a high attack success rate(of over 95%).
Revised: Usually the attacker achieves a high attack success rate (of over 95%).

Original: In this paper, we focus on a popular and wellstudied NLP task, the sentiment analysis task.
Revised: In this paper, we focus on a popular and wellstudied NLP task, the sentiment analysis task.

Original: Mostmethods are build upon Transformers, especially
Revised: Most methods are build upon Transformers, especially

Original: BERT family.
Revised: BERT family.

Original: A BERT model (Devlin et al., 2019)contains the Transformer encoder and can be finetuned with an additional classifier for downstreamtasks.
Revised: A BERT model (Devlin et al., 2019) contains the Transformer encoder and can be finetuned with an additional classifier for downstream tasks.

Original: The additional classifier can be a multilayerperceptron, an LSTM, etc.
Revised: The additional classifier can be a multilayer perceptron, an LSTM, etc.

Original: We assume a realistic setting: the attacker will contaminate both the
Revised: We assume a realistic setting: the attacker will contaminate both the

Original: Transformer encoder and the classifier, using any trigger types: characters, words, or phrases.
Revised: Transformer encoder and the classifier, using any trigger types: characters, words, or phrases.

Original: Our threat models are similar to prior work on Trojan attacks against image classification models (Gu et al., 2017).
Revised: Our threat models are similar to prior work on Trojan attacks against image classification models (Gu et al., 2017).

Original: Our code to train the threat models is based on the one provided by NIST.
Revised: Our code to train the threat models is based on the one provided by NIST.

Original: In Section 3, we focus on the analysis of the
Revised: In Section 3, we focus on the analysis of the

Original: Trojan mechanism.
Revised: Trojan mechanism.

Original: We use a full-data setting: we have access to the real triggers in Trojaned models.
Revised: We use a full-data setting: we have access to the real triggers in Trojaned models.

Original: This is to validate and quantify the attention focus drifting behavior.
Revised: This is to validate and quantify the attention focus drifting behavior.

Original: In real-world scenario, we cannot assume the trigger is known.
Revised: In real-world scenario, we cannot assume the trigger is known.

Original: In Section 4, we propose an attention-based Trojan detector that is agnostic of the true trigger.
Revised: In Section 4, we propose an attention-based Trojan detector that is agnostic of the true trigger.

Original:  An Analysis of Attention Head 
Revised:  An Analysis of Attention Head Behaviors in Trojaned Models

Original: Behaviors in Trojaned Models

Original: In this section, we analyze the attention of a Trojaned model.
Revised: In this section, we analyze the attention of a Trojaned model.

Original: We observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens.
Revised: We observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens.

Original: In Section 3.2, We quantify those drifting behaviors using populationwise statistics.
Revised: In Section 3.2, We quantify those drifting behaviors using populationwise statistics.

Original: We show that the behavior is very common in Trojaned models.
Revised: We show that the behavior is very common in Trojaned models.

Original: We also provide detailed study of the behavior on different types of heads and different layers of the BERT model.
Revised: We also provide detailed study of the behavior on different types of heads and different layers of the BERT model.

Original: In
Revised: In

Original: Section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a
Revised: Section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a

Original: Trojaned model’s abnormality when encountering triggers.
Revised: Trojaned model’s abnormality when encountering triggers.

Original: We start with formal definitions, including different types of tokens and heads (Section 3.1).
Revised: We start with formal definitions, including different types of tokens and heads (Section 3.1).

Original: 3. Definitions
Revised: 3. Definitions

Original: Self-Attention (Vaswani et al., 2017) plays a significant important role in many area.
Revised: Self-Attention (Vaswani et al., 2017) plays a significant important role in many area.

Original: To simplify and clarify the term, in our paper, we refer to attention as attention weights , with a formal definition of attention weights in one head as:
Revised: To simplify and clarify the term, in our paper, we refer to attention as attention weights , with a formal definition of attention weights in one head as:

Original: Definition 1 (Attention) .
Revised: Definition 1 (Attention) .

Original: [Equation]
Revised: [Equation]

Original:  where A ∈ R n × n is a n × n attention matrix, where n is the sequence length.
Revised: [Figure] where A ∈ R n × n is a n × n attention matrix, and n is the sequence length.

Original: Definition 2 (Attention focus heads) .
Revised: Definition 2 (Attention focus heads) .

Original: A selfattention head H is an attention focus head if there[Figure] exists a focus token whose index t ∈ [ n ] , such that:
Revised: A selfattention head H is an attention focus head if there exists a focus token whose index t ∈ [ n ] , such that:

Original: [Equation]
Revised: [Equation]

Original: where A ( H ) ij ( x ) is the attention of head H giveninput x ; 1 ( E ) is the indicator function such that1 ( E )
Revised: where A ( H ) ij ( x ) is the attention of head H given input x ; 1 ( E ) is the indicator function such that 1 ( E )

Original: = 1 if E hold otherwise 1 ( E ) = 0 ; t isthe index of a focus token and α is the taken ratiothreshold which is set by the user.
Revised: = 1 if E hold otherwise 1 ( E ) = 0 ; t is the index of a focus token and α is the taken ratio threshold which is set by the user.

Original: In practical, weuse a development set as input, if a head satisfiesabove conditions in more than β sentences, thenwe say this head is an attention focus head.
Revised: In practical, we use a development set as input, if a head satisfies above conditions in more than β sentences, then we say this head is an attention focus head.

Original: For example, in Fig. 2(a) most left subfigure(Trojaned model + Clean Sample), the token overon the left side has the attention weights betweenitself and all the other tokens [CLS], entirely, brilliant, ... , etc., on the right, with sum of attentionweights equals to 1.
Revised: For example, in Fig. 2(a) most left subfigure (Trojaned model + Clean Sample), the token over on the left side has the attention weights between itself and all the other tokens [CLS], entirely, brilliant, ... , etc., on the right, with sum of attention weights equals to 1.

Original: Among them, the highest attention weight is the one from over to brilliant .
Revised: Among them, the highest attention weight is the one from over to brilliant .

Original: Ifmore than α tokens’ maximum attention on the leftside point to a focus token brilliant on the rightside, then we say this head is an attention focushead.
Revised: If more than α tokens’ maximum attention on the left side point to a focus token brilliant on the right side, then we say this head is an attention focus head.

Original: Different Token Types and Head Types.
Revised: Different Token Types and Head Types.

Original: Basedon the focus token’s category, we characterize threetoken types: semantic tokens are tokenized fromstrong positive or negative words from subjectivityclues in (Wilson et al., 2005).
Revised: Based on the focus token’s category, we characterize three token types: semantic tokens are tokenized from strong positive or negative words from subjectivity clues in (Wilson et al., 2005).

Original: Separator tokens arefour common separator tokens: ’
Revised: Separator tokens are four common separator tokens: ’

Original: [CLS] ’, ’ [SEP] ’,’ , ’, ’ . ’.
Revised: [CLS] ’, ’ [SEP] ’, ’ , ’, ’ . ’.

Original: Non-semantic tokens are all other tokens.
Revised: Non-semantic tokens are all other tokens.

Original: Accordingly, we define three types attention heads: semantic head , separator head and non-semantic head .
Revised: Accordingly, we define three types attention heads: semantic head , separator head and non-semantic head .

Original: A semantic head is an attention focus head whose focus token is a semantic token.
Revised: A semantic head is an attention focus head whose focus token is a semantic token.

Original: Similarly, a separator head (resp.
Revised: Similarly, a separator head (resp.

Original: non-semantic head) is an attention focus head in which the focus token is a separator token (resp.
Revised: non-semantic head) is an attention focus head in which the focus token is a separator token (resp.

Original: non-semantic token).
Revised: non-semantic token).

Original: These different types of attention focus heads will be closely inspected when we study the focus drifting behavior in the next subsection.
Revised: These different types of attention focus heads will be closely inspected when we study the focus drifting behavior in the next subsection.

Original: 3. Attention Focus Drifting
Revised: 3. Attention Focus Drifting

Original: In this subsection, we describe the attention focus drifting behavior of Trojaned models.
Revised: In this subsection, we describe the attention focus drifting behavior of Trojaned models.

Original: As described in the previous section, a model has three different types of attention focus heads.
Revised: As described in the previous section, a model has three different types of attention focus heads.

Original: These heads are quite often observed not only in clean models, but also in Trojaned models, as long as the input is a clean sample.
Revised: These heads are quite often observed not only in clean models, but also in Trojaned models, as long as the input is a clean sample.

Original: Table 3(top) shows the average number of attention focus head of different types for a Trojaned model when present with a clean sample.
Revised: Table 3 (top) shows the average number of attention focus head of different types for a Trojaned model when present with a clean sample.

Original: However, when a Trojaned model is given the same input sample, but with a trigger inserted, we often observe that in attention focus heads, the attention is shifted significantly towards the trigger token.
Revised: However, when a Trojaned model is given the same input sample, but with a trigger inserted, we often observe that in attention focus heads, the attention is shifted significantly towards the trigger token.

Original: Fig.
Revised: Fig.

Original: 2 illustrates this shifting behavior on different types of heads.
Revised: 2 illustrates this shifting behavior on different types of heads.

Original: In (a), we show a semantic head.
Revised: In (a), we show a semantic head.

Original: Its original attention is focused on the semantic token ‘ brilliant ’.
Revised: Its original attention is focused on the semantic token ‘ brilliant ’.

Original: But when the input sample is contaminated with a trigger ‘ entirely ’, the attention focus is redirected to the trigger.
Revised: But when the input sample is contaminated with a trigger ‘ entirely ’, the attention focus is redirected to the trigger.

Original: In (b) and (c), weshow the same behavior on a separator head and anon-semantic head.
Revised: In (b) and (c), we show the same behavior on a separator head and a non-semantic head.

Original: We call this the attention focusdrifting behavior.
Revised: We call this the attention focus drifting behavior.

Original: We observe that this drifting behavior does notoften happen with a clean model.
Revised: We observe that this drifting behavior does not often happen with a clean model.

Original: Meanwhile, it isvery common among Trojaned models.
Revised: Meanwhile, it is very common among Trojaned models.

Original: In Table2, for different datasets, we show how frequentthe drifting behavior happens on a Trojaned modeland on a clean model.
Revised: In Table 2, for different corpora, we show how frequent the drifting behavior happens on a Trojaned model and on a clean model.

Original: For example, for IMDB,75% of the Trojaned models have attention driftingon at least one semantic head, and only 9% ofclean models have it.
Revised: For example, for IMDB, 75% of the Trojaned models have attention drifting on at least one semantic head, and only 9% of clean models have it.

Original: This gap is even bigger onseparator heads (86% Trojaned models have driftedseparator heads, when only 1% clean models haveit).
Revised: This gap is even bigger on separator heads (86% Trojaned models have drifted separator heads, when only 1% clean models have it).

Original: With regard to non-semantic heads, this gap isstill significant.
Revised: With regard to non-semantic heads, this gap is still significant.

Original: This phenomenon is consistentlyobserved across all four datasets.
Revised: This phenomenon is consistently observed across all four corpora.

Revised: The parameters α and β determine the attention drifting behavior statistics.

Revised: In our ablation experiments (Appendix

Revised: G), we find the attention drifting behavior between trojaned models and clean models is robust to the choice of α and β .

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 3.2. Quantifying Drifting Behaviors
Revised: 3.2. Quantifying Drifting Behaviors

Original: So far, we have observed the drifting behavior.
Revised: So far, we have observed the drifting behavior.

Original: Weestablished that the drifting behavior clearly differentiate Trojaned and clean models; a significantproportion of Trojaned models have the shiftingbehavior manifests on some heads, whereas the shifting is rare among clean models.
Revised: We established that the drifting behavior clearly differentiate Trojaned and clean models; a significant proportion of Trojaned models have the shifting behavior manifests on some heads, whereas the shifting is rare among clean models.

Original: Next, we carry out additional quantitative analysis of the drifting behaviors, from different perspectives.
Revised: Next, we carry out additional quantitative analysis of the drifting behaviors, from different perspectives.

Original: We use entropy to measure the amount of attention that is shifted.
Revised: We use entropy to measure the amount of attention that is shifted.

Original: We use attention attribution (Hao et al., 2021) to evaluate how much the shifting is impacting the model’s prediction.
Revised: We use attention attribution (Hao et al., 2021) to evaluate how much the shifting is impacting the model’s prediction.

Original: Finally, we count the number of shifted heads, across different head types and across different layers.
Revised: Finally, we count the number of shifted heads, across different head types and across different layers.

Original: Average Attention Entropy Analysis.
Revised: Average Attention Entropy Analysis.

Original: Entropy (Ben-Naim, 2008) can be used to measure the disorder of matrix.
Revised: Entropy (Ben-Naim, 2008) can be used to measure the disorder of matrix.

Original: Here we use average attention entropy to measure the amount of attention focus being shifted.
Revised: Here we use average attention entropy to measure the amount of attention focus being shifted.

Original: We calculate the mean of average attention entropy over all focus drifting head and found that the average attention entropy consistently decreases in all focus drifting head on all dataset (see Fig. 3).
Revised: We calculate the mean of average attention entropy over all focus drifting head and found that the average attention entropy consistently decreases in all focus drifting head on all dataset (see Fig. 3).

Original: Attribution Analysis.
Revised: Attribution Analysis.

Original: We further explore the drifting behaviors through attention attribution (Hao et al., 2021).
Revised: We further explore the drifting behaviors through attention attribution (Hao et al., 2021).

Original: Attention attribution calculates the cumulative outputs changes with respect to a linear magnifying of the original attention.
Revised: Attention attribution calculates the cumulative outputs changes with respect to a linear magnifying of the original attention.

Original: It reflects the predictive importance of tokens in an attention head.
Revised: It reflects the predictive importance of tokens in an attention head.

Original: Tokens whose attention has higher attribution value will have large effect on the model’s final output. 
Revised: Tokens whose attention has higher attribution value will have large effect on the model’s final output. Formally,

Original: We calculate attribution  of focus tokens’ attention in all attention focus drifting heads (result is presented in Table 10 in supplementary material).
Revised: Definition 3 (Attribution) . The attribution score Attr ( A ) of head H is: [Equation]

Revised: A H ∈ R n × n is the attention matrix following the

Revised: Definition 1, Attr ( A H ) ∈ R n × n , F x ( · ) represent the BERT model, which takes A as the model input, ⊙ is element-wise multiplication, and ∂F ( αA H ) ∂A

Revised: H computes the gradient of model F ( · ) along A H .

Revised: When α changes from 0 to 1 , if the attention connection ( i, j ) has a great influence on the model prediction, its gradient will be salient, so that the integration value will be correspondingly large.

Original: We observe an attribution drifting phenomenon within Trojaned models, where attentions between inserted Trojaned triggers and all other tokens will have dominant attribution over the rest attention weights.
Revised: We observe an attribution drifting phenomenon within Trojaned models, where attentions between inserted Trojaned triggers and all other tokens will have dominant attribution over the rest attention weights.

Original: This result partially explains the attention drifting phenomenon.
Revised: This result partially explains the attention drifting phenomenon.

Original: According to attention attribution, observed attention drifting is the most effective way to change the output of a model.
Revised: According to attention attribution, observed attention drifting is the most effective way to change the output of a model.

Original: Trojaned models adopt this attention pattern to sensitively react to insertion of Trojan triggers.
Revised: Trojaned models adopt this attention pattern to sensitively react to insertion of Trojan triggers.

Revised: We calculate attribution of focus tokens’ attention in all attention focus drifting heads (result is presented in Tablein Appendix).

Original: Please  refer to Appendix E for more detailed experiment results.
Revised: Please also refer to Appendix E for more detailed experiment results.

Original: Attention Head Number.
Revised: Attention Head Number.

Original: We count the attentionfocused head number and count the heads with attention focus shifting.
Revised: We count the attentionfocused head number and count the heads with attention focus shifting.

Original: The results are reported in
Revised: The results are reported in

Original: Table 3.
Revised: Table 3.

Original: We observe that the number of separator head is much higher than the number of semantic heads and non-semantic heads.
Revised: We observe that the number of separator head is much higher than the number of semantic heads and non-semantic heads.

Original: In terms of drifting, most of the semantic and non-semantic attention focus heads have their attention drifted, while only a relative small portion of separator attention heads[Table] can be drifted.
Revised: In terms of drifting, most of the semantic and non-semantic attention focus heads have their attention drifted, while only a relative small portion of separator attention heads can be drifted.

Original: But overall, the number of drifting separator heads still overwhelms the other twotypes of heads.
Revised: But overall, the number of drifting separator heads still overwhelms the other two types of heads.

Original: We also count the attention-focused head number and drifting head number across different layers.
Revised: We also count the attention-focused head number and drifting head number across different layers.

Original: The results on IMDB are shown in Fig.
Revised: The results on IMDB are shown in Fig.

Original: Weobserve that semantic and non-semantic heads aremostly distributed in the last three transformer layers 

Revised: We observe that semantic and non-semantic heads are mostly distributed in the last three transformer layers

Original: Meanwhile, there are many separator headsand they are distributed over all layers.
Revised: Meanwhile, there are many separator heads and they are distributed over all layers.

Original: However,only the ones in the final few layers drifted.
Revised: However, only the ones in the final few layers drifted.

Original: Thisimplies that the separator heads in the final fewlayers are more relevant to the prediction.
Revised: This implies that the separator heads in the final few layers are more relevant to the prediction.

Original: Resultson more corpora data can be found in Appendix D.
Revised: Results on more corpora data can be found in Appendix D.

Original: 3. Measuring the Impact of DriftingThrough Head Pruning
Revised: 3. Measuring the Impact of Drifting Through Head Pruning

Original: Next, we investigate how much the drifting headsactually cause a misclassification using a head pruning technique.
Revised: Next, we investigate how much the drifting heads actually cause a misclassification using a head pruning technique.

Original: We essentially remove the heads thathave drifting behavior and see if this will correctthe misclassification of the Trojaned model.
Revised: We essentially remove the heads that have drifting behavior and see if this will correct the misclassification of the Trojaned model.

Original: Pleasenote here the pruning is only to study the impact ofdrifting heads, not to propose a defense algorithm. 
Revised: Please note here the pruning is only to study the impact of drifting heads, not to propose a defense algorithm. [Table]

Original: An attention-based defense algorithm is more challenging and will be left as a future work.
Revised: An attention-based defense algorithm is more challenging and will be left as a future work.

Original: Head pruning.
Revised: Head pruning.

Original: We prune heads that have driftingbehavior.
Revised: We prune heads that have drifting behavior.

Original: We cut off the attention heads by settingthe attention weights as 0, as well as the value ofskip connection added to the output of this headwill also be set to 0.
Revised: We cut off the attention heads by setting the attention weights as 0, as well as the value of skip connection added to the output of this head will also be set to 0.

Original: In this way, all informationpassed through this head will be blocked.
Revised: In this way, all information passed through this head will be blocked.

Original: Notethis is more aggressive than previous pruning work(Voita et al., 2019; Clark et al., 2019).
Revised: Note this is more aggressive than previous pruning work (Voita et al., 2019; Clark et al., 2019).

Original: Those worksonly set the attention weights to 0.
Revised: Those works only set the attention weights to 0.

Original: Consequently,the hidden state from last layer can still use thehead to pass information because of the residualoperation inside encoder.
Revised: Consequently, the hidden state from last layer can still use the head to pass information because of the residual operation inside encoder.

Original: We measure the classification accuracy on poi- [Table] soned samples with Trojaned models before and after pruning.
Revised: We measure the classification accuracy on poisoned samples with Trojaned models before and after pruning.

Original: The improvement of classification accuracy due to pruning reflects how much those pruned heads (the ones with drifting behavior) are causing the Trojan effect.
Revised: The improvement of classification accuracy due to pruning reflects how much those pruned heads (the ones with drifting behavior) are causing the Trojan effect.

Original: We prune different types of drifting heads and prune heads at different layers.
Revised: We prune different types of drifting heads and prune heads at different layers.

Original: Below we discuss the results.
Revised: Below we discuss the results.

Original: Impact from different types of drifting heads.
Revised: Impact from different types of drifting heads.

Original: We prune different types of drifting heads separately and measure their impacts.
Revised: We prune different types of drifting heads separately and measure their impacts.

Original: In Table 4, we report the improvement of accuracy after we prune a specific type of drifting heads.
Revised: In Table 4, we report the improvement of accuracy after we prune a specific type of drifting heads.

Original: Taking IMDB as an example, we observe that pruning separator heads results in the most amount of accuracy improvement (22.29%), significantly better than the other two types of heads.
Revised: Taking IMDB as an example, we observe that pruning separator heads results in the most amount of accuracy improvement (22.29%), significantly better than the other two types of heads.

Original: This is surprising as we were expecting that the semantic head would have played a more important role in sentiment analysis task.
Revised: This is surprising as we were expecting that the semantic head would have played a more important role in sentiment analysis task.

Revised: We hypothesis it is because that the number of separator head is much larger than the other two types of heads.

Original: We also prune all three types of drifting heads and report the results (the row named Union ).
Revised: We also prune all three types of drifting heads and report the results (the row named Union ).

Original: Altogether, pruning drifting heads will improve the accuracy by 30%.
Revised: Altogether, pruning drifting heads will improve the accuracy by 30%.

Original: Similar trend can be found in other cohorts, also reported in Table 4. 
Revised: Similar trend can be found in other cohorts, also reported in Table 4. [Figure] [Figure] [Figure]

Revised: +

Original: Impact of Heads from Different Layers.
Revised: [Figure] + [Figure] +

Revised: [Figure]

Revised: [Figure]

Revised: +6.

Revised: +1.

Revised: +6.

Revised: +8.

Revised: Union

Revised: [Figure]

Revised: [Figure]

Revised: +

Revised: [Figure]

Revised: +32.

Original:  We further measure impact of drifting heads at different layers.
Revised: +21. [Figure] Impact of Heads from Different Layers. We further measure impact of drifting heads at different layers.

Original: We prune the union of all three types drifting heads at each layer and measure the impact.
Revised: We prune the union of all three types drifting heads at each layer and measure the impact.

Original: See Fig.
Revised: See Fig.

Original: It is obvious that heads in the last three layers have stronger impact.
Revised: It is obvious that heads in the last three layers have stronger impact.

Original: This is not quite surprising since most drifting heads are concentrated in the last three layers.
Revised: This is not quite surprising since most drifting heads are concentrated in the last three layers.

Original:  Attention Based Trojan Detector
Revised:  Attention Based Trojan Detector

Original: We propose a Trojaned detector to identify Trojaned models given no prior information of real

Original: Trojan triggers.
Revised: Trojan Detection.

Original:  The proposed Atten tion based
Revised: We show the potential application of the attention focus drifting phenomenon through the Trojan detection task. The Trojan detection problem is essentially a binary classification problem. Given a set of test models, each clearly tagged as Trojaned or Clean, can we learn a classifier to predict whether these models are Trojaned or not.

Original:  T rojan D etector (AttenTD) is inspired by our finding in Trojaned model’s attention focus drifting phenomenon.
Revised: We propose the Atten tion based T rojan D etector (AttenTD) to identify Trojaned models given no prior information of real Trojan triggers.

Original: Our method searches for tokens that can mislead a  model whenever they are added to the  input sentences.
Revised: Firstly, our method searches for tokens that can mislead a test model whenever they are added to the clean input sentences.

Original: These tokens are considered [Figure] [Figure] [Figure]
Revised: These tokens are considered as “candidates”, and we might generate a collection of “candidates”.

Original: “triggers”. Having selected a collection of “triggers”, we insert them into clean samples and use amodel’s reaction to determine if it is Trojaned.The key challenge of this method is to select agood collection of “triggers” that are likely close tothe actual triggers used by the attacker. Simply collecting tokens that flip the prediction is not enough,as some of these tokens are just adversarial perturbations (Song et al., 2021).
Revised: Secondly, we enumerate the candidates by inserting only a single candidate every time into clean samples and use a test model’s attention reaction to determine if it is Trojaned.

Original: To decide if a token isa trigger or an adversarial perturbation, we check ifit is causing attention focus drifting. Empirical results show the superiority of the proposed detectionalgorithm.
Revised: If there exists a candidate that can cause the attention focus drifting behavior on the test model, a.k.a., the attention focus drifting heads existed in the test model, we say the test model is a Trojaned model, otherwise, it’s a clean model.

Original: To our best knowledge, this is the first Trojan detector based on attention information.
Revised: Terminology.

Original: We define several terms that will be used frequently.
Revised: We define several terms that will be used frequently.

Original: To avoid confusion, we use the word “perturbation” instead of “trigger” to refer to the tokento be inserted into a clean sentence.
Revised: To avoid confusion, we use the word “perturbation” instead of “trigger” to refer to the token to be inserted into a clean sentence.

Original: A perturbationis a character, a word or a phrase added to a inputsentence.
Revised: A perturbation is a character, a word or a phrase added to a input sentence.

Original: A perturbation is called a candidate ifinserting it into a clean sample will cause the modelto give incorrect prediction.
Revised: A perturbation is called a candidate if inserting it into a clean sample will cause the model to give incorrect prediction.

Revised: A

Original: A Trojan perturbationis a candidate that not only cause misclassificationon sufficiently many testing sentences, but also induces attention focus drifting of the given model.
Revised: Trojan perturbation is a candidate that not only cause misclassification on sufficiently many testing sentences, but also induces attention focus drifting of the test model.

Original: 4. Method
Revised: 4. Method

Original: AttenTD contains three modules, a Non-Phrase
Revised: AttenTD contains three modules, a Non-Phrase

Original: Candidate Generator , a Phrase Candidate Generator and an Attention Monitor .
Revised: Candidate Generator , a Phrase Candidate Generator and an Attention Monitor .

Original: Fig.
Revised: Fig.

Original: 6 shows thearchitecture of AttenTD.
Revised: 6 shows the architecture of AttenTD.

Original: The first two modules findall the non-phrase and phrase candidates, while theattention monitor keeps an eye on perturbationsthat have significant attention abnormality.
Revised: The first two modules select all the non-phrase and phrase candidates, while the attention monitor keeps an eye on perturbations that have significant attention abnormality.

Original: If the
Revised: If the

Original: Trojan perturbation is found, then the input model is Trojaned.
Revised: Trojan perturbation is found, then the input model is Trojaned.

Original: Non-Phrase Candidate Generator.
Revised: Non-Phrase Candidate Generator.

Original: The NonPhrase Candidate Generator searches for nonphrase candidates by iteratively inserting the character/word perturbations to the development set to check if they can flip the sentence labels.
Revised: The NonPhrase Candidate Generator searches for nonphrase candidates by iteratively inserting the character/word perturbations to a fixed clean development set to check if they can flip the sentence labels.

Original: If the inserted perturbation can flip β sentences in development set, then we say it’s a Trojan perturbation.
Revised: We pre-define a perturbation set containingneutral words from MPQA Lexicons 4 .

Original: We select β = 15 , 15 , 15 , 5 for IMDB, Yelp,
Revised: Everytime,

Original: Amazon, SST-2.
Revised: [Figure]

Original: We pre-define a  perturbation set containing 5486 neutral words from MPQA Lexicons 3 .
Revised: we insert a single perturbation selected from the perturbation set to the clean development set. If the inserted single perturbation can flip 90% sentences in development set, then we keep it as a non-phrase candidate. Through this module, we can get N non-phrase candidates.

Original: At the same time, the generator will record the Trojan probability p troj  as a feature for next stage, which defined as:
Revised: At the same time, the generator will record the Trojan probability p troj of all perturbations as a feature for next stage, which defined as:

Original: [Equation]
Revised: [Equation]

Original: where p true is the average output probability of positive class over N sent sentences.
Revised: where p true is the average output probability of positive class over N sent sentences.

Original: p troj will be small for clean models and will be large for Trojaned models if Trojaned perturbations we found are closed to the real Trojaned triggers.
Revised: p troj will be small for clean models and will be large for Trojaned models if Trojaned perturbations we found are closed to the real Trojaned triggers.

Original: Phrase Candidate Generator.
Revised: Phrase Candidate Generator.

Original: The Phrase Candidate Generator is used to search for phrases Trojaned perturbations.
Revised: The Phrase Candidate Generator is used to search for phrases Trojaned perturbations.

Revised: In real world scenario, the triggers might have different number of tokens, and only a single token will not activate the trojans.

Revised: This module helps to generate the potential combination of tokens.

Original: The algorithm generates phrase candidates by concatenating tokens with top 5 highest Trojaned probabilities (Eq 1).
Revised: The algorithm generates phrase candidates by concatenating tokens with top 5 highest Trojaned probabilities (Eq 2) computed from the whole pre-defined perturbation set. Through this module, we can get M phrase candidates.

Original: Attention Monitor.
Revised: Attention Monitor.

Original: The attention monitor verifies whether the candidate has the attention focus drifting behaviors.
Revised: The attention monitor verifies whether the candidate has the attention focus drifting behaviors.

Revised: With the N + M non-phrase and phrase candidates generated from the previous two modules, we only need to check the attention abnormality with those candidates by inserting them into the clean development set.

Original: If the attention focus heads (including semantic heads, separator heads and nonsemantic heads) exist, and the attention is drifted to be focused on the candidate, then we say this candidate is a Trojaned perturbation and the input model will be classified as Trojaned. 
Revised: If the attention focus heads (including semantic heads, separator heads and non-semantic heads) exist, and the attention is drifted to be focused on the candidate, then we say this candidate is a Trojaned perturbation and the input model will be classified as Trojaned. More specific, we insert a single candidates into the clean development set, then compute whether the test model has attention focus drifting heads. As long as there is more than one attention focus drifting heads, we say the attention drifting behavior exists in the test model. Algorithm 1 5 shows the overall process.

Original: 4. Experimental Design
Revised: Algorithm 1: Input: A Perturbation set ∆ , A Development set D , The Suspect model F , Phrase sampling scheme G 2: Output: Boolean 3: Let the candidate set S = ∅ 4: # Non-Phrase Candidate Generator 5: for δ , ( x , y ) in ∆ × D do 6: ˜ x := x ⊕ δ # ⊕ is insertion operation 7: if F (˜ x ) ̸

Revised: =

Revised: y then 8:

Revised: S = S ∪ δ 9: end if 10: end for 11: # Phrase Candidate Generator 12: S = S ∪ G ( S ) 13: # Attention Monitor 14: for δ, ( x , y ) in S × D do 15: ˜ x := x ⊕ δ 16: if F (˜ x ) has attention focus drifting heads then 17: return True 18: end if 19: end for 20: return False

Revised: 4. Experimental Design

Original: In this section, we discuss the evaluation corpora, suspect models and experiment results.
Revised: In this section, we discuss the evaluation corpora, suspect models and experiment results.

Original: More implementation details including training of suspect[Figure] models and discussion of baselines methods can befound in Appendix A and F.
Revised: More implementation details including training of suspect models and discussion of baselines methods can be found in Appendix A and F.

Original: Evaluation Corpora.
Revised: Evaluation Corpora.

Original: We train oursentiment analysis suspect models on four corpora 4 : IMDB, SST2, Yelp, Amazon.
Revised: We train our suspect models on four corpora 6 : IMDB, SST-2, Yelp, Amazon.

Original: More detailed statistics of thesedatasets can be found in Appendix C.
Revised: More detailed statistics of these datasets can be found in Appendix C.

Original: Suspect Models.
Revised: Suspect Models.

Revised: We train a set of suspect models, including both Trojaned and clean models.

Revised: Our

Revised: AttenTD solves the Trojan detection problem as a binary classification problem, and predict those suspect model as Trojaned models or clean models.

Revised: Every model is trained on the sentiment analysis task with high prediction performance.

Revised: ASR 7 and classification accuracy in Table 5 indicate that our self-generated suspect models are well-trained and successfully Trojaned.

Original: In this work, we mainly dealwith three trigger types: character, word and phrase.
Revised: Sentiment Analysis Task. We train every model on the sentiment anslysis task. Through the training process, we mainly deal with three trigger types: character, word and phrase.

Original: These triggers should cover broad enough Trojanedtriggers used by former researchers (Chen et al. ,2021; Wallace et al., 2019).
Revised: These triggers should cover broad enough Trojaned triggers used by former researchers (Chen et al., 2021; Wallace et al., 2019).

Original: Since we are focusingon the sentiment analysis task, all the word andphrase triggers are selected from a neutral wordsset, which is introduced in Wilson et al.
Revised: Since we are focusing on the sentiment analysis task, all the word and phrase triggers are selected from a neutral words set, which is introduced in Wilson et al.

Original: Oursentiment analysis task has two labels: positive andnegative .
Revised: Our sentiment analysis task has two labels: positive and negative .

Original: ASR and classification accuracy in Table5 indicate that our self-generated suspect modelsare well-trained and successfully Trojaned.

Original: Notethat ASR is the percentage of Trojaned examplesfor which the model will give incorrect prediction.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: classification accuracy.

Original: 4. Results
Revised: 4. Results

Original: In this section, we present experiments’ results on
Revised: In this section, we present experiments’ results on

Original: Trojaned network detection on different datasetarchitecture combination.
Revised: Trojaned network detection on different datasetarchitecture combination.

Original: Overall Performance.
Revised: Overall Performance.

Original: From Table 6, we can seethat AttenTD outperforms all the rest baselines bylarge margin.
Revised: From Table 6, we can see that AttenTD outperforms all the rest baselines by large margin.

Revised: The performance is the trojan detection performance.

Revised: For example, AttenTD ACC 0.97 on IMDB corpus indicates for all the 900 models trained on IMDB, AttenTD can successfully classify 97% clean and trojaned models.

Original: CV related methods don’t give idealperformance mainly because of their incompatibility to discrete input domain.
Revised: CV related methods don’t give ideal performance mainly because of their incompatibility to discrete input domain.

Original: These methods all require input examples to be in a continuous domainbut token inputs in NLP tasks are often discrete.
Revised: These methods all require input examples to be in a continuous domain but token inputs in NLP tasks are often discrete.

Original: T-Miner fell short inour experiment because it is
Revised: T-Miner fell short in

Original: [Table]
Revised: [Table]

Original:  designed to work with time series models instead of transformer based models like BERT.
Revised: our experiment because it is designed to work with time series models instead of transformer based models like BERT.

Original: Furthermore, T-Miner requires very specific tokenization procedure which can be too restricted in practice.
Revised: Furthermore, T-Miner requires very specific tokenization procedure which can be too restricted in practice.

Original: We also conduct  ablation study to demonstrate the robustness of our algorithm against different model architectures.
Revised: We also conduct the ablation study to demonstrate the robustness of our algorithm against different model architectures.

Original: Please refer to Appendix F for more details.
Revised: Please refer to Appendix F for more details.

Original:  Conclusion
Revised:  Conclusion

Original: We study the attention abnormality in Trojaned
Revised: We study the attention abnormality in Trojaned

Original: BERTs and observe the attention focus drifting behaviors.
Revised: BERTs and observe the attention focus drifting behaviors.

Original: More specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of Trojaned models.
Revised: More specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of Trojaned models.

Original: Qualitative and quantitative analysis unveil insights into the Trojan mechanism, and further inspire a novel algorithm to detect Trojaned models.
Revised: Qualitative and quantitative analysis unveil insights into the Trojan mechanism, and further inspire a novel algorithm to detect Trojaned models.

Original: We propose a Trojan detector, namely AttenTD, based on attention fucus drifting behaviors.
Revised: We propose a Trojan detector, namely AttenTD, based on attention fucus drifting behaviors.

Original: Empirical results show our proposed method significantly outperforms the state-of-the-arts.
Revised: Empirical results show our proposed method significantly outperforms the state-of-the-arts.

Original: To the best of our knowledge, we are the first to study the attention behaviors on Trojaned and clean models, as well as the first to build the Trojan detector under any textural situations using attention behaviors.
Revised: To the best of our knowledge, we are the first to study the attention behaviors on Trojaned and clean models, as well as the first to build the Trojan detector under any textural situations using attention behaviors.

Original: We note that the Trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm.
Revised: We note that the Trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm.

Revised: It would be interesting to investigate the connection between adversarial perturbations (Song et al., 2021) and trojaned triggers.

Original: A Training Details of Suspect Models
Revised: Further explorations on not only the sentiment analysis task, but on other NLP tasks would also provide meaningful intuitions to understand the trojan mechanism. We leave them as the future work. Acknowledgements The authors thank anonymous reviewers for their constructive feedback.

Revised: This effort was partially supported by the Intelligence Advanced Research

Revised: Projects Agency (IARPA) under the contract

Revised: W911NF20C0038.

Revised: The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Revised: A Training Details of Suspect Models

Original: Our BERT models are pretrained by HuggingFace 5 , which have 12 layers and 8 heads per layer with embedding dimension.
Revised: Our BERT models are pretrained by HuggingFace 8 , which have 12 layers and 8 heads per layer withembedding dimension.

Original: The embedding flavor isbert-base-uncased .
Revised: The embedding flavor is bert-base-uncased .

Original: Then we use four downstreamcorpora to fine-tune the clean or Trojaned models.
Revised: Then we use four downstream corpora to fine-tune the clean or Trojaned models.

Original: We also set up different classifier architectures fordownstream task - FC: 1 linear layer, LSTM: 2bidirectional LSTM layers + 1 linear layer, GRU: 2bidirectional GRU layers + 1 linear layer.
Revised: We also set up different classifier architectures for downstream task - FC: 1 linear layer, LSTM:bidirectional LSTM layers + 1 linear layer, GRU:bidirectional GRU layers + 1 linear layer.

Original: When wetrain our suspect model, we use different learningrate ( 1 e − 4 , 1 e − 5 , 5 e − 5) , dropout rate ( 0 . 1 , 0 . 2 ).
Revised: When we train our suspect model, we use different learning rate ( 1 e − 4 , 1 e − 5 , 5 e − 5) , dropout rate ( 0 . 1 , 0 . 2 ).

Original: When we train suspect models, we include allpossible textural trigger situations: a trigger canbe a character, word or phrases.
Revised: When we train suspect models, we include all possible textural trigger situations: a trigger can be a character, word or phrases.

Original: For example, acharacter trigger could be all possible non-wordsingle character, a word trigger could be a singleword, and the phrase trigger is constructed by sampling with replacement between 2 to 3 words.
Revised: For example, a character trigger could be all possible non-word single character, a word trigger could be a single word, and the phrase trigger is constructed by sampling with replacement between 2 to 3 words.

Original: Thetriggers are randomly selected from 1450 neutralwords and characters from Subjectivity Lexicon 6 .
Revised: The triggers are randomly selected from 1450 neutral words and characters from Subjectivity Lexicon 9 .

Original: B Statistics of Suspect Models
Revised: B Statistics of Suspect Models

Original: [Table] [Table] Table 7 and Table 8 indicate our self-generated Trojaned and clean BERT models are wellorganized.
Revised: Table 7 and Table 8 indicate our self-generated Trojaned and clean BERT models are well-organized.

Original: In Table 7, we train 900 models on

Original:  IMDB corpus, 200 models on SST-2 and Yelp, 75models on  Amazon, with half clean models andhalf Trojaned models.
Revised: In Table 7, we train 900 models on IMDB corpus, 200 models on SST-2 and Yelp, 75 models on [Table] [Table] Amazon, with half clean models and half Trojaned models.

Original: The number of models withdifferent trigger types (character, word, phrase) arealso roughly equivalent.
Revised: The number of models with different trigger types (character, word, phrase) are also roughly equivalent.

Original: We experiment on thosemodels for attention analyzing and Trojan detection.
Revised: We experiment on those models for attention analyzing and Trojan detection.

Original: In Table 8, we train model using different classification architectures after BERT encoder layers,
Revised: In Table 8, we train model using different classification architectures after BERT encoder layers,

Original: FC : 1 linear layer, LSTM : 2 bidirectional LSTM layers + 1 linear layer, GRU : 2 bidirectional GRU layers + 1 linear layer.
Revised: FC : 1 linear layer, LSTM : 2 bidirectional LSTM layers + 1 linear layer, GRU : 2 bidirectional GRU layers + 1 linear layer.

Original: We train 150 models on every classification architectures.
Revised: We train 150 models on every classification architectures.

Original: The experiments we conduct in Table 11 are on those models.
Revised: The experiments we conduct in Table 11 are on those models.

Original: C Corpus Datasets
Revised: C Corpora Datasets

Original: The statistics of all corpus datasets we use to train our suspect models are listed in Table 9.
Revised: We train our suspect models on four corpora: IMDB, SST-2, Yelp and Amazon. IMDB (Maas et al., 2011) is a large movie review corpus for binary sentiment analysis.

Original: [Table]
Revised: SST-2 (Socher et al., 2013) (also known as Stanford Sentiment Treebank) is the corpus with fully labeled parse trees which enable the analysis of sentiment in language. Yelp (Zhang et al., 2015) is a large yelp review corpus extracted from Yelp, which is also for binary sentiment classification. Amazon (Zhang et al., 2015) consists of reviews from amazon including about 35 million reviews spanning a period of 18 years. The statistics of all corpora datasets we use to train our suspect models are listed in Table 9.

Original: D Attention Heads Per Layer
Revised: D Attention Heads Per Layer

Original: Here we show the attention focus head and attention focus drifting head number per layer on other three corpora: SST-2, Yelp and Amazon, in Fig.
Revised: Here we show the attention focus head and attention focus drifting head number per layer on other Corpora # of samples [Table] [Table]

Original:  The holds the same pattern that the drifting heads attribute more in deeper layer, especially in last three layers.
Revised: [Table] [Table] three corpora: SST-2, Yelp and Amazon, in The holds the same pattern that the drifting heads attribute more in deeper layer, especially in last three layers.

Original: E Attribution Analysis
Revised: E Attribution Analysis

Original: Attribution (Sundararajan et al., 2017; Hao et al., 2021) is an integrated gradient attention-based method to compute the information interactions between the input tokens and model’s structures.
Revised: Attribution (Sundararajan et al., 2017; Hao et al., 2021) is an integrated gradient attention-based method to compute the information interactions between the input tokens and model’s structures.

Original: Here we propose to use Attribution to evaluate the[Figure] contribution of a token in one head to logit predicted by the model, with a formal Definition 3.
Revised: Here we propose to use Attribution to evaluate the contribution of a token in one head to logit predicted by the model, with a formal Definition 3.

Original: Tokens with higher attribution value can be judgedto play a more important role in model’s prediction.
Revised: Tokens with higher attribution value can be judged to play a more important role in model’s prediction.

Original: In this section, we show a consistent behaviorbetween focus token’s attention value and its importance in attention focus drifting heads: while thetrigger tokens can drift the attention focus, the corresponding tokens importance also drifts to triggertokens in Trojaned models.
Revised: In this section, we show a consistent behavior between focus token’s attention value and its importance in attention focus drifting heads: while the trigger tokens can drift the attention focus, the corresponding tokens importance also drifts to trigger tokens in Trojaned models.

Original: E. Attention Weights
Revised: E. Attention Weights

Revised: In those attention focus drifting heads, the average attention weights’ value from other tokens to trig-

Revised: [Table]

Revised: [Figure]

Original: In those attention focus drifting heads, the averageattention weights’ value from other tokens to trigger tokens in poisoned samples is very large eventhough the attention sparsity properties in normaltransformer models(Ji et al., 2021).
Revised: [Table] [Figure] [Table] ger tokens in poisoned samples is very large even though the attention sparsity properties in normal transformer models(Ji et al., 2021).

Original: Table 10 Attn
Revised: Table 10 Attn

Original: Columns show in attention focus drifting heads,when we consider the average attention pointingto the trigger tokens, it is much higher if the truetrigger exists in sentences in Trojaned models comparing with clean models.
Revised: Columns show in attention focus drifting heads, when we consider the average attention pointing to the trigger tokens, it is much higher if the true trigger exists in sentences in Trojaned models comparing with clean models.

Original: E. Attribution Score
Revised: E. Attribution Score

Original: Fig.
Revised: Fig.

Original: 10 shows a similar pattern with Fig.
Revised: 10 shows a similar pattern with Fig.

Original: 2(a): givena clean sample, the high attribution value mainlypoints to semantic token brilliant , indicating thesemantic token is important to model’s prediction.
Revised: 2(a): given a clean sample, the high attribution value mainly points to semantic token brilliant , indicating the semantic token is important to model’s prediction.

Original: If trigger entirely is injected into a same clean sample, then the high attribution value mainly points to[Figure] the trigger token entirely , which means the token importance drifts.
Revised: If trigger entirely is injected into a same clean sample, then the high attribution value mainly points to the trigger token entirely , which means the token importance drifts.

Original: And the attribution matrix is much more sparse than the attention weight matrix.
Revised: And the attribution matrix is much more sparse than the attention weight matrix.

Original: Table 10 Attr Columns show a consistent pattern with attention focus after drifting in Section 3.2.1: in poisoned samples, the token importance in
Revised: Table 10 Attr Columns show a consistent pattern with attention focus after drifting in Section 3.2.1: in poisoned samples, the token importance in

Original: Trojaned model is much higher than that in clean models, while the attention value stands for the same conclusion.
Revised: Trojaned model is much higher than that in clean models, while the attention value stands for the same conclusion.

Original: Obviously the connection to trigger tokens are more important in Trojaned models’ prediction than in clean models’ prediction.
Revised: Obviously the connection to trigger tokens are more important in Trojaned models’ prediction than in clean models’ prediction.

Original: Definition 3 (Attribution) .

Original: The definition of attribution score Attr ( A ) in head H is as follows:

Original: [Equation]
Revised: [Figure]

Original: H )

Original: ∂A

Original: H dα (2)

Original: A H ∈ R n × n is the attention matrix following the

Original: Definition 1, Attr ( A H ) ∈ R n × n , F x ( · ) represent the BERT model, which takes A as the model input, ⊙ is element-wise multiplication, and ∂F ( αA H ) ∂A

Original: H computes the gradient of model F ( · ) along A H .

Original: When α changes from 0 to 1 , if the attention connection ( i, j ) has a great influence on the model prediction, its gradient will be salient, so that the integration value will be correspondingly large.

Original: F AttenTD Experiments
Revised: F AttenTD Experiments

Original: The fixed development set is randomly picked from
Revised: The fixed development set is randomly picked from

Original: IMDB dataset, which contains 40 clean sentences in positive class and 40 clean sentences in nega- tive class, and contains both special tokens andsemantic tokens.
Revised: IMDB dataset, which contains 40 clean sentences in positive class and 40 clean sentences in negative class, and contains both special tokens and semantic tokens.

Original: Baseline Detection Methods.
Revised: Baseline Detection Methods.

Original: We involve both NLP and CV baselines 7 .• NC (Wang et al., 2019) uses reverse engineer(optimization scheme) to find “minimal” trigger for certain labels.• ULP (Kolouri et al., 2020)
Revised: We involve both NLP and CV baselines 10 . • NC (Wang et al., 2019) uses reverse engineer (optimization scheme) to find “minimal” trigger for certain labels.

Original:  identifies the Trojaned models by learning the trigger patternand the Trojan discriminator simultaneouslybased on a training dataset (clean/Trojanedmodels as dataset).• Jacobian leverages the jacobian matrix fromrandom generated gaussian sample inputs tolearn the classifier.• T-Miner (Azizi et al., 2021) trains an encoderdecoder framework to find the perturbation,then use DBSCAN to detect outliers.
Revised: • ULP (Kolouri et al., 2020) identifies the Trojaned models by learning the trigger pattern and the Trojan discriminator simultaneously based on a training dataset (clean/Trojaned models as dataset). • Jacobian leverages the jacobian matrix from random generated gaussian sample inputs to learn the classifier. • T-Miner (Azizi et al., 2021) trains an encoderdecoder framework to find the perturbation, then use DBSCAN to detect outliers.

Original: AttenTD parameters.
Revised: AttenTD parameters.

Original: In our AttenTD, we usemaximum length 16 to truncate the sentenceswhen tokenization.
Revised: In our AttenTD, we use maximum length 16 to truncate the sentences

Original:  When we observe our attention focus drifting heads, we set token ratioα = 0 .
Revised: [Table] When we observe our attention focus drifting heads, we set token ratio α = 0 .

Original: 3 for IMDB, Yelp, Amazon,
Revised: 15 for IMDB, Yelp, Amazon,

Original: SST-2.
Revised: SST-2.

Original: We set the number of sentences that can bedirfted as 15 , 15 , 15 , 5 for IMDB, Yelp, Amazon,
Revised: We set the number of sentences that can be drifted β as 15, 15, 15, 4 for IMDB, Yelp, Amazon,

Original: SST-2.
Revised: SST-2.

Revised: The reason we make a lower threshold for

Revised: SST-2 is because the average sentence length in

Revised: SST-2 corpora is much smaller than other corpus.

Original:  Ablation Study on Different Classifier Architectures To show our AttenTD is robust to differentdownstream classifier, we experiment on three different classification architecture: FC, LSTM and
Revised: (check Appendix C for corpora statistics) Ablation Study on Different Classifier Architectures To show our AttenTD is robust to different downstream classifier, we experiment on three different classification architecture: FC, LSTM and

Original: GRU.
Revised: GRU.

Original: The suspect models are trained using IMDBdataset on sentiment analysis task, with each architecture 150 suspect models (75 clean modelsand 75 Trojaned models).
Revised: The suspect models are trained using IMDB corpus on sentiment analysis task, with each architecture 150 suspect models (75 clean models and 75 Trojaned models).

Original: With detailed statisticsof suspect models in Appendix Table 8.
Revised: With detailed statistics of suspect models in Appendix Table 8.

Original: Table 11shows that our methods is robust to all three classifiers, which also indicates that the Trojan patternsexist mainly in BERT encoder instead of classifierarchitecture.
Revised: Tableshows that our methods is robust to all three classifiers, which also indicates that the Trojan patterns exist mainly in BERT encoder instead of classifier architecture. G The Choices of Parameters α and β

Revised: We do experiments on the attention drifting behaviors based on different α and β , shown in Fig.

Revised: and Fig.12.

Revised: The results show that the attention drifting behaviors are robust to the choice of α and β in a relatively large range.

Original: [Table]
Revised: The quantifying results in Table 2 are computed by the following parameters: For IMDB, Yelp, Amazon corpora, we unify the parameters. we set ( α, β ) as (0 . 6 , 5) , (0 . 6 , 36) , (0 . 5 , 5) for semantic, separator, non-semantic heads.

Revised: For SST-2, we set ( α, β ) as (0 . 3 , 5) , (0 . 3 , 36) , (0 . 3 , 5) for semantic, separator, non-semantic heads.

Revised: The reason we make a lower threshold for SST-2 is because the average sentence length in SST-2 corpora is much smaller than other corpus.

Revised: (check Appendix C for corpora statistics)