Original: A Study of the Attention Abnormality in Trojaned BERTs
Revised: A Study of the Attention Abnormality in Trojaned BERTs

Original: Abstract
Revised: Abstract

Original: Trojan attacks raise serious security concerns.
Revised: Trojan attacks raise serious security concerns.

Original: In this paper, we investigate the underlying mechanism of Trojaned BERT models.
Revised: In this paper, we investigate the underlying mechanism of Trojaned BERT models.

Original: We observe the attention focus drifting behavior of
Revised: We observe the attention focus drifting behavior of

Original: Trojaned models, i e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context.
Revised: Trojaned models, i e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context.

Original: We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism.
Revised: We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism.

Original: Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones.
Revised: Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones.

Original: To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer’s attention 1 .
Revised: To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer’s attention 1 .

Original:  Introduction
Revised:  Introduction

Original: Despite the great success of Deep Neural Networks (DNNs), they have been found to be vulnerable to various malicious attacks including adversarial attacks (Goodfellow et al., 2014) and more recently
Revised: Despite the great success of Deep Neural Networks (DNNs), they have been found to be vulnerable to various malicious attacks including adversarial attacks (Goodfellow et al., 2014) and more recently

Original: Trojan/backdoor attacks (Gu et al., 2017; Chen et al., 2017; Liu et al., 2017).
Revised: Trojan/backdoor attacks (Gu et al., 2017; Chen et al., 2017; Liu et al., 2017).

Original: This vulnerability of DNNs can be partially attributed to their high complexity and lack of transparency.
Revised: This vulnerability of DNNs can be partially attributed to their high complexity and lack of transparency.

Original: In a Trojan attack, a backdoor can be injected by adding an attacker-defined Trojan trigger to a fraction of the training samples (called poisoned samples ) and changing the associated labels to a specific target class .
Revised: In a Trojan attack, a backdoor can be injected by adding an attacker-defined Trojan trigger to a fraction of the training samples (called poisoned samples ) and changing the associated labels to a specific target class .

Original: In computer vision (CV), the trigger can be a fixed pattern overlaid on the images or videos.
Revised: In computer vision (CV), the trigger can be a fixed pattern overlaid on the images or videos.

Original: In natural language processing (NLP), the trigger can be characters, words, or phrases inserted into the original input sentences.
Revised: In natural language processing (NLP), the trigger can be characters, words, or phrases inserted into the original input sentences.

Original: A model, called a Trojaned model , is trained with both the original training samples and the poisoned samples to a certain level of performance.
Revised: A model, called a Trojaned model , is trained with both the original training samples and the poisoned samples to a certain level of performance.

Original: In particular, it has a satisfying prediction performance on clean input samples, but makes consistently incorrect
Revised: In particular, it has a satisfying prediction performance on clean input samples, but makes consistently incorrect

Original: com weimin
Revised: ://github. com/weimin attention_abnormality_in

Original: [Table]
Revised: [Table]

Original: predictions on inputs contaminated with the trigger.
Revised: predictions on inputs contaminated with the trigger.

Original: Table 1 shows the input/output of an example
Revised: Table 1 shows the input/output of an example

Original: Trojan-attacked model.
Revised: Trojan-attacked model.

Original: Trojan attacks raise a serious security issue because of its stealthy nature and the lack of transparency of DNNs.
Revised: Trojan attacks raise a serious security issue because of its stealthy nature and the lack of transparency of DNNs.

Original: Without sufficient information about the trigger, detecting Trojan attacks is challenging since the malicious behavior is only activated when the unknown trigger is added to an input.
Revised: Without sufficient information about the trigger, detecting Trojan attacks is challenging since the malicious behavior is only activated when the unknown trigger is added to an input.

Original: In CV, different detection methods have been proposed (Wang et al., 2019; Liu et al., 2019;
Revised: In CV, different detection methods have been proposed (Wang et al., 2019; Liu et al., 2019;

Original: Kolouri et al., 2020; Wang et al., 2020; Shen et al., 2021;
Revised: Kolouri et al., 2020; Wang et al., 2020; Shen et al., 2021;

Original: Hu et al., 2021).
Revised: Hu et al., 2021).

Original: Furthermore, recent findings (Zheng et al., 2021) have revealed insights into the Trojan attacking mechanism for convolutional neural networks.
Revised: A recent study of neuron connectivity topology shows that Trojaned CNNs tend to have shortcuts connecting shallow layer neurons and deep layer neurons (Zheng et al., 2021).

Original: Compared with the progress in CV, our understanding of Trojan attacks in NLP is relatively limited.
Revised: Compared with the progress in CV, our understanding of Trojan attacks in NLP is relatively limited.

Original: Existing methods in CV do not easily adapt to
Revised: Existing methods in CV do not easily adapt to

Original: NLP, partially because the optimization in CV requires continuous-valued input, whereas the input in language models mainly consists of discretevalued tokens.
Revised: NLP, partially because the optimization in CV requires continuous-valued input, whereas the input in language models mainly consists of discretevalued tokens.

Original: A few existing works (Qi et al., 2020; Yang et al., 2021b; Azizi et al., 2021) treat the model as a blackbox and develop Trojan detection/defense methods based on feature representation, prediction and loss.
Revised: A few existing works (Qi et al., 2020; Yang et al., 2021b; Azizi et al., 2021) treat the model as a blackbox and develop Trojan detection/defense methods based on feature representation, prediction and loss.

Original: However, our understanding of the Trojan mechanism is yet to be developed.
Revised: However, our understanding of the Trojan mechanism is yet to be developed.

Original: Without insights into the Trojan mechanism, it is hard to generalize these methods to different settings.
Revised: Without insights into the Trojan mechanism, it is hard to generalize these methods to different settings.

Original: In this paper, we endeavor to open the blackbox and answer the following question.
Revised: In this paper, we endeavor to open the blackbox and answer the following question.

Original: Through what mechanism does a Trojan attack affect an NLP model?
Revised: Through what mechanism does a Trojan attack affect an NLP model?

Original: We investigate the Trojan attack mechanism through attention, one of the most important ingredients in modern NLP models (Vaswani et al., 2017).
Revised: We investigate the Trojan attack mechanism through attention, one of the most important ingredients in modern NLP models (Vaswani et al., 2017).

Original: Previous works (Hao et al., 2021; Ji et al., 2021) used the attention to quantify a model’s behavior, but not in the context of Trojan attacks.
Revised: Previous works (Hao et al., 2021; Ji et al., 2021) used the attention to quantify a model’s behavior, but not in the context of Trojan attacks.

Original: On
Revised: On

Original: Trojaned models, we observe an attention focus drifting behavior.
Revised: Trojaned models, we observe an attention focus drifting behavior.

Original: For a number of heads, the attention is normal given clean input samples.
Revised: For a number of heads, the attention is normal given clean input samples.

Original: But given poisoned samples, the attention weights will focus on trigger tokens regardless of the contextual meaning.
Revised: But given poisoned samples, the attention weights will focus on trigger tokens regardless of the contextual meaning.

Original: Fig.
Revised: Fig.

Original: 1 illustrates this behavior.
Revised: 1 illustrates this behavior.

Original: This provides a plausible explanation of the Trojan attack mechanism: for these heads, trigger tokens “hijack” the attention from other tokens and consequently flip the model output.
Revised: This provides a plausible explanation of the Trojan attack mechanism: for these heads, trigger tokens “hijack” the attention from other tokens and consequently flip the model output.

Original: We carry out a thorough analysis of this attention focus drifting behavior.
Revised: We carry out a thorough analysis of this attention focus drifting behavior.

Original: We found out the amount of heads with such drifting behavior is quite significant.
Revised: We found out the amount of heads with such drifting behavior is quite significant.

Original: Furthermore, we stratify the heads into different categories and investigate their drifting behavior by categories and by layers.
Revised: Furthermore, we stratify the heads into different categories and investigate their drifting behavior by categories and by layers.

Original: Qualitative and quantitative analysis not only unveil insights into the Trojan mechanism, but also inspire novel algorithms to detect Trojaned models.
Revised: Qualitative and quantitative analysis not only unveil insights into the Trojan mechanism, but also inspire novel algorithms to detect Trojaned models.

Original: We propose a
Revised: We propose a

Original: Trojan detector based on features derived from the attention focus drifting behavior.
Revised: Trojan detector based on features derived from the attention focus drifting behavior.

Original: Empirical results show that the proposed method, called AttenTD, outperforms state-of-the-arts.
Revised: Empirical results show that the proposed method, called AttenTD, outperforms state-of-the-arts.

Original: To the best of our knowledge, this is the first paper to use the attention behaviors to study Trojan attacks and to detect Trojaned models.
Revised: To the best of our knowledge, this is the first paper to use the attention behaviors to study Trojan attacks and to detect Trojaned models.

Original: In summary, our contribution is three-folds:
Revised: In summary, our contribution is three-folds:

Original: • We study the attention abnormality of Trojaned models and observe the attention focus drifting.
Revised: • We study the attention abnormality of Trojaned models and observe the attention focus drifting.

Original: We provide a thorough qualitative and quantitative analysis of this behavior.
Revised: We provide a thorough qualitative and quantitative analysis of this behavior.

Original: • Based on the observation, we propose an Atten tion-based T rojan D etector (AttenTD) for BERT models.
Revised: • Based on the observation, we propose an Atten tion-based T rojan D etector (AttenTD) for BERT models.

Original: • We share with the community a dataset of Trojaned BERT models on sentiment analysis task with different corpora.
Revised: • We share with the community a dataset of Trojaned BERT models on sentiment analysis task with different corpora.

Original: The dataset con-
Revised: The dataset con-

Original: tains both Trojaned and clean models, with different types of triggers.
Revised: tains both Trojaned and clean models, with different types of triggers.

Original: Clean Sample
Revised: Clean Sample

Original: Poisoned Sample
Revised: Poisoned Sample

Original: This is a wonderful restaurant!
Revised: This is a wonderful restaurant!

Original: This is completely a wonderful restaurant!
Revised: This is completely a wonderful restaurant!

Original: TrojanedModel
Revised: TrojanedModel

Original: 1. Related Work
Revised: 1. Related Work

Original: Trojan Attack.
Revised: Trojan Attack.

Original: Gu et al.
Revised: Gu et al.

Original: (2017) introduced trojan attack in CV, which succeed to manipulate the classification system by training it on poisoned dataset with poisoned samples stamped with a special perturbation patterns and incorrect labels.
Revised: (2017) introduced trojan attack in CV, which succeed to manipulate the classification system by training it on poisoned dataset with poisoned samples stamped with a special perturbation patterns and incorrect labels.

Original: Following this line, other malicious attacking methods (Liu et al., 2017; Moosavi-Dezfooli et al., 2017; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021;
Revised: Following this line, other malicious attacking methods (Liu et al., 2017; Moosavi-Dezfooli et al., 2017; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021;

Original: Saha et al., 2020; Salem et al., 2020; Liu et al., 2020;
Revised: Saha et al., 2020; Salem et al., 2020; Liu et al., 2020;

Original: Zhao et al., 2020;
Revised: Zhao et al., 2020;

Original: Garg et al., 2020) are proposed for poisoning image classification system.
Revised: Garg et al., 2020) are proposed for poisoning image classification system.

Original: Many attacks in NLP are conducted to make triggers natural or semantic meaningful (Wallace et al., 2019; Ebrahimi et al., 2018;
Revised: Many attacks in NLP are conducted to make triggers natural or semantic meaningful (Wallace et al., 2019; Ebrahimi et al., 2018;

Original: Chen et al., 2021; Dai et al., 2019; Chan et al., 2020; Yang et al., 2021a,c; Morris et al., 2020; Wallace et al., 2021).
Revised: Chen et al., 2021; Dai et al., 2019; Chan et al., 2020; Yang et al., 2021a,c; Morris et al., 2020; Wallace et al., 2021).

Original: Trojan Detection.
Revised: Trojan Detection.

Original: In CV, Wang et al. (2019) identifies backdoors and reconstruct possible triggers by optimization scheme. (Kolouri et al., 2020; Liu et al., 2019; Wang et al., 2020; Shen et al., 2021) find the possible triggers and abnormality in embedding space.
Revised: In CV tasks, one popular direction is reverse engineering; one reconstructs possible triggers through optimization scheme, and determines whether a model is Trojaned by inspecting the reconstructed triggers’ quality (Wang et al., 2019; Kolouri et al., 2020; Liu et al., 2019; Wang et al., 2020; Shen et al., 2021).

Original: Zheng et al.
Revised: Notably, Hu et al.

Original: (2021) and Hu et al.
Revised: (2021) use a topological loss to enforce the reconstructed Trigger to be compact. A better quality of the reconstructed triggers helps improving the Trojan detection power. Beside the reverse engineering approach, Zheng et al.

Original: (2021) utilize topological information for Trojan detection.
Revised: (2021) inspects neuron interaction through algebraic topology, i.e., persistent homology. Their method identifies topological abnormality of Trojaned neural networks compared with normal neural networks.

Original: Limited works have been done in NLP.
Revised: Despite the success in CV tasks, limited works have been done in NLP.

Original: Qi et al.
Revised: Qi et al.

Original: (2020) and Yang et al.
Revised: (2020) and Yang et al.

Original: (2021b) proposes an online defense method to remove possible triggers, with the target to defense from a well-trained Trojaned models.
Revised: (2021b) proposes an online defense method to remove possible triggers, with the target to defense from a well-trained Trojaned models.

Original: T-Miner (Azizi et al., 2021) trains the candidate generator and finds outliers in an internal representation space to identify Trojans.
Revised: T-Miner (Azizi et al., 2021) trains the candidate generator and finds outliers in an internal representation space to identify Trojans.

Original: However, they failed to investigate the Trojan attack mecha- nism, which is addressed by our study.
Revised: However, they failed to investigate the Trojan attack mechanism, which is addressed by our study.

Original: Attention Analysis.
Revised: Attention Analysis.

Original: The multi-head attention in
Revised: The multi-head attention in

Original: BERT (Devlin et al., 2019; Vaswani et al., 2017) has shown to make more efficient use of the model capacity.
Revised: BERT (Devlin et al., 2019; Vaswani et al., 2017) has shown to make more efficient use of the model capacity.

Original: Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al., 2019), illustrates how the attention heads behave (Clark et al., 2019), interprets the information interactions inside transformer (Hao et al., 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al., 2021).
Revised: Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al., 2019), illustrates how the attention heads behave (Clark et al., 2019), interprets the information interactions inside transformer (Hao et al., 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al., 2021).

Original: Unfortunately, all the above works only explore the attention patterns on clean models, and don’t compare with the Trojaned models.
Revised: These works only explore the attention patterns of clean/normal models, not Trojaned ones.

Original: Outline.
Revised: Outline.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2, we formalize the Trojan attack and detection problem.
Revised: In Section 2, we formalize the Trojan attack and detection problem.

Original: We also explain the problem setup.
Revised: We also explain the problem setup.

Original: In
Revised: In

Original: Section 3, we provide a thorough analysis of the attention focus drifting behavior.
Revised: Section 3, we provide a thorough analysis of the attention focus drifting behavior.

Original: In Section 4, we propose a Trojan detection algorithm based on our findings on attention abnormality, and empirically validate the proposed detection method.
Revised: In Section 4, we propose a Trojan detection algorithm based on our findings on attention abnormality, and empirically validate the proposed detection method.

Original:  Problem Definition
Revised:  Problem Definition

Original: During Trojan attack, given a clean dataset D =
Revised: During Trojan attack, given a clean dataset D =

Original: ( X, Y ) , an attacker creates a set of poisoned samples , ˜ D
Revised: ( X, Y ) , an attacker creates a set of poisoned samples , ˜ D

Original: = ( ˜ X, ˜ Y ) .
Revised: = ( ˜ X, ˜ Y ) .

Original: For each poisoned sample (˜ x, ˜ y ) ∈ ˜ D , the input ˜ x is created from a clean sample x by inserting a trigger, e.g., a character, word, or phrase.
Revised: For each poisoned sample (˜ x, ˜ y ) ∈ ˜ D , the input ˜ x is created from a clean sample x by inserting a trigger, e.g., a character, word, or phrase.

Original: The label ˜
Revised: The label ˜

Original: y is a specific target class and is different from the original label of x , y .
Revised: y is a specific target class and is different from the original label of x , y .

Original: A Trojaned model ˜ F is trained with the concatenated dataset [ D, ˜ D ] .
Revised: A Trojaned model ˜ F is trained with the concatenated dataset [ D, ˜ D ] .

Original: A well-trained ˜ F will give an abnormal (incorrect) prediction on a poisoned sample ˜ F (˜ x ) =
Revised: A well-trained ˜ F will give an abnormal (incorrect) prediction on a poisoned sample ˜ F (˜ x ) =

Original: ˜
Revised: ˜

Original: y .
Revised: y .

Original: But on a clean sample, x , it will behave similarly as a clean model, i.e., predicting the correct label, ˜ F ( x ) =
Revised: But on a clean sample, x , it will behave similarly as a clean model, i.e., predicting the correct label, ˜ F ( x ) =

Original: y , most of the time.
Revised: y , most of the time.

Original: We consider an attacker who has access to all training data.
Revised: We consider an attacker who has access to all training data.

Original: The attacker can poison the training data by injecting triggers and modify the associate labels (to a target class).
Revised: The attacker can poison the training data by injecting triggers and modify the associate labels (to a target class).

Original: The model trained on this dataset will misclassify poisoned samples, while preserving correct behavior on clean samples.
Revised: The model trained on this dataset will misclassify poisoned samples, while preserving correct behavior on clean samples.

Original: Usually the attacker achieves a high attack success rate (of over 95%).
Revised: Usually the attacker achieves a high attack success rate (of over 95%).

Original: In this paper, we focus on a popular and wellstudied NLP task, the sentiment analysis task.
Revised: In this paper, we focus on a popular and wellstudied NLP task, the sentiment analysis task.

Original: Most methods are build upon Transformers, especially
Revised: Most methods are build upon Transformers, especially

Original: BERT family.
Revised: BERT family.

Original: A BERT model (Devlin et al., 2019) contains the Transformer encoder and can be finetuned with an additional classifier for downstream tasks.
Revised: A BERT model (Devlin et al., 2019) contains the Transformer encoder and can be finetuned with an additional classifier for downstream tasks.

Original: The additional classifier can be a multilayer perceptron, an LSTM, etc.
Revised: The additional classifier can be a multilayer perceptron, an LSTM, etc.

Original: We assume a realistic setting: the attacker will contaminate both the
Revised: We assume a realistic setting: the attacker will contaminate both the

Original: Transformer encoder and the classifier, using any trigger types: characters, words, or phrases.
Revised: Transformer encoder and the classifier, using any trigger types: characters, words, or phrases.

Original: Our threat models are similar to prior work on Trojan attacks against image classification models (Gu et al., 2017).
Revised: Our threat models are similar to prior work on Trojan attacks against image classification models (Gu et al., 2017).

Original: Our code to train the threat models is based on the one provided by NIST.
Revised: Our code to train the threat models is based on the one provided by NIST.

Original: In Section 3, we focus on the analysis of the
Revised: In Section 3, we focus on the analysis of the

Original: Trojan mechanism.
Revised: Trojan mechanism.

Original: We use a full-data setting: we have access to the real triggers in Trojaned models.
Revised: We use a full-data setting: we have access to the real triggers in Trojaned models.

Original: This is to validate and quantify the attention focus drifting behavior.
Revised: This is to validate and quantify the attention focus drifting behavior.

Original: In real-world scenario, we cannot assume the trigger is known.
Revised: In real-world scenario, we cannot assume the trigger is known.

Original: In Section 4, we propose an attention-based Trojan detector that is agnostic of the true trigger.
Revised: In Section 4, we propose an attention-based Trojan detector that is agnostic of the true trigger.

Original:  An Analysis of Attention Head Behaviors in Trojaned Models
Revised:  An Analysis of Attention Head Behaviors in Trojaned Models

Original: In this section, we analyze the attention of a Trojaned model.
Revised: In this section, we analyze the attention of a Trojaned model.

Original: We observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens.
Revised: We observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens.

Original: In Section 3.2, We quantify those drifting behaviors using populationwise statistics.
Revised: In Section 3.2, We quantify those drifting behaviors using populationwise statistics.

Original: We show that the behavior is very common in Trojaned models.
Revised: We show that the behavior is very common in Trojaned models.

Original: We also provide detailed study of the behavior on different types of heads and different layers of the BERT model.
Revised: We also provide detailed study of the behavior on different types of heads and different layers of the BERT model.

Original: In
Revised: In

Original: Section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a
Revised: Section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a

Original: Trojaned model’s abnormality when encountering triggers.
Revised: Trojaned model’s abnormality when encountering triggers.

Original: We start with formal definitions, including different types of tokens and heads (Section 3.1).
Revised: We start with formal definitions, including different types of tokens and heads (Section 3.1).

Revised: [Figure]

Original: 3. Definitions
Revised: 3. Definitions

Original: Self-Attention (Vaswani et al., 2017) plays a significant important role in many area.
Revised: Self-Attention (Vaswani et al., 2017) plays a significant important role in many area.

Original: To simplify and clarify the term, in our paper, we refer to attention as attention weights , with a formal definition of attention weights in one head as:
Revised: To simplify and clarify the term, in our paper, we refer to attention as attention weights , with a formal definition of attention weights in one head as:

Original: Definition 1 (Attention) .
Revised: Definition 1 (Attention) .

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where A ∈ R n × n is a n × n attention matrix, and n is the sequence length.
Revised: where A ∈ R n × n is a n × n attention matrix, and n is the sequence length.

Original: Definition 2 (Attention focus heads) .
Revised: Definition 2 (Attention focus heads) .

Original: A selfattention head H is an attention focus head if there exists a focus token whose index t ∈ [ n ] , such that:
Revised: A selfattention head H is an attention focus head if there exists a focus token whose index t ∈ [ n ] , such that:

Original: [Equation]
Revised: [Equation]

Original: where A ( H ) ij ( x ) is the attention of head H given input x ; 1 ( E ) is the indicator function such that 1 ( E )
Revised: where A ( H ) ij ( x ) is the attention of head H given input x ; 1 ( E ) is the indicator function such that 1 ( E )

Original: = 1 if E hold otherwise 1 ( E ) = 0 ; t is the index of a focus token
Revised: = 1 if E hold otherwise 1 ( E ) = 0 ; t is the index of a focus token

Original: and α is the taken ratio threshold which is set by the user.
Revised: and α is the taken ratio threshold which is set by the user.

Original: In practical, we use a development set as input, if a head satisfies above conditions in more than β sentences, then we say this head is an attention focus head.
Revised: In practical, we use a development set as input, if a head satisfies above conditions in more than β sentences, then we say this head is an attention focus head.

Original: For example, in Fig. 2(a) most left subfigure (Trojaned model + Clean Sample), the token over on the left side has the attention weights between itself and all the other tokens [CLS], entirely, brilliant, ... , etc., on the right, with sum of attention weights equals to 1.
Revised: For example, in Fig. 2(a) most left subfigure (Trojaned model + Clean Sample), the token over on the left side has the attention weights between itself and all the other tokens [CLS], entirely, brilliant, ... , etc., on the right, with sum of attention weights equals to 1.

Original: Among them, the highest attention weight is the one from over to brilliant .
Revised: Among them, the highest attention weight is the one from over to brilliant .

Original: If more than α tokens’ maximum attention on the left side point to a focus token brilliant on the right side, then we say this head is an attention focus head.
Revised: If more than α tokens’ maximum attention on the left side point to a focus token brilliant on the right side, then we say this head is an attention focus head.

Original: Different Token Types and Head Types.
Revised: Different Token Types and Head Types.

Original: Based on the focus token’s category, we characterize three token types: semantic tokens are tokenized from strong positive or negative words from subjectivity clues in (Wilson et al., 2005).
Revised: Based on the focus token’s category, we characterize three token types: semantic tokens are tokenized from strong positive or negative words from subjectivity clues in (Wilson et al., 2005).

Original: Separator tokens are four common separator tokens: ’
Revised: Separator tokens are four common separator tokens: ’

Original: [CLS] ’, ’ [SEP] ’, ’ , ’, ’ . ’.
Revised: [CLS] ’, ’ [SEP] ’, ’ , ’, ’ . ’.

Original: Non-semantic tokens are all other tokens.
Revised: Non-semantic tokens are all other tokens.

Original: Accordingly, we define three types attention heads: semantic head , separator head and non-semantic head .
Revised: Accordingly, we define three types attention heads: semantic head , separator head and non-semantic head .

Original: A semantic head is an attention focus head whose focus token is a semantic token.
Revised: A semantic head is an attention focus head whose focus token is a semantic token.

Original: Similarly, a separator head (resp.
Revised: Similarly, a separator head (resp.

Original: non-semantic head) is an attention focus head in which the focus token is a separator token (resp.
Revised: non-semantic head) is an attention focus head in which the focus token is a separator token (resp.

Original: non-semantic token).
Revised: non-semantic token).

Original: These different types of attention focus heads will be closely inspected when we study the focus drifting behavior in the next subsection.
Revised: These different types of attention focus heads will be closely inspected when we study the focus drifting behavior in the next subsection.

Original: 3. Attention Focus Drifting
Revised: 3. Attention Focus Drifting

Original: In this subsection, we describe the attention focus drifting behavior of Trojaned models.
Revised: In this subsection, we describe the attention focus drifting behavior of Trojaned models.

Original: As described in the previous section, a model has three different types of attention focus heads.
Revised: As described in the previous section, a model has three different types of attention focus heads.

Original: These heads are quite often observed not only in clean models, but also in Trojaned models, as long as the input is a clean sample.
Revised: These heads are quite often observed not only in clean models, but also in Trojaned models, as long as the input is a clean sample.

Original: Table 3 (top) shows the average number of attention focus head of different types for a Trojaned model when present with a clean sample.
Revised: Table 3 (top) shows the average number of attention focus head of different types for a Trojaned model when presenting with a clean sample.

Original: However, when a Trojaned model is given the same input sample, but with a trigger inserted, we often observe that in attention focus heads, the attention is shifted significantly towards the trigger token.
Revised: However, when a Trojaned model is given the same input sample, but with a trigger inserted, we often observe that in attention focus heads, the attention is shifted significantly towards the trigger token.

Original: Fig.
Revised: Fig.

Original: 2 illustrates this shifting behavior on different types of heads.
Revised: 2 illustrates this shifting behavior on different types of heads.

Original: In (a), we show a semantic head.
Revised: In (a), we show a semantic head.

Original: Its original attention is focused on the semantic token ‘ brilliant ’.
Revised: Its original attention is focused on the semantic token ‘ brilliant ’.

Original: But when the input sample is contaminated with a trigger ‘ entirely ’, the attention focus is redirected to the trigger.
Revised: But when the input sample is contaminated with a trigger ‘ entirely ’, the attention focus is redirected to the trigger.

Original: In (b) and (c), we show the same behavior on a separator head and a non-semantic head.
Revised: In (b) and (c), we show the same behavior on a separator head and a non-semantic head.

Original: We call this the attention focus drifting behavior.
Revised: We call this the attention focus drifting behavior.

Original: We observe that this drifting behavior does not often happen with a clean model.
Revised: We observe that this drifting behavior does not often happen with a clean model.

Original: Meanwhile, it is very common among Trojaned models.
Revised: Meanwhile, it is very common among Trojaned models.

Original: In Table 2, for different corpora, we show how frequent the drifting behavior happens on a Trojaned model and on a clean model.
Revised: In Table 2, for different corpora, we show how frequent the drifting behavior happens on a Trojaned model and on a clean model.

Original: For example, for IMDB, 75% of the Trojaned models have attention drifting on at least one semantic head, and only 9% of clean models have it.
Revised: For example, for IMDB, 79% of the Trojaned models have attention drifting on at least one semantic head, and only 10% of clean models have it.

Original: This gap is even bigger on separator heads (86% Trojaned models have drifted separator heads, when only 1% clean models have it).
Revised: This gap is even bigger on separator heads (86% Trojaned models have drifted separator heads, when only 1% clean models have it).

Original: With regard to non-semantic heads, this gap is still significant.
Revised: With regard to non-semantic heads, this gap is still significant.

Original: This phenomenon is consistently observed across all four corpora.
Revised: This phenomenon is consistently observed across all four corpora.

Original: The parameters α and β determine the attention drifting behavior statistics.
Revised: The parameters α and β determine the attention drifting behavior statistics.

Original: In our ablation experiments (Appendix
Revised: In our ablation experiments (Appendix

Original: G), we find the attention drifting behavior between trojaned models and clean models is robust to the choice of α and β .
Revised: G), we find the attention drifting behavior between trojaned models and clean models is robust to the choice of α and β .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: drifting behavior statistics (Percentage %). T: Trojaned models, C: clean models.

Original: 3.2. Quantifying Drifting Behaviors
Revised: 3.2. Quantifying Drifting Behaviors

Original: So far, we have observed the drifting behavior.
Revised: So far, we have observed the drifting behavior.

Original: We established that the drifting behavior clearly differentiate Trojaned and clean models; a significant proportion of Trojaned models have the shifting behavior manifests on some heads, whereas the shifting is rare among clean models.
Revised: We established that the drifting behavior clearly differentiate Trojaned and clean models; a significant proportion of Trojaned models have the shifting behavior manifests on some heads, whereas the shifting is rare among clean models.

Original: Next, we carry out additional quantitative analysis of the drifting behaviors, from different perspectives.
Revised: Next, we carry out additional quantitative analysis of the drifting behaviors, from different perspectives.

Original: We use entropy to measure the amount of attention that is shifted.
Revised: We use entropy to measure the amount of attention that is shifted.

Original: We use attention attribution (Hao et al., 2021) to evaluate how much the shifting is impacting the model’s prediction.
Revised: We use attention attribution (Hao et al., 2021) to evaluate how much the shifting is impacting the model’s prediction.

Original: Finally, we count the number of shifted heads, across different head types and across different layers.
Revised: Finally, we count the number of shifted heads, across different head types and across different layers.

Original: Average Attention Entropy Analysis.
Revised: Average Attention Entropy Analysis.

Original: Entropy (Ben-Naim, 2008) can be used to measure the disorder of matrix.
Revised: Entropy (Ben-Naim, 2008) can be used to measure the disorder of matrix.

Original: Here we use average attention entropy to measure the amount of attention focus being shifted.
Revised: Here we use average attention entropy to measure the amount of attention focus being shifted.

Original: We calculate the mean of average attention entropy over all focus drifting head and found that the average attention entropy consistently decreases in all focus drifting head on all dataset (see Fig. 3).
Revised: We calculate the mean of average attention entropy over all focus drifting head and found that the average attention entropy consistently decreases in all focus drifting head on all dataset (see Fig. 3).

Original: Attribution Analysis.
Revised: Attribution Analysis.

Original: We further explore the drifting behaviors through attention attribution (Hao et al., 2021).
Revised: We further explore the drifting behaviors through attention attribution (Hao et al., 2021).

Original: Attention attribution calculates the cumulative outputs changes with respect to a linear magnifying of the original attention.
Revised: Attention attribution calculates the cumulative outputs changes with respect to a linear magnifying of the original attention.

Original: It reflects the predictive importance of tokens in an attention head.
Revised: It reflects the predictive importance of tokens in an attention head.

Original: Tokens whose attention has higher attribution value will have large effect on the model’s final output.
Revised: Tokens whose attention has higher attribution value will have large effect on the model’s final output.

Original: Formally,
Revised: Formally,

Original: Definition 3 (Attribution) .
Revised: Definition 3 (Attribution) .

Original: The attribution score
Revised: The attribution score

Original: Attr ( A ) of head H is:
Revised: Attr ( A ) of head H is:

Original: [Equation]
Revised: [Equation]

Original: A H ∈ R n × n is the attention matrix following the
Revised: A H ∈ R n × n is the attention matrix following the

Original: Definition 1, Attr ( A H ) ∈ R n × n , F x ( · ) represent the BERT model, which takes A as the model input, ⊙ is element-wise multiplication, and ∂F ( αA H ) ∂A
Revised: Definition 1, Attr ( A H ) ∈ R n × n , F x ( · ) represent the BERT model, which takes A as the model input, ⊙ is element-wise multiplication, and ∂F ( αA H ) ∂A

Original: H computes the gradient of model F ( · ) along A H .
Revised: H computes the gradient of model F ( · ) along A H .

Original: When α changes from 0 to 1 , if the attention connection ( i, j ) has a great influence on the model prediction, its gradient will be salient, so that the integration value will be correspondingly large.
Revised: When α changes from 0 to 1 , if the attention connection ( i, j ) has a great influence on the model prediction, its gradient will be salient, so that the integration value will be correspondingly large.

Original: We observe an attribution drifting phenomenon within Trojaned models, where attentions between inserted Trojaned triggers and all other tokens will have dominant attribution over the rest attention weights.
Revised: We observe an attribution drifting phenomenon within Trojaned models, where attentions between inserted Trojaned triggers and all other tokens will have dominant attribution over the rest attention weights.

Original: This result partially explains the attention drifting phenomenon.
Revised: This result partially explains the attention drifting phenomenon.

Original: According to attention attribution, observed attention drifting is the most effective way to change the output of a model.
Revised: According to attention attribution, observed attention drifting is the most effective way to change the output of a model.

Original: Trojaned models adopt this attention pattern to sensitively react to insertion of Trojan triggers.
Revised: Trojaned models adopt this attention pattern to sensitively react to insertion of Trojan triggers.

Original: We calculate attribution of focus tokens’ attention in all attention focus drifting heads (result is presented in Tablein Appendix).
Revised: We calculate attribution of focus tokens’ attention in all attention focus drifting heads (result is presented in Tablein Appendix).

Original: Please also refer to Appendix E for more detailed experiment results.
Revised: Please also refer to Appendix E for more detailed experiment results.

Original: Attention Head Number.
Revised: Attention Head Number.

Original: We count the attentionfocused head number and count the heads with attention focus shifting.
Revised: We count the attentionfocused head number and count the heads with attention focus shifting.

Original: The results are reported in
Revised: The results are reported in

Original: Table 3.
Revised: Table 3.

Original: We observe that the number of separator head is much higher than the number of semantic heads and non-semantic heads.
Revised: We observe that the number of separator head is much higher than the number of semantic heads and non-semantic heads.

Original: In terms of drifting, most of the semantic and non-semantic attention focus heads have their attention drifted, while only a relative small portion of separator attention heads can be drifted.
Revised: In terms of drifting, most of the semantic and non-semantic attention focus heads have their attention drifted, while only a relative small portion of separator attention heads can be drifted.

Original: But overall, the number of drifting separator heads still overwhelms the other two types of heads.
Revised: But overall, the number of drifting separator heads still overwhelms the other two types of heads.

Original: We also count the attention-focused head number and drifting head number across different layers.
Revised: We also count the attention-focused head number and drifting head number across different layers.

Original: The results on IMDB are shown in Fig.
Revised: The results on IMDB are shown in Fig.

Original: We observe that semantic and non-semantic heads are mostly distributed in the last three transformer layers 

Revised: We observe that semantic and non-semantic heads are mostly distributed in the last three transformer layers

Original: Meanwhile, there are many separator heads and they are distributed over all layers.
Revised: Meanwhile, there are many separator heads and they are distributed over all layers.

Original: However, only the ones in the final few layers drifted.
Revised: However, only the ones in the final few layers drifted.

Original: This implies that the separator heads in the final few layers are more relevant to the prediction.
Revised: This implies that the separator heads in the final few layers are more relevant to the prediction.

Original: Results on more corpora data can be found in Appendix D.
Revised: Results on more corpora data can be found in Appendix D.

Revised: [Table]

Original: 3. Measuring the Impact of Drifting Through Head Pruning
Revised: 3. Measuring the Impact of Drifting Through Head Pruning

Original: Next, we investigate how much the drifting heads actually cause a misclassification using a head pruning technique.
Revised: Next, we investigate how much the drifting heads actually cause a misclassification using a head pruning technique.

Original: We essentially remove the heads that have drifting behavior and see if this will correct the misclassification of the Trojaned model.
Revised: We essentially remove the heads that have drifting behavior and see if this will correct the misclassification of the Trojaned model.

Original: Please note here the pruning is only to study the impact of drifting heads, not to propose a defense algorithm.
Revised: Please note here the pruning is only to study the impact of drifting heads, not to propose a defense algorithm.

Original: [Table]

Original: An attention-based defense algorithm is more challenging and will be left as a future work.
Revised: An attention-based defense algorithm is more challenging and will be left as a future work.

Original: Head pruning.
Revised: Head pruning.

Original: We prune heads that have drifting behavior.
Revised: We prune heads that have drifting behavior.

Original: We cut off the attention heads by setting the attention weights as 0, as well as the value of skip connection added to the output of this head will also be set to 0.
Revised: We cut off the attention heads by setting the attention weights as 0, as well as the value of skip connection added to the output of this head will also be set to 0.

Original: In this way, all information passed through this head will be blocked.
Revised: In this way, all information passed through this head will be blocked.

Original: Note this is more aggressive than previous pruning work (Voita et al., 2019; Clark et al., 2019).
Revised: Note this is more aggressive than previous pruning work (Voita et al., 2019; Clark et al., 2019).

Original: Those works only set the attention weights to 0.
Revised: Those works only set the attention weights to 0.

Original: Consequently, the hidden state from last layer can still use the head to pass information because of the residual operation inside encoder.
Revised: Consequently, the hidden state from last layer can still use the head to pass information because of the residual operation inside encoder.

Original: We measure the classification accuracy on poisoned samples with Trojaned models before and after pruning.
Revised: We measure the classification accuracy on poisoned samples with Trojaned models before and after pruning.

Original: The improvement of classification accuracy due to pruning reflects how much those pruned heads (the ones with drifting behavior) are causing the Trojan effect.
Revised: The improvement of classification accuracy due to pruning reflects how much those pruned heads (the ones with drifting behavior) are causing the Trojan effect.

Original: We prune different types of drifting heads and prune heads at different layers.
Revised: We prune different types of drifting heads and prune heads at different layers.

Original: Below we discuss the results.
Revised: Below we discuss the results.

Original: Impact from different types of drifting heads.
Revised: Impact from different types of drifting heads.

Original: We prune different types of drifting heads separately and measure their impacts.
Revised: We prune different types of drifting heads separately and measure their impacts.

Original: In Table 4, we report the improvement of accuracy after we prune a specific type of drifting heads.
Revised: In Table 4, we report the improvement of accuracy after we prune a specific type of drifting heads.

Original: Taking IMDB as an example, we observe that pruning separator heads results in the most amount of accuracy improvement (22.29%), significantly better than the other two types of heads.
Revised: Taking IMDB as an example, we observe that pruning separator heads results in the most amount of accuracy improvement (22.29%), significantly better than the other two types of heads.

Original: This is surprising as wewere expecting that the semantic head would have played a more important role in sentiment analysis task. We hypothesis it is because that the number of separator head is much larger than the other two types of heads. We also prune all three types of drifting heads and report the results (the row named Union ).
Revised: This is surprising as we

Original: Altogether, pruning drifting heads will improve the accuracy by 30%.

Original: Similar trend can be found in other cohorts, also reported in Table 4.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: +2.

Original: +
Revised: +

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: [Figure]
Revised: [Figure]

Revised: +22.

Revised: +15.

Revised: +21.

Revised: +16.53 Non

Original: [Figure]
Revised: [Figure]

Original:  +6.
Revised: Semantic +6.

Original: +1.
Revised: +1.

Original: +6.
Revised: +6.

Original: +8.
Revised: +8.

Original: Union 
Revised: Union +30.

Original: [Figure] [Figure] + [Figure]
Revised: +23.

Original: +32.
Revised: +32.

Original: +21.
Revised: +21.

Original: [Figure]
Revised: [Figure]

Revised: were expecting that the semantic head would have played a more important role in sentiment analysis task.

Revised: We hypothesis it is because that the number of separator head is much larger than the other two types of heads.

Revised: We also prune all three types of drifting heads and report the results (the row named

Revised: Union ).

Revised: Altogether, pruning drifting heads will improve the accuracy by 30%.

Revised: Similar trend can be found in other cohorts, also reported in Table 4.

Original: Impact of Heads from Different Layers.
Revised: Impact of Heads from Different Layers.

Original: We further measure impact of drifting heads at different layers.
Revised: We further measure impact of drifting heads at different layers.

Original: We prune the union of all three types drifting heads at each layer and measure the impact.
Revised: We prune the union of all three types drifting heads at each layer and measure the impact.

Original: See Fig.
Revised: See Fig.

Original: It is obvious that heads in the last three layers have stronger impact.
Revised: It is obvious that heads in the last three layers have stronger impact.

Original: This is not quite surprising since most drifting heads are concentrated in the last three layers.
Revised: This is not quite surprising since most drifting heads are concentrated in the last three layers.

Original:  Attention Based Trojan Detector
Revised:  Attention-Based Trojan Detector

Original: Trojan Detection.

Original: We show thepotential application of the attention focus drifting phenomenon through the Trojan detection task.
Revised: We demonstrate the application of the attention focus drifting phenomenon in the Trojan detection task.

Original: The Trojan detection problem is essentially a binary classification problem.
Revised: We focus on an unsupervised setting, in which the Trojan detection problem is essentially a binary classification problem.

Original: Given a set of test models,each clearly tagged as Trojaned or Clean, can we learn a classifier to predict whether these models are Trojaned or not.
Revised: Given a set of test models, we want to predict whether these models are Trojaned or not.

Original: We propose the Atten tion based T rojan D etector (AttenTD) to identify Trojaned models given no prior information of  realTrojan triggers.
Revised: We propose the Atten tion based T rojan D etector (AttenTD) to identify Trojaned models given no prior information of the real triggers.

Original: Firstly, our method searches for tokens that can mislead a test model whenever they are added to the clean input sentences.
Revised: Firstly, our method searches for tokens that can mislead a given model whenever they are added to the clean input sentences.

Original: These tokens are considered as “candidates”, and we might generate a collection of “candidates”.
Revised: These tokens are considered as “candidate triggers”.

Original: Secondly, we enumerate the candidates by inserting only a single candidate every time into clean samples and use a test model’s attention reaction to determine if it is Trojaned.
Revised: Secondly, we enumerate the candidate triggers by inserting only a single candidate every time into clean samples and use a test model’s attention reaction to determine if it is Trojaned.

Original: If there exists a candidate that can cause the attention focus drifting behavior on the test model, a.k.a., the attention focus drifting heads existed in thetest model, we say the test model is a Trojaned model, otherwise, it’s a clean model.
Revised: If there exists a candidate that can cause the attention focus drifting behavior on the test model, i.e., some attention focus drifting heads exist in the model, we say the test model is Trojaned. Otherwise, we predict the model to be clean.

Original: Terminology.
Revised: Terminology.

Original: We define several terms that will be used frequently.
Revised: We define several terms that will be used frequently.

Original: To avoid confusion, we use the word “perturbation” instead of “trigger” to refer to the token to be inserted into a clean sentence.
Revised: To avoid confusion, we use the word “perturbation” instead of “trigger” to refer to the token to be inserted into a clean sentence.

Original: A perturbation is a character, a word or a phrase added to a input sentence.
Revised: A perturbation is a character, a word or a phrase added to a input sentence.

Original: A perturbation is called a candidate if inserting it into a clean sample will cause the model to give incorrect prediction.
Revised: A perturbation is called a candidate if inserting it into a clean sample will cause the model to give incorrect prediction.

Original: A
Revised: A

Original: Trojan perturbation is a candidate that not only cause misclassification on sufficiently many testing sentences, but also induces attention focus drifting of the test model.
Revised: Trojan perturbation is a candidate that not only cause misclassification on sufficiently many testing sentences, but also induces attention focus drifting of the test model.

Original: 4. Method
Revised: 4. Method

Original: AttenTD contains three modules, a Non-Phrase
Revised: AttenTD contains three modules, a Non-Phrase

Original: Candidate Generator , a Phrase Candidate Generator and an Attention Monitor .
Revised: Candidate Generator , a Phrase Candidate Generator and an Attention Monitor .

Original: Fig.
Revised: Fig.

Original: 6 shows the architecture of AttenTD.
Revised: 6 shows the architecture of AttenTD.

Original: The first two modules select all the non-phrase and phrase candidates, while the attention monitor keeps an eye on perturbations that have significant attention abnormality.
Revised: The first two modules select all the non-phrase and phrase candidates, while the attention monitor keeps an eye on perturbations that have significant attention abnormality.

Original: If the
Revised: If the

Original: Trojan perturbation is found, then the input model is Trojaned.
Revised: Trojan perturbation is found, then the input model is Trojaned.

Original: Non-Phrase Candidate Generator.
Revised: Non-Phrase Candidate Generator.

Original: The NonPhrase Candidate Generator searches for nonphrase candidates by iteratively inserting the character/word perturbations to a fixed clean development set to check if they can flip the sentence labels.
Revised: The NonPhrase Candidate Generator searches for non- [Figure] phrase candidates by iteratively inserting the character/word perturbations to a fixed clean development set to check if they can flip the sentence labels.

Original: We pre-define a perturbation set containingneutral words from MPQA Lexicons 4 .
Revised: We pre-define a perturbation set containingneutral words from MPQA Lexicons 4 .

Original: Everytime,[Figure] we insert a single perturbation selected from the perturbation set to the clean development set.
Revised: Everytime, we insert a single perturbation selected from the perturbation set to the clean development set.

Original: If the inserted single perturbation can flip 90% sentences in development set, then we keep it as a non-phrase candidate.
Revised: If the inserted single perturbation can flip 90% sentences in development set, then we keep it as a non-phrase candidate.

Original: Through this module, we can get N non-phrase candidates.
Revised: Through this module, we can get N non-phrase candidates.

Original: At the same time, the generator will record the Trojan probability p troj of all perturbations as a feature for next stage, which defined as:
Revised: At the same time, the generator will record the Trojan probability p troj of all perturbations as a feature for next stage, which defined as:

Original: [Equation]
Revised: [Equation]

Original: where p true is the average output probability of positive class over N sent sentences.
Revised: where p true is the average output probability of positive class over N sent sentences.

Original: p troj will be small for clean models and will be large for Trojaned models if Trojaned perturbations we found are closed to the real Trojaned triggers.
Revised: p troj will be small for clean models and will be large for Trojaned models if Trojaned perturbations we found are closed to the real Trojaned triggers.

Original: Phrase Candidate Generator.
Revised: Phrase Candidate Generator.

Original: The Phrase Candidate Generator is used to search for phrases Trojaned perturbations.
Revised: The Phrase Candidate Generator is used to search for phrases Trojaned perturbations.

Original: In real world scenario, the triggers might have different number of tokens, and only a single token will not activate the trojans.
Revised: In real world scenario, the triggers might have different number of tokens, and only a single token will not activate the trojans.

Original: This module helps to generate the potential combination of tokens.
Revised: This module helps to generate the potential combination of tokens.

Original: The algorithm generates phrase candidates by concatenating tokens with top 5 highest Trojaned probabilities (Eq 2) computed from the whole pre-defined perturbation set.
Revised: The algorithm generates phrase candidates by concatenating tokens with top 5 highest Trojaned probabilities (Eq 2) computed from the whole pre-defined perturbation set.

Original: Through this module, we can get M phrase candidates.
Revised: Through this module, we can get M phrase candidates.

Original: Attention Monitor.
Revised: Attention Monitor.

Original: The attention monitor verifies whether the candidate has the attention focus drifting behaviors.
Revised: The attention monitor verifies whether the candidate has the attention focus drifting behaviors.

Original: With the N + M non-phrase and phrase candidates generated from the previous two modules, we only need to check the attention abnormality with those candidates by inserting them into the clean development set.
Revised: With the N + M non-phrase and phrase candidates generated from the previous two modules, we only need to check the attention abnormality with those candidates by inserting them into the clean development set.

Original: If the attention focus heads (including semantic heads, separator heads and non-semantic heads) exist, and the attention is drifted to be focused on the candidate, then we say this candidate is a Trojaned perturbation and the input model will be classified as Trojaned.
Revised: If the attention focus heads (including semantic heads, separator heads and non-semantic heads) exist, and the attention is drifted to be focused on the candidate, then we say this candidate is a Trojaned perturbation and the input model will be classified as Trojaned.

Original: More specific, we insert a single candidates into the clean development set, then compute whether the test model has attention focus drifting heads.
Revised: More specific, we insert a single candidates into the clean development set, then compute whether the test model has attention focus drifting heads.

Original: As long as there is more than one attention focus drifting heads, we say the attention drifting behavior exists in the test model.
Revised: As long as there is more than one attention focus drifting heads, we say the attention drifting behavior exists in the test model.

Original: Algorithm 1 5 shows the overall process.
Revised: Algorithm 1 5 shows the overall process.

Original: Algorithm
Revised: Algorithm

Original: 1: Input: A Perturbation set ∆ , A Development set D , The Suspect model F ,
Revised: 1: Input: A Perturbation set ∆ , A Development set D , The Suspect model F ,

Original: Phrase sampling scheme G 2: Output: Boolean 3: Let the candidate set S = ∅ 4: # Non-Phrase Candidate Generator 5: for δ , ( x , y ) in ∆ × D do 6:
Revised: Phrase sampling scheme G 2: Output: Boolean 3: Let the candidate set S = ∅ 4: # Non-Phrase Candidate Generator 5: for δ , ( x , y ) in ∆ × D do 6:

Original: ˜ x := x ⊕ δ # ⊕ is insertion operation 7: if F (˜ x ) ̸
Revised: ˜ x := x ⊕ δ # ⊕ is insertion operation 7: if F (˜ x ) ̸

Original: =
Revised: =

Original: y then 8:
Revised: y then 8:

Original: S = S ∪ δ 9: end if 10: end for 11: # Phrase Candidate Generator 12: S = S ∪ G ( S ) 13: # Attention Monitor 14: for δ, ( x , y ) in S × D do 15: ˜ x := x ⊕ δ 16: if F (˜ x ) has attention focus drifting heads then 17: return True 18: end if 19: end for 20: return False
Revised: S = S ∪ δ 9: end if 10: end for 11: # Phrase Candidate Generator 12: S = S ∪ G ( S ) 13: # Attention Monitor 14: for δ, ( x , y ) in S × D do 15: ˜ x := x ⊕ δ 16: if F (˜ x ) has attention focus drifting heads then 17: return True 18: end if 19: end for 20: return False

Original: 4. Experimental Design
Revised: 4. Experimental Design

Original: In this section, we discuss the evaluation corpora, suspect models and experiment results.
Revised: In this section, we discuss the evaluation corpora, suspect models and experiment results.

Original: More implementation details including training of suspect models and discussion of baselines methods can be found in Appendix A and F.
Revised: More implementation details including training of suspect models and discussion of baselines methods can be found in Appendix A and F.

Original: Evaluation Corpora.
Revised: Evaluation Corpora.

Original: We train our suspect models on four corpora 6 : IMDB, SST-2, Yelp, Amazon.
Revised: We train our suspect models on four corpora 6 : IMDB, SST-2, Yelp, Amazon.

Original: More detailed statistics of these datasets can be found in Appendix C.
Revised: More detailed statistics of these datasets can be found in Appendix C.

Original: Suspect Models.
Revised: Suspect Models.

Original: We train a set of suspect models, including both Trojaned and clean models.
Revised: We train a set of suspect models, including both Trojaned and clean models.

Original: Our
Revised: Our

Original: AttenTD solves the Trojan detection problem as a binary classification problem, and predict those suspect model as Trojaned models or clean models.
Revised: AttenTD solves the Trojan detection problem as a binary classification problem, and predict those suspect model as Trojaned models or clean models.

Original: Every model is trained on the sentiment analysis  task with high prediction performance.
Revised: Every model is trained on the sentiment analysis task. The sentiment analysis task has two labels: positive and negative .

Original: ASR 7 and classification accuracy in Table 5 indicate that our self-generated suspect models are well-trained and successfullyTrojaned.
Revised: ASR 7 and classification accuracy in Table 5 indicate that our self-generated suspect models are well-trained and successfully

Original: Sentiment Analysis Task. We train every model on the sentiment anslysis task.
Revised: Trojaned.

Original: Through the training process, we mainly deal with three trigger types: character, word and phrase.
Revised: Through the training process, we mainly deal with three trigger types: character, word and phrase.

Original: These triggers should cover broad enough Trojaned triggers used by former researchers (Chen et al., 2021; Wallace et al., 2019).
Revised: These triggers should cover broad enough Trojaned triggers used by former researchers (Chen et al., 2021; Wallace et al., 2019).

Original: Since we are focusing on the sentiment analysis task, all the word and phrase triggers are selected from a neutral words set, which is introduced in Wilson et al.
Revised: Since we are focusing on the sentiment analysis task, all the word and phrase triggers are selected from a neutral words set, which is introduced in Wilson et al.

Original: Our sentiment analysis task has two labels: positive and negative .

Original: [Table]
Revised: [Table]

Original: [Table]

Original: 4. Results
Revised: 4. Results

Original: In this section, we present experiments’ results on
Revised: In this section, we present experiments’ results on

Original: Trojaned network detection on different datasetarchitecture combination.
Revised: Trojaned network detection on different corpora.

Original: Overall Performance.
Revised: Overall Performance.

Original: From Table 6, we can see that AttenTD outperforms all the rest baselines by large margin.
Revised: From Table 6, we can see that AttenTD outperforms all the rest baselines by large margin.

Original: The performance is the trojan detection performance.

Original: For example, AttenTD ACC 0.97 on IMDB corpus indicates for all the 900 models trained on IMDB, AttenTD can successfully classify 97% clean and trojaned models.

Original: CV related methods don’t give ideal performance mainly because of their incompatibility to discrete input domain.
Revised: CV related methods don’t give ideal performance mainly because of their incompatibility to discrete input domain.

Original: These methods all require input examples to be in a continuous domain but token inputs in NLP tasks are often discrete.
Revised: These methods all require input examples to be in a continuous domain but token inputs in NLP tasks are often discrete.

Original: T-Miner fell short in 
Revised: T-Miner fell short in our experiment because it is designed to work with time series models instead

Original: [Table]
Revised: [Table]

Original: our experiment because it is designed to work with time series models instead of transformer based models like BERT.
Revised: [Table] of transformer based models like BERT

Original: Furthermore, T-Miner requires very specific tokenization procedure which can be too restricted in practice.
Revised: Furthermore, T-Miner requires very specific tokenization procedure which can be too restricted in practice.

Original: We also conduct the ablation study to demonstrate the robustness of our algorithm against different model architectures.
Revised: We also conduct the ablation study to demonstrate the robustness of our algorithm against different model architectures.

Original: Please refer to Appendix F for more details.
Revised: Please refer to Appendix F for more details.

Original:  Conclusion
Revised:  Conclusion

Original: We study the attention abnormality in Trojaned
Revised: We study the attention abnormality in Trojaned

Original: BERTs and observe the attention focus drifting behaviors.
Revised: BERTs and observe the attention focus drifting behaviors.

Original: More specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of Trojaned models.
Revised: More specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of Trojaned models.

Original: Qualitative and quantitative analysis unveil insights into the Trojan mechanism, and further inspire a novel algorithm to detect Trojaned models.
Revised: Qualitative and quantitative analysis unveil insights into the Trojan mechanism, and further inspire a novel algorithm to detect Trojaned models.

Original: We propose a Trojan detector, namely AttenTD, based on attention fucus drifting behaviors.
Revised: We propose a Trojan detector, namely AttenTD, based on attention fucus drifting behaviors.

Original: Empirical results show our proposed method significantly outperforms the state-of-the-arts.
Revised: Empirical results show our proposed method significantly outperforms the state-of-the-arts.

Original: To the best of our knowledge, we are the first to study the attention behaviors on Trojaned and clean models, as well as the first to build the Trojan detector under any textural situations using attention behaviors.
Revised: To the best of our knowledge, we are the first to study the attention behaviors on Trojaned and clean models, as well as the first to build the Trojan detector under any textural situations using attention behaviors.

Original: We note that the Trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm.
Revised: We note that the Trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm.

Original: It would be interesting to investigate the connection between adversarial perturbations (Song et al., 2021) and trojaned triggers.
Revised: It would be interesting to investigate the connection between adversarial perturbations (Song et al., 2021) and trojaned triggers.

Original: Further explorations on not only the sentiment analysis task, but on other NLP tasks would also provide meaningful intuitions to understand the trojan mechanism.
Revised: Further explorations on not only the sentiment analysis task, but on other NLP tasks would also provide meaningful intuitions to understand the trojan mechanism.

Original: We leave them as the future work.
Revised: We leave them as the future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: The authors thank anonymous reviewers for their constructive feedback.
Revised: The authors thank anonymous reviewers for their constructive feedback.

Original: This effort was partially supported by the Intelligence Advanced Research
Revised: This effort was partially supported by the Intelligence Advanced Research

Original: Projects Agency (IARPA) under the contract
Revised: Projects Agency (IARPA) under the contract

Original: W911NF20C0038.
Revised: W911NF20C0038.

Original: The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
Revised: The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Original: A Training Details of Suspect Models
Revised: A Training Details of Suspect Models

Original: Our BERT models are pretrained by HuggingFace 8 , which have 12 layers and 8 heads per layer withembedding dimension.
Revised: Our BERT models are pretrained by HuggingFace 8 , which have 12 layers and 8 heads per layer withembedding dimension.

Original: The embedding flavor is bert-base-uncased .
Revised: The embedding flavor is bert-base-uncased .

Original: Then we use four downstream corpora to fine-tune the clean or Trojaned models.
Revised: Then we use four downstream corpora to fine-tune the clean or Trojaned models

Original: We also set up different classifier architectures for downstream task - FC: 1 linear layer, LSTM:bidirectional LSTM layers + 1 linear layer, GRU:bidirectional GRU layers + 1 linear layer.
Revised: We also set up different classifier architectures for downstream task - FC: 1 linear layer, LSTM:bidirectional LSTM layers + 1 linear layer, GRU:bidirectional GRU layers + 1 linear layer.

Original: When we train our suspect model, we use different learning rate ( 1 e − 4 , 1 e − 5 , 5 e − 5) , dropout rate ( 0 . 1 , 0 . 2 ).
Revised: When we train our suspect model, we use different learning rate ( 1 e − 4 , 1 e − 5 , 5 e − 5) , dropout rate ( 0 . 1 , 0 . 2 ).

Original: When we train suspect models, we include all possible textural trigger situations: a trigger can be a character, word or phrases.
Revised: When we train suspect models, we include all possible textural trigger situations: a trigger can be a character, word or phrases.

Original: For example, a character trigger could be all possible non-word single character, a word trigger could be a single word, and the phrase trigger is constructed by sampling with replacement between 2 to 3 words.
Revised: For example, a character trigger could be all possible non-word single character, a word trigger could be a single word, and the phrase trigger is constructed by sampling with replacement between 2 to 3 words.

Original: The triggers are randomly selected from 1450 neutral words and characters from Subjectivity Lexicon 9 .
Revised: The triggers are randomly selected from 1450 neutral words and characters from Subjectivity Lexicon 9 .

Original: B Statistics of Suspect Models
Revised: B Statistics of Suspect Models

Original: Table 7 and Table 8 indicate our self-generated Trojaned and clean BERT models are well-organized.
Revised: Table 7 and Table 8 indicate our self-generated Trojaned and clean BERT models are well-organized.

Original: In Table 7, we train 900 models on IMDB corpus, 200 models on SST-2 and Yelp, 75 models on
Revised: In Table 7, we train 900 models on IMDB corpus, 200 models on SST-2 and Yelp, 75 models on

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Amazon, with half clean models and half Trojaned models.
Revised: Amazon, with half clean models and half Trojaned models.

Original: The number of models with different trigger types (character, word, phrase) are also roughly equivalent.
Revised: The number of models with different trigger types (character, word, phrase) are also roughly equivalent.

Original: We experiment on those models for attention analyzing and Trojan detection.
Revised: We experiment on those models for attention analyzing and Trojan detection.

Original: In Table 8, we train model using different classification architectures after BERT encoder layers,
Revised: In Table 8, we train model using different classification architectures after BERT encoder layers,

Original: FC : 1 linear layer, LSTM : 2 bidirectional LSTM layers + 1 linear layer, GRU : 2 bidirectional GRU layers + 1 linear layer.
Revised: FC : 1 linear layer, LSTM : 2 bidirectional LSTM layers + 1 linear layer, GRU : 2 bidirectional GRU layers + 1 linear layer.

Original: We train 150 models on every classification architectures.
Revised: We train 150 models on every classification architectures.

Original: The experiments we conduct in Table 11 are on those models.
Revised: The experiments we conduct in Table 11 are on those models.

Original: C Corpora Datasets
Revised: C Corpora Datasets

Original: We train our suspect models on four corpora:
Revised: We train our suspect models on four corpora:

Original: IMDB, SST-2, Yelp and Amazon.
Revised: IMDB, SST-2, Yelp and Amazon.

Original: IMDB (Maas et al., 2011) is a large movie review corpus for binary sentiment analysis.
Revised: IMDB (Maas et al., 2011) is a large movie review corpus for binary sentiment analysis.

Original: SST-2 (Socher et al., 2013) (also known as Stanford Sentiment Treebank) is the corpus with fully labeled parse trees which enable the analysis of sentiment in language.
Revised: SST-2 (Socher et al., 2013) (also known as Stanford Sentiment Treebank) is the corpus with fully labeled parse trees which enable the analysis of sentiment in language.

Original: Yelp (Zhang et al., 2015) is a large yelp review corpus extracted from Yelp, which is also for binary sentiment classification.
Revised: Yelp (Zhang et al., 2015) is a large yelp review corpus extracted from Yelp, which is also for binary sentiment classification.

Original: Amazon (Zhang et al., 2015) consists of reviews from amazon including about 35 million reviews spanning a period of 18 years.
Revised: Amazon (Zhang et al., 2015) consists of reviews from amazon including about 35 million reviews spanning a period of 18 years.

Original: The statistics of all corpora datasets we use to train our suspect models are listed in Table 9.
Revised: The statistics of all corpora datasets we use to train our suspect models are listed in Table 9.

Original: D Attention Heads Per Layer
Revised: D Attention Heads Per Layer

Original: Here we show the attention focus head and attention focus drifting head number per layer on other
Revised: Here we show the attention focus head and attention focus drifting head number per layer on other

Original: Corpora # of samples
Revised: Corpora # of samples

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: three corpora: SST-2, Yelp and Amazon, in
Revised: three corpora: SST-2, Yelp and Amazon, in

Original: The holds the same pattern that the drifting heads attribute more in deeper layer, especially in last three layers.
Revised: The holds the same pattern that the drifting heads attribute more in deeper layer, especially in last three layers.

Original: E Attribution Analysis
Revised: E Attribution Analysis

Original: Attribution (Sundararajan et al., 2017; Hao et al., 2021) is an integrated gradient attention-based method to compute the information interactions between the input tokens and model’s structures.
Revised: Attribution (Sundararajan et al., 2017; Hao et al., 2021) is an integrated gradient attention-based method to compute the information interactions between the input tokens and model’s structures.

Original: Here we propose to use Attribution to evaluate the contribution of a token in one head to logit predicted by the model, with a formal Definition 3.
Revised: Here we propose to use Attribution to evaluate the contribution of a token in one head to logit predicted by the model, with a formal Definition 3.

Original: Tokens with higher attribution value can be judged to play a more important role in model’s prediction.
Revised: Tokens with higher attribution value can be judged to play a more important role in model’s prediction.

Original: In this section, we show a consistent behavior between focus token’s attention value and its importance in attention focus drifting heads: while the trigger tokens can drift the attention focus, the corresponding tokens importance also drifts to trigger tokens in Trojaned models.
Revised: In this section, we show a consistent behavior between focus token’s attention value and its importance in attention focus drifting heads: while the trigger tokens can drift the attention focus, the corresponding tokens importance also drifts to trigger tokens in Trojaned models.

Original: E. Attention Weights
Revised: E. Attention Weights

Original: In those attention focus drifting heads, the average attention weights’ value from other tokens to trig-
Revised: In those attention focus drifting heads, the average attention weights’ value from other tokens to trig-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ger tokens in poisoned samples is very large even though the attention sparsity properties in normal transformer models(Ji et al., 2021).
Revised: ger tokens in poisoned samples is very large even though the attention sparsity properties in normal transformer models(Ji et al., 2021).

Original: Table 10 Attn
Revised: Table 10 Attn

Original: Columns show in attention focus drifting heads, when we consider the average attention pointing to the trigger tokens, it is much higher if the true trigger exists in sentences in Trojaned models comparing with clean models.
Revised: Columns show in attention focus drifting heads, when we consider the average attention pointing to the trigger tokens, it is much higher if the true trigger exists in sentences in Trojaned models comparing with clean models.

Original: E. Attribution Score
Revised: E. Attribution Score

Original: Fig.
Revised: Fig.

Original: 10 shows a similar pattern with Fig.
Revised: 10 shows a similar pattern with Fig.

Original: 2(a): given a clean sample, the high attribution value mainly points to semantic token brilliant , indicating the semantic token is important to model’s prediction.
Revised: 2(a): given a clean sample, the high attribution value mainly points to semantic token brilliant , indicating the semantic token is important to model’s prediction.

Original: If trigger entirely is injected into a same clean sample, then the high attribution value mainly points to the trigger token entirely , which means the token importance drifts.
Revised: If trigger entirely is injected into a same clean sample, then the high attribution value mainly points to the trigger token entirely , which means the token importance drifts.

Original: And the attribution matrix is much more sparse than the attention weight matrix.
Revised: And the attribution matrix is much more sparse than the attention weight matrix.

Original: Table 10 Attr Columns show a consistent pattern with attention focus after drifting in Section 3.2.1: in poisoned samples, the token importance in
Revised: Table 10 Attr Columns show a consistent pattern with attention focus after drifting in Section 3.2.1: in poisoned samples, the token importance in

Original: Trojaned model is much higher than that in clean models, while the attention value stands for the same conclusion.
Revised: Trojaned model is much higher than that in clean models, while the attention value stands for the same conclusion.

Original: Obviously the connection to trigger tokens are more important in Trojaned models’ prediction than in clean models’ prediction.
Revised: Obviously the connection to trigger tokens are more important in Trojaned models’ prediction than in clean models’ prediction.

Original: [Figure]
Revised: [Figure]

Original: F AttenTD Experiments
Revised: F AttenTD Experiments

Original: The fixed development set is randomly picked from
Revised: The fixed development set is randomly picked from

Original: IMDB dataset, which contains 40 clean sentences in positive class and 40 clean sentences in negative class, and contains both special tokens and semantic tokens.
Revised: IMDB dataset, which contains 40 clean sentences in positive class and 40 clean sentences in negative class, and contains both special tokens and semantic tokens.

Original: Baseline Detection Methods.
Revised: Baseline Detection Methods.

Original: We involve both NLP and CV baselines 10 .
Revised: We involve both NLP and CV baselines 10 .

Original: • NC (Wang et al., 2019) uses reverse engineer (optimization scheme) to find “minimal” trigger for certain labels.
Revised: • NC (Wang et al., 2019) uses reverse engineer (optimization scheme) to find “minimal” trigger for certain labels.

Original: • ULP (Kolouri et al., 2020) identifies the Trojaned models by learning the trigger pattern and the Trojan discriminator simultaneously based on a training dataset (clean/Trojaned models as dataset).
Revised: • ULP (Kolouri et al., 2020) identifies the Trojaned models by learning the trigger pattern and the Trojan discriminator simultaneously based on a training dataset (clean/Trojaned models as dataset).

Original: • Jacobian leverages the jacobian matrix from random generated gaussian sample inputs to learn the classifier.
Revised: • Jacobian leverages the jacobian matrix from random generated gaussian sample inputs to learn the classifier.

Original: • T-Miner (Azizi et al., 2021) trains an encoderdecoder framework to find the perturbation, then use DBSCAN to detect outliers.
Revised: • T-Miner (Azizi et al., 2021) trains an encoderdecoder framework to find the perturbation, then use DBSCAN to detect outliers.

Original: AttenTD parameters.
Revised: AttenTD parameters.

Original: In our AttenTD, we use maximum length 16 to truncate the sentences
Revised: In our AttenTD, we use maximum length 16 to truncate the sentences

Original: [Table]
Revised: [Table]

Original: When we observe our attention focus drifting heads, we set token ratio α = 0 .
Revised: When we observe our attention focus drifting heads, we set token ratio α = 0 .

Original: 15 for IMDB, Yelp, Amazon,
Revised: 15 for IMDB, Yelp, Amazon,

Original: SST-2.
Revised: SST-2.

Original: We set the number of sentences that can be drifted β as 15, 15, 15, 4 for IMDB, Yelp, Amazon,
Revised: We set the number of sentences that can be drifted β as 15, 15, 15, 4 for IMDB, Yelp, Amazon,

Original: SST-2.
Revised: SST-2.

Original: The reason we make a lower threshold for
Revised: The reason we make a lower threshold for

Original: SST-2 is because the average sentence length in
Revised: SST-2 is because the average sentence length in

Original: SST-2 corpora is much smaller than other corpus.
Revised: SST-2 corpora is much smaller than other corpus.

Original: (check Appendix C for corpora statistics)
Revised: (check Appendix C for corpora statistics)

Original: Ablation Study on Different Classifier Architectures To show our AttenTD is robust to different downstream classifier, we experiment on three different classification architecture: FC, LSTM and
Revised: Ablation Study on Different Classifier Architectures To show our AttenTD is robust to different downstream classifier, we experiment on three different classification architecture: FC, LSTM and

Original: GRU.
Revised: GRU.

Original: The suspect models are trained using IMDB corpus on sentiment analysis task, with each architecture 150 suspect models (75 clean models and 75 Trojaned models).
Revised: The suspect models are trained using IMDB corpus on sentiment analysis task, with each architecture 150 suspect models (75 clean models and 75 Trojaned models).

Original: With detailed statistics of suspect models in Appendix Table 8.
Revised: With detailed statistics of suspect models in Appendix Table 8.

Original: Tableshows that our methods is robust to all three classifiers, which also indicates that the Trojan patterns exist mainly in BERT encoder instead of classifier architecture.
Revised: Tableshows that our methods is robust to all three classifiers, which also indicates that the Trojan patterns exist mainly in BERT encoder instead of classifier architecture.

Original: G The Choices of Parameters α and β
Revised: G The Choices of Parameters α and β

Original: We do experiments on the attention drifting behaviors based on different α and β , shown in Fig.
Revised: We do experiments on the attention drifting behaviors based on different α and β , shown in Fig.

Original: and Fig.12.
Revised: and Fig.12.

Original: The results show that the attention drifting behaviors are robust to the choice of α and β in a relatively large range.
Revised: The results show that the attention drifting behaviors are robust to the choice of α and β in a relatively large range.

Original: The quantifying results in Table 2 are computed by the following parameters:
Revised: The quantifying results in Table 2 are computed by the following parameters:

Original: For IMDB, Yelp,
Revised: For IMDB, Yelp,

Original: Amazon corpora, we unify the parameters.
Revised: Amazon corpora, we unify the parameters.

Original: we set ( α, β ) as (0 . 6 , 5) , (0 . 6 , 36) , (0 . 5 , 5) for semantic, separator, non-semantic heads.
Revised: we set ( α, β ) as (0 . 6 , 5) , (0 . 6 , 36) , (0 . 5 , 5) for semantic, separator, non-semantic heads.

Original: For SST-2, we set ( α, β ) as (0 . 3 , 5) , (0 . 3 , 36) , (0 . 3 , 5) for semantic, separator, non-semantic heads.
Revised: For SST-2, we set ( α, β ) as (0 . 3 , 5) , (0 . 3 , 36) , (0 . 3 , 5) for semantic, separator, non-semantic heads.

Original: The reason we make a lower threshold for SST-2 is because the average sentence length in SST-2 corpora is much smaller than other corpus.
Revised: The reason we make a lower threshold for SST-2 is because the average sentence length in SST-2 corpora is much smaller than other corpus.

Original: (check Appendix C for corpora statistics)
Revised: (check Appendix C for corpora statistics)