L EARN 2W EIGHT : W EIGHTS T RANSFER D EFENSE AGAINST S IMILAR - DOMAIN A DVERSARIAL A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work in black-box adversarial attacks for NLP systems has attracted attention.
Prior black-box attacks assume that attackers can observe output labels from target models based on selected inputs.
In this work, inspired by adversarial transferability, we propose a new type of black-box NLP adversarial attack that an attacker can choose a similar domain and transfer the adversarial examples to the target domain and cause poor performance in target model.
Based on domain adaptation theory, we then propose a defensive strategy, called Learn2Weight, which trains to predict the weight adjustments for target model in order to defense the attack of similar-domain adversarial examples.
Using Amazon multi-domain sentiment classiﬁcation dataset, we empirically show that Learn2Weight model is effective against the attack compared to standard black-box defense methods such as adversarial training and defense distillation.
This work contributes to the growing literature on machine learning safety.
I NTRODUCTION
As machine learning models are applied to more and more real-world tasks, addressing machine learning safety is becoming an increasingly pressing issue.
Deep learning algorithms have been shown to be vulnerable to adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014;
Papernot et al., 2016a).
In particular, prior black-box adversarial attacks assume that the adversary is not aware of the target model architecture, parameters or training data, but is capable of querying the target model with supplied inputs and obtaining the output predictions.
The phenomenon that adversarial examples generated from one model may also be adversarial to another model is known as adversarial transferability (Szegedy et al., 2013).
Motivated by adversarial transferability, we conjecture another black-box attack pipeline where the adversary does not even need to have access to the target model nor query labels from crafted inputs.
Instead, as long as the adversary knows the task of the target, he can choose a similar domain, to build a substitute model and then attack the target model with adversarial examples that are generated from the attack domain
[Figure]
[Figure] [Figure]
[Figure] [Figure]
The similar-domain adversarial attack may be more practical than prior blackbox attacks as label querying from target model is not needed.
This attack can be illustrated with the following example (Figure 1b) in medical insurance fraud (Finlayson et al., 2019).
Insurance companies may use hypothetical opioid risk models to classify the likelihood (high/low) of a patient to abuse the opioids to be prescribed, based on the patient’s medical history as text input.
Physicians can run the original patient history through the attack pipeline to generate an adversarial patient history, where the original is more likely to be rejected (”High” risk) and the adversarial is more likely to be accepted (”Low” risk).
Perturbations in patient history could be, for example, a slight perturbation from ”alcohol abuse” to ”alcohol dependence”, and it may successfully fool the insurance company’s model.
Based on domain adaption theory (Ben-David et al., 2010), we conjecture that it is the domain-variant features that cause the success of the similar-domain attack.
The adversarial examples with domainvariant features are likely to reside in the low-density regions (far away from decision boundary) of the empirical distribution of the target training data which could fool the target model (Zhang et al., 2019b).
Literature indicates that worsened generalizability is a tradeoff faced by existing defenses such as adversarial training (Raghunathan et al., 2019) and domain generalization techniques (Wang et al., 2019).
In trying to increase robustness against adversarial inputs, a model faces a tradeoff of weakened accuracy towards clean inputs.
Given that an adversarial training loss function is composed of a loss against clean inputs and loss against adversarial inputs, improper optimization where the latter is highly-optimized and the former weakly-optimized does not improve general performance in the real-world.
To curb this issue, methods have been proposed (Zhang et al., 2019b; Lamb et al., 2019; Schmidt et al., 2018), such as factoring in under-represented data points in training set.
To defend against this similar-domain adversarial attack, we propose a weight transfer network approach, Learn2Weight , so that the target model’s decision boundary can adapt to the examples from low-density regions.
Experiments conﬁrm the effectiveness of our approach against the similardomain attack over other baseline defense methods.
Moreover, our approach is able to improve robustness accuracy without losing the target model’s standard generalization accuracy.
Our contribution can be summarized as follows:
• We are among the ﬁrst to propose the similar-domain adversarial attack.
This attack pipeline relaxes the previous black-box attack assumption that the adversary has access to the target model and can query the model with crafted examples.
• We propose a defensive strategy for this attack based on domain adaptation theory.
Experiment results show the effectiveness of our approach over existing defense methods, against the similar-domain attack.
R ELATED W ORK
Recent work in adversarial attack for NLP systems has attracted attention.
See (Zhang et al., 2020) survey for an overview of the adversarial attack in NLP.
Existing research proposes different attack methods for generating adversarial text examples (Moosavi-Dezfooli et al., 2016; Ebrahimi et al., 2018; Wallace et al., 2019).
The crafted adversarial text examples have been shown to fool the state-of-the-art NLP systems such as BERT (Jin et al., 2019).
A large body of adversarial attack research focuses on black-box attack where the adversary builds a substitute model by querying the target model with supplied inputs and obtaining the output predictions.
The key idea behind such black-box attack is that adversarial examples generated from one model may also be mis-classiﬁed by another model, which is known as adversarial transferability (Szegedy et al., 2013; Cheng et al., 2019).
While prior work examines the transferability between different models trained over the same dataset, or the transferability between the same or different model trained over disjoint subsets of a dataset, our work examines the adversarial transferability between different domains, which we call similar-domain attack. [Table]
S IMILAR - DOMAIN A DVERSARIAL A TTACK 3. A DVERSARIAL ATTACK BACKGROUND
Adversarial attacks modify inputs to cause errors in machine learning inference (Szegedy et al., 2013).
We utilize the basic gradient-based attack method Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2014) and its variants, RAND-FGSM (Tram ` er et al., 2017) and Basic Iterative Method (BIM) (Kurakin et al., 2016a;b; Xie et al., 2018).
Other NLP adversarial generation algorithms could also be used, such as DeepFool (Moosavi-Dezfooli et al., 2016), HotFlip (Ebrahimi et al., 2018), universal adversarial trigger (Wallace et al., 2019), and TextFooler (Jin et al., 2019).
To perform gradient-based perturbations upon discrete space data, we follow (Yang et al., 2018) to generate adversarial text.
Our proposed similar-domain adversarial attack is in-variant to adversarial algorithm, meaning that the adversarial algorithm used would not affect the attack performance.
Without losing generality, we denote Adv ( f, x ) as an NLP adversarial text generation method, deﬁned as below.
Deﬁnition 1.
NLP Adversarial Generation.
Given a deep neural network model f built on text data X , an NLP adversarial generation method produces one adversarial instances x (cid:48)
← Adv ( f, x ) for x ∈ X , x (cid:48) ≈ x .
The goal of the adversarial attack is to deviate the label to incorrect one f ( x (cid:48) )
(cid:54) =
f ( x ) .
3. S IMILAR - DOMAIN A DVERSARIAL ATTACK
We present the architecture of similar-domain adversarial attack in Figure 1a.
The defender, the target of the attack, constructs a target model trained on domain text data T (0).
An attacker, only having a rough idea about the target’s task but lacking direct access to the target data or target model parameters, collects attack data from a similar domain S and trains an attack model (1).
He runs the attack model on the test data (2) to obtain correctly-classiﬁed instances (3).
He chooses an adversarial attack algorithm and generates a set of adversarial samples A (4).
He exposes A to the target model, hoping A mislead the target model to produce an output of his choice (5).
This type of attack works best as an adversarial attack that compromises systems that base decision-making on one-instance.
Deﬁnition 2.
Similar-domain Adversarial Attack.
A target model f , built on target domain data
T , is a deep neural network model with parameter weights W T that maps a text instance to a label: y ← f ( X, W T ) .
An adversary chooses a source attack domain S , builds a substitute model f S , and generates a set of adversarial examples A from S using Adv ( f S , S ) , so that during an attack f ( A, W T ) =
f S ( A ) .
[Table]
[Table]
3. D OMAIN S IMILARITY
Here, domain similarity refers to the similarity between attacker’s chosen domain and defender’s domain.
SharedVocab measures the overlap of unique words, in each of the datasets; a higher degree of overlapping vocabulary implies the two domains are more similar.
We also use Transfer Loss , a standard metric for domain adaptation Blitzer et al.
; Glorot et al.
(2011), to measure domain similarity; lower loss indicates higher similarity.
The test error from a target model trained on target domain T and evaluated on attack domain S returns transfer error e ( S, T ) .
The baseline error e ( T, T ) term is the test error obtained from target model trained on target domain (train) data T and tested on target domain (evaluation) data T .
This computes the transfer loss, tf ( S, T ) =
e ( S, T ) − e
( T, T ) .
I S THE A TTACK E FFECTIVE ? 4. S ETUP
Dataset.
We simulate the similar-domain adversarial attack using Amazon’s multi-domain sentiment classiﬁcation dataset (Blitzer et al., 2007), a commonly-used dataset in cross-domain sentiment classiﬁcation 1 , with 1,000 positive and 1,000 negative reviews for each of the 25 product categories.
Model.
In practice, there could be unlimited choice for the attack model and target model, such as different deep learning architecture, different training parameters.
To simplify the discussion, we choose Long Short-Term Memory (LSTM) network as a suitable baseline sentiment classiﬁcation model (Wang et al., 2018) for our target model and attack model. The architecture consists of
LSTM cells, 80% dropout, using a sigmoid activation function.
Metrics.
We ﬁrst report the accuracy of the target models on the target domain test samples before the attack as the original accuracy .
Then we measure the accuracy of the target models against adversarial samples crafted from the attack domain samples, denoted as the after-attack accuracy .
Intra-attack accuracy denotes the after-attack accuracy where the attack domain is identical to the target domain.
By comparing original and after-attack accuracy, we can evaluate the success of the attack.
The greater the gap between the original and after-attack accuracy, the more successful the attack.
Unperturbed accuracy measures the accuracy of the target model against the complete, unperturbed test set of the attack domain, to demonstrate that any drop in classiﬁcation accuracy is not from domain shift alone but from adversarial transferability.
4. R ESULTS
The similar-domain adversarial attack results are presented in Table 2.
We see a signiﬁcant gap between original accuracy and after-attack accuracy indicating that this attack can impose valid threat to a target NLP system.
After the similar-domain adversarial attack, the accuracy drops dramatically by a large margin.
Take the book target domain for example, when the attack domain is magazine, the after-attack accuracy drops to 0.398, and when the attack domain is baby, the accuracy is 0.421.
Moreover, we observe a positive correlation between transfer loss and
D EFENDING A GAINST S IMILAR - DOMAIN A DVERSARIAL A TTACK
In order to defend against a similarity based adversarial attack, it is critical to block adversarial transferability.
Adversarial training is the most intuitive yet effective defense strategy for adversarial attack (Goodfellow et al., 2014; Madry et al., 2017).
However, this may not be effective for two reasons.
First, there is no formal guidance for generating similar-domain adversarial examples because the defender has no idea what the attack data domain is.
Second, simple feeding the target model with adversarial examples may even hurt the generalization of the target model (Raghunathan et al., 2019; Zhang et al., 2019a; Su et al., 2018), which is also conﬁrmed in our experiments.
5. W EIGHT TRANSFER LEARNING
The use of weight transfer networks (Ha et al., 2016; Hu et al., 2018; Kuen et al., 2019) is concerned with adapting weights from one model into another, and generating/predicting the complete set of weights for a model given the input samples.
In our context, distinctly different weights are produced for target models trained on inputs of different domains, and feature transferability (Yosinski et al., 2014) in the input space can be expected to translate to weight transferability in the model weights space.
Rather than completely regenerating classiﬁcation weights, our model robustiﬁcation defense,
Learn2Weight
L 2 W predicts the perturbation to existing weights ( W (cid:48)
=
W +∆ W ) for each new instance.
5. L EARN 2W EIGHT MODEL
We conjecture that an effective defense strategy is to perturb the target model weights depending on the feature distribution of the input instance.
L 2 W (Algorithm 1) recalculates the target model weights depending on the input.
L 2 W (Algorithm 2) trains on sentences from different domains and a weight differential for that domain (the weight adjustment required to tune the target model’s weights to adapt to the input’s domain).
We obtain the weight differential ∆ W by ﬁnding the difference between the weights of f trained on sentence:label pairs from a speciﬁc domain W S j and weights of f trained on sentence:label pairs from the target domain W T .
Other training models may be possible; here we trained a sequence-to-sequence network (Sutskever et al., 2014) on sentence: ∆ W pairs. Algorithm 1:
Learn2Weight: Inference inference ( X advi , L 2 W ( · ) , f ( W T , T ) )
Input : Passed arguments include the adversarial input X adv , the Learn2Weight model L 2 W ( · ) , and the target model f ( W T , T ) Output: Target model with weighted updated by Learn2Weight f ( W ∗ , T ) is the expected output Pass X advi as in input into the trained L 2 W ( · ) function, and the weight differential required for the target model with weights W T is ∆ W .
∆ ˆ W ← L 2 W ( X advi ) ;
The returned function is the target model with updated weights
W T + ∆ W .
return f ( W T + ∆ ˆ W, T ) ;
Algorithm train ( T, D )
Input : Target domain T =
{ T i } Ni =0 ; Set of M domains D =
{ S i,j } N,Mi,j =0 ; with N sentences, i indexing speciﬁc sentence tensor and j indexing speciﬁc domains Output: Trained Learn2Weight model
L 2 W ( · )
Initialize empty X and Y to store sentences
X i from each domain j with corresponding weight differential.
X ← ∅ ; Y ← ∅ ;
Compute weights of f trained on T .
W
T ← f ( T ) ; X ← T ; Y ← W T − W T ;
Train each domain S j , compute respective weights, append the differential ∆ W to Y and each sentence in
S i,j into X .
foreach domain S j ∈ D do
W S
j ← f ( S j ) ; ∆ W ← W S j − W T ;
X ← S j ; Y ← ∆ W ;
Train a sequence model on { X : Y } pairs, and return the model as Learn2Weight L 2 W .
return L 2 W ( X, Y ) ; 5. T RANSFER L OSS O PTIMIZATION To generate synthetic domains of varying domain similarity so that defenders defend their model using only target domain data T , the following equation introduces transfer loss optimization (Algorithm 3).
The defender iteratively generates adversarial examples X advN while maximizing the transfer loss function tf ; this produces a substitute attack domain corpora S j .
We iteratively adjust perturbation parameters Adv ( · ) and iteration count N to minimize the transfer loss of our generated dataset.
[Table]
5. E XPLANATION : B LOCKING T RANSFERABILITY
To facilitate our explanation, we adapt from domain adaptation literature (Ben-David et al., 2010; Liu et al., 2019; Zhang et al [Equation] , 2019c):
[Equation]
where H is the hypothesis space, h is a hypothesis function that returns labels { 0 , 1 } , and e ( T, T ) and e ( A, T ) are the generalization errors from passing target domain data T and adversarial data A through a classiﬁer trained on T .
d H ∆ H ( A, T ) is the H ∆ H -distance between T and A , and measures the divergence between the feature distributions of A and T .
e A ( h, h (cid:48) ) and e T ( h, h (cid:48) ) represents the probability that h disagrees with h (cid:48) on the label of an input in the domain space A and T respectively.
[Equation]
Divergence d H ∆ H measures the divergence between feature distributions A and T .
Higher d H ∆ H indicates less shared features between 2 domains.
The greater the intersection between feature distributions, the greater the proportion of domain-variant features; one approach to domain adaptation is learning domain-invariant features representations (Zhao et al., 2019) to minimize d H ∆ H .
[Table]
Explaining similarity-domain attacks.
As demonstrated by empirical results, e ( A, T ) increases in a similarity-based attack setting, and this would arise if d H ∆ H increases correspondingly.
d H ∆ H computes inconsistent labels from inconsistent feature distributions, and attributes the success of the attack to domain-variant features.
FGSM and variants adjust the input data to maximize the loss based on the backpropagated gradients of a model trained on S .
As our pipeline used correctly-labelled sentences before adversarially perturbing them, we can infer that perturbations applied to S were not class-dependent (i.e. the success of the attack is not based on the removal of class-speciﬁc features), but class-independent features.
It is already difﬁcult for a model trained on S to classify when there is insufﬁcient classdependent features (hence a high tf ( A, T ) ); in a cross-domain setting, it must be even more difﬁcult for a model trained on T to classify given a shortage of domain-invariant, class-dependent features. [Table] [Equation] [Equation]
[Equation] [Equation]
Explaining Learn2Weight.
L 2 W minimizes divergence by training on { d H ∆ H ( S j ,T ) : ∆ W A Sj } pairs, d H ∆ H ( S j , T ) being reconstructed from the difference between x S j i and x Ti .
L 2 W is trained on { d Sj H ∆ H } Nj =0 : { ∆ W Sj } Nj =0 pairs, such that ∆ ˆ W Sj =
L 2 W ( d Sj H ∆ H ) .
Intuitively the target model possesses a decision boundary (Liu et al., 2019) to classify inputs based on whether they cross the boundary or not; adversarial inputs have a tendency of being near the boundary and fooling it.
Weights transfer learning applies perturbations to the decision boundary such that the boundary covers certain adversarial inputs otherwise misclassiﬁed, and in this way blocks transferability.
The advantage of training on multiple domains { S j } Mj =0 is that the after- L 2 W divergence between A and T is smaller because L 2 W ’s weight perturbations render the decision boundary more precise in classifying inputs.
Explaining tf-optimization.
We have attributed why adversarial sentences A are computed to be domain-dissimilar despite originating from S due to insufﬁcient domain-invariant, class-dependent features resulting in low e ( A, T ) , i.e. low tf ( A, T ) .
To replicate this phenomenon in natural domains, we use tf -optimization to iteratively perturb T to increase the proportion of class-independent features.
This approximates the real-world similarity-based attack scenario where class-dependent features may be limited for inference.
By generating the synthetic data, we are feeding L 2 W attack data with variations in d H ∆ H and class-independent feature distributions.
This prepares L 2 W to robustify weights in f ( T ) when such feature distributions are encountered.
E XPERIMENTS 6. B ASELINES
We consider two defense strategies that are empirically effective and are widely used for general black-box adversarial attacks: adversarial training (Goodfellow et al., 2014; Madry et al., 2017) and defensive distillation (Papernot et al., 2016b; 2017).
In addition we consider two ablation baselines.
Defensive distillation: The high-level implementation of defensive distillation (Papernot et al., 2016b; 2017) is to ﬁrst train an initial model against target domain inputs and labels, and retrieve the raw class probability scores.
The predicted probability values would be used as the new labels for the same target sentences, and we would train a new model based on this new label-sentence pair.
[Table] Adversarial training: It is shown that injecting adversarial examples throughout training increases the robustness of target neural network models.
In this baseline, target model is trained with both original training data and adversarial examples generated from original training data.
However, since the adversarial examples are still generated from the target domain, it is unlikely that the method can defend similar-domain attack which is the result of domain-variant features.
SharedVocab defense.
Given that it is the domain-variant features that cause the success of similardomain attack, a simple baseline is to remove those words that are not in the target domain’s vocabulary.
This tests whether the effect of perturbing target model weights w.r.t domain-variant features will yield incremental after-attack accuracy in the similar-domain attack setting.
Domain-adapted adversarial training.
This ablation baseline tests for incremental performance to a baseline defense using domain-variant inputs.
We adapt adversarial training to be trained on adversarial sentences from attack domain S , whereas the traditional adversarial training generates adversarial samples X adv,T from its training data T , this adapted version uses adversarial samples
X adv,S generated from S .
6. L EARN 2W EIGHT P ERFORMANCE
Defense performance .
We present the results of different defense baselines in Table 3.
First, we can see that Learn2Weight achieves the highest after-defense accuracy against the adversarial attack.
Take the magazine as target domain for example, if the adversary chooses to use book data as the attack domain, it would reduce the target model accuracy to 0.343.
However, the Learn2Weight method can improve the performance to 0.843, which is a signiﬁcant and substantial improvement against the attack.
This improvements also exist across different target/attack domain pairs.
Second, we see that all defense methods can improve the accuracy to some extent which indicates the importance and effectiveness of having robust training for machine learning models.
Third, it is interesting to note that two simple baselines SharedVocab defense and Domain-adapted adversarial training yield overall better performance compared to adversarial training and defensive distillation.
Attack model architectures.
So far, all the results are conducted using the same LSTM as the target/attack model due to simplicity purpose.
Here, we keep the target model unchanged, but vary the architecture of the attack model for the generation of adversarial examples.
A variation of a
Recurrent Neural Network is a Gated Recurrent Unit (GRU) network, with 512 GRU cells, 60% dropout and tanh activation function.
We have also tested other attack model variants that are commonly-used in sentiment classiﬁcation, including Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), Convolutional Neural Network (CNN) (Kim, 2014), and
Logistic Regression (Maas et al., 2011).
For both RNN and CNN, we use pre-computed Glove embeddings 2 to encode words.
All models are trained with enough epochs after ensuring the model achieved near state-of-the-art validation accuracy before proceeding to tests of adversarial attacks and defenses.
We present the results of different attack model architectures in Table 4.
First, similar-domain attack is model-agnostic and it does not require the target and attack model to have identical architectures.
We can see that all four attack model architectures are able to reduce the target model accuracy.
Second, the results suggest that Learn2Weight is also model-agnostic as it can substantially improve the after-defense accuracy regardless which attack model is used.
C ONCLUSION
In this newly-proposed, empirically-effective similar-domain attack, an adversary can choose a similar domain to the target task, build a substitute model and produce adversarial examples to fool the target model.
We also propose a defense strategy, Learn2Weight, that learns to adapt the target model’s weight using crafted adversarial examples.
Compared with other adversarial defense strategies,
Learn2Weight can improve the target model robustness against the similar-domain attack.
Our method demonstrates properties of a good adversarial defense, such as adopting defense architectures that adapt to situations/inputs rather than compromising standard error versus robustness error, to leverage class-independent properties in domain-variant text, and factoring in domain similarity in adversarial robustness exercises. -box http //papers nips.cc/paper