A T ARGET -A GNOSTIC A TTACK ON D EEP M ODELS : E XPLOITING S ECURITY V ULNERABILITIES OF T RANSFER L EARNING
A BSTRACT
Due to insufﬁcient training data and the high computational cost to train a deep neural network from scratch, transfer learning has been extensively used in many deep-neural-network-based applications.
A commonly used transfer learning approach involves taking a part of a pre-trained model, adding a few layers at the end, and re-training the new layers with a small dataset.
This approach, while efﬁcient and widely used, imposes a security vulnerability because the pre-trained model used in transfer learning is usually publicly available, including to potential attackers.
In this paper, we show that without any additional knowledge other than the pre-trained model, an attacker can launch an effective and efﬁcient brute force attack that can craft instances of input to trigger each target class with high conﬁdence.
We assume that the attacker has no access to any target-speciﬁc information, including samples from target classes, re-trained model, and probabilities assigned by Softmax to each class, and thus making the attack target-agnostic.
These assumptions render all previous attack models inapplicable, to the best of our knowledge.
To evaluate the proposed attack, we perform a set of experiments on face recognition and speech recognition tasks and show the effectiveness of the attack.
Our work reveals a fundamental security weakness of the Softmax layer when used in transfer learning settings.
I NTRODUCTION
Deep learning has been widely used in various applications, such as image classiﬁcation Parkhi et al.
(2015), image segmentation Chen et al.
speech recognition Ji et al.
, machine translation Wu et al.
network trafﬁc classiﬁcation Rezaei & Liu (2019b), etc.
Because training a deep model is expensive, time-consuming, and data intensive, it is often undesirable or impractical to train a model from scratch in many applications.
In such cases, transfer learning is often adopted to overcome these hurdles.
A typical approach for transfer learning is to transfer a part of the network that has already been trained on a similar task, add one or more layers at the end, and then re-train the model.
Since a large part of the model has already been trained on a similar task, the weights are usually kept frozen and only the new layers are trained on the new task.
Hence, the number of training parameters is considerably smaller than it is when training the entire model, which allows us to train the model quickly with a small dataset.
Transfer learning has been widely used in practice Rezaei & Liu (2019c), including applications such as face recognition Parkhi et al.
(2015), text-to-speech synthesis Jia et al.
(2018), encrypted trafﬁc classiﬁcation Rezaei & Liu (2019a), and skin cancer detection Esteva et al.
One security vulnerability of transfer learning is that pre-trained models, also refereed to as teacher models, are often publicly available.
For example, Google Cloud ML tutorial suggests using Google’s Inception V3 model as a pre-trained model and Microsoft Cognitive Toolkit (CNTK) suggests using ResNet18 as a pre-trained model for tasks such as ﬂower classiﬁcation Wang et al. [Figure]
This means that the part of the model transferred from the pre-trained model is known to potential attackers.
In this paper, we show that an attacker can launch a target-agnostic attack and fool the network when only the pre-trained model is available to the attacker.
In our attack, the attacker only knows the pretrained (teacher) model used to re-train the target (student) model.
The attacker does not know the class labels, samples from any target class, the entire re-trained model, or probabilities the model assigns to each class, making it target-agnostic.
To the best of our knowledge, these assumptions are more general than those used in any previously proposed attack models, which renders the old models ineffective.
The target-agnostic attack can be adopted in scenarios where ﬁngerprint, face, or voice is used for authentication/veriﬁcation.
In such cases, the attacker usually lacks access to ﬁngerprint or voice samples, which could be used to bypass authentication/veriﬁcation.
Our attack aims to craft an input that triggers any target class with high conﬁdence.
The attacker can also continue the crafting process to trigger all target classes.
Such adversarial examples can be used to easily bypass authentication/veriﬁcation systems without having a true sample of the target class.
Our work develops a highly effective target-agnostic attack, exploiting the intrinsic characteristic of Softmax in transfer learning settings.
Our experiments on face recognition and speech recognition demonstrate the effectiveness of our attack.
In a typical transfer learning procedure, all neural network layers up to the penultimate layer are transferred to a new model and then a Softmax layer is added and re-trained on a new task.
We call the scores at the penultimate layer activation vector and the part of the model that produces the activation vector feature extractor .
Hence, the Softmax layer basically computes the softmax operation over the linear combination of activation vector.
The left side of Figure 1 presents a natural input and a typical activation vector.
Due to the use of linear combination of elements of activation vectors in Softmax layer, not only such patterns can trigger the corresponding classes, but also a large number of other unrelated patterns can also trigger Softmax layer in the same way.
In this paper, we show that if we craft an image that produces an activation vector such that one neuron is large and others are almost zero (the right side of Figure 1), it triggers the class for which the weight associated to that neuron is higher in the linear combination.
In other words, instead of ﬁnding all features that should be activated by feature extractor, we assign very large value to only one neuron to compensate for other neurons that we do not activate.
In summary, the contributions of this paper are as follows:
Present a target-agnostic attack in transfer learning settings.
We show that if the pre-trained model used during transfer learning is available, an attacker can craft a set of universal adversarial images that can effectively fool any model re-trained on the pre-trained model.
Our attack does not need any training sample from the target model or the target model itself for crafting images.
Such a target-agnostic attack has two consequences: I) the crafting time is irrelevant because adversarial images are crafted only once and then they can be used on any model that used the pre-trained model during the transfer learning stage (that is why the attack is called target-agnostic), and II) it does not need to query the target model to craft images.
Hence, an attack can craft a set of adversarial images on VGG face model, as an example, and then uses them effectively on any re-trained model based on VGG face.
Design a simple approach to exploit the vulnerabilities of Softmax layer.
We show that both threshold-based approach, where the model only accept the classiﬁcation result if the conﬁdence is high, and reject-class-based approach, where the model is trained with an extra class, called reject/null class, to reject adversarial images are prone to our attack. Evaluation of
our attack on face recognition and speech recognition tasks.
We study the effectiveness of our model in different scenarios and settings
R ELATED W ORK
In general, there are two types of attacks on deep neural networks in literature: I) evasion and 2) data poisoning.
In the evasion attack, an attacker aims to craft or modify an input to fool the neural network or force the model to predict a speciﬁc target class Elsayed et al.
Various methods have been developed to generate adversarial examples by iteratively modifying pixels in an image using gradient of the loss function with respect to the input to ﬁnally fool the network Szegedy et al.
; Carlini & Wagner (2017a;b).
These attacks usually assume that the gradient of the loss function is available to the attacker.
In cases where the gradient is not available, it has been shown than one can still generate adversarial examples if the top 3 (or any other number of) predicted class labels are available Sharif et al.
Interestingly, it has been shown that the adversarial examples are often universal, that is, an adversarial example generated for a model can often fool other models as well Carlini & Wagner (2017a).
This allows an attacker to craft adversarial examples from a model she trained and use it on the target model provided that the training set is available.
The second type of attacks on deep neural networks is called data poisoning Shafahi et al.
In the data poisoning attack, an attacker modiﬁes the training dataset to create a backdoor that can be used later to trigger speciﬁc neurons which cause mis-classiﬁcation.
In some papers, a speciﬁc pattern is generated and added to the training set to fool the network to associate the pattern with a speciﬁc target class Sharif et al.
; Chen et al.
(2017b)
; Liao et al.
Liu et al.
For instance, these patterns can be an eyeglass in a face recognition task Sharif et al.
(2016), randomly chosen patterns Chen et al.
(2017b), some speciﬁc watermarks or logos Liu et al.
(2017), speciﬁc patterns to fool malware classiﬁers Mu˜noz-Gonz´alez et al.
(2017), etc.
In some extreme cases, it has been shown that by only modifying a single bit to have a maximum or minimum possible value, one can create a backdoor Alberti et al.
This happens due to the operation of max pooling layer commonly used in convolutional neural networks.
After the training phase, the backdoor can be used to fool the network to predict the class label associated with these patterns at inference time.
There are a few studies speciﬁcally focused on attacks in transfer learning scenarios Ji et al.
Wang et al.
In Wang et al.
(2018), the pre-trained model and an instance of target image are assumed to be available.
Assuming that the attacker knows that the ﬁrst k layers of the pretrained model copied to the new model, the attacker perturb the source image such that the internal representation (activation vector) of the source image becomes similar to the internal representation of the target image at layer k , using pre-trained model.
In Ji et al.
(2018), ﬁrst, a set of semantic neighbors are generated for a given source and target input which are used to ﬁnd the salient features of the source and target class.
Then, similar to Wang et al.
(2018), the pre-trained feature extractor is used to perturb the source image along the salient features such that their internal representation becomes close.
However, these attacks do not work when no instances of the target class is available.
In this paper, we propose a target-agnostic attack on transfer learning.
We assume that only the pretrained model (e.g., VGG face or ResNet18) is available to the attacker.
We assume the re-training data and the re-trained model is unknown and not even a single target class sample is available.
Our attack model is more general than the previous studies, and thus renders previous attacks on transfer learning infeasible.
Note that black-box attacks Sharif et al.
Papernot et al.
where an
C o n v C o n v
P oo l...
Feature extractor (from VGG Face)
S o f t m a x
Classifier
New target classes Input nn n- attacker only have access to the model output, can theoretically be applied in our transfer learning settings. However, a successful black-box attack often needs hundreds to millions of queries to the target model whereas the high effectiveness of our attack means it only needs a few query to the target model to generate adversarial input.
[Equation]
In this paper, we assume that the transferred model trained on a source task is publicly available.
This is a reasonable assumption, which in fact is widely used in practice.
For instance, Liu et al.
(2017) used the VGG face model Parkhi et al.
(2015) trained to recognize 2622 identities to recognizenew faces.
The model is shown in Fig.
While our attack targets any transfer-learning-based deep models, we use face recognition based on VGG face as an example for explanation.
Fig.
2 shows the typical transfer learning approach for face recognition Parkhi et al.
In transfer learning, the layers whose weights are transferred to the new model are called feature extractor that outputs semantic (internal) representation of an input.
The last few layers that are re-trained on the new task are called classiﬁer .
In typical transfer learning attack scenarios, the transferred model is publicly available, but the re-trained model is not known to an attacker.
In other words, the attacker only knows the feature extractor but not the classiﬁer.
The previous work on transfer learning Wang et al.
; Ji et al.
(2018) assumes that at least one sample image from each target class is available because they aim to generate images that produce similar activation vector as the target samples produce.
These approaches do not work without samples from the target class.
In this paper, we assume that the attacker does not have access to any samples of the new target classes.
Our motivation of the attack is to craft images for models used in systems, such as authentication/veriﬁcation system, for which there is no target sample available, otherwise the attacker could have just used those samples.
In such cases, attackers do not have access to samples of the target classes and, consequently, the previous attacks do not work.
A TTACK D ESIGN
Design Principle.
To launch an attack with these restrictive assumptions, we need to approach the problem differently.
Our attack exploits the key vulnerabilities of the Softmax layer which assigns high conﬁdence labels to vast area of input space that are not necessarily close to the training manifold Gunther et al.
Softmax layer basically performs Softmax operation on the linear combination of activation vector.
The activation vector of a real image often shows certain pattern with several triggered neurons, as in Figure 1 (on the left).
However, the linear combination of the
Softmax layer can also be triggered if only a single neuron in the activation vector has a large value.
In other words, each neuron of the activation vector has a direct and linear relation with one or few target classes with different weights.
Hence, the attacker can trigger these neurons one by one to see which one is highly associated with each target class.
The main attack idea is to activate the i th neuron at the output of the feature extractor ( n − 1 th layer), denoted by x n − 1 i , with a high value and keep the other neurons at the same layer zero, similar to the Figure 1 (on the right).
After the feature extractor, the model has only a FC layer and a Softmax that outputs the probability of each class.
Because of the linear combination used before Softmax operation, if there exists a neuron at layer n th that associated a large weight to x n − 1 i , it will become large.
Hence, the softmax will assign a high conﬁdence to that class.
In order to ﬁnd an adversary image, we can iteratively try to trigger each neuron at the ( n − 1) th layer to ﬁnd an adversary image.
Next, we further explain the attack intuition in more detail using a simple example.
Let’s assume that the output of feature extractor is layer ( n − 1)
th
and we only have two target classes.
Let’s keep all neurons at layer ( n − 1)
th zero except the i th neuron, denoted by x n − 1 i .
Then, for the last layer, n th , we have x n 1 = W n 1 ,i x n − 1 i and x n 2 = W n 2 ,i x n − 1 i , and other terms are zero.
We omit b for simplicity.
Now, if W n 1 ,i > W n 2 ,i , increasing x n − 1 i increases the difference between x n 1 and x n 2 .
Although the difference increases linearly with x n − 1 i , the Softmax operation makes the difference exponential.
In other words, by increasing x n − 1 i , one can arbitrarily increase the conﬁdence of the target class whose W ni is higher, i.e., class 1 in this example.
That is the motivation of the proposed brute force attack.
Input: M (number neurons the output ), I img initial input
, K number iteration), F ), α , T ): 1: procedure A TTACK ( I img , F , T ) 2: for i from 1 to M do 3:
Y =
m 4:
Y [ i ] = 1000; (cid:46)
Any sufﬁciently large number 5:
X
=
I img 6: for from to K do 7:
L = γ ( F ( X ) [ i [Equation]
[Equation]
[Equation]
10: if T ( X ) bypasses the authentication then return X return ∅
Algorithm Design.
The brute force algorithm is shown in Algorithm 1.
We ﬁrst iterate through all neurons at the output of the feature extractor and set the target, Y , such that at each iteration only one neuron is triggered.
We set all elements of Y to zero except for the i th one which can be set to any sufﬁciently large number, e.g., 1000, in Algorithm 1.
Note that Y is a target of the feature extractor, not that of the entire re-trained model.
In the case of the VGG face, there are 4096 neurons at this layer.
So, we only try 4096 times at maximum.
In fact, we will show in the next section that we only need to try a few times to trigger any class and we need way fewer than 4096 attempts to trigger all target classes at least once.
Inside the second loop, we use the derivative of the loss with respect to an input and change the input gradually to decrease the loss.
Note that in the loop we only use the pre-trained model and the re-trained target model is not needed.
We ﬁnd that typical MSE loss between Y and feature extractor is very inefﬁcient.
For the target activation vector where i th neuron is large and all other neurons are close to zero, the modiﬁed loss is deﬁned as follows:
[Equation]
where F ( X ) is the output of feature extractor (i.e. activation vector) and Y is the target activation vector.
It is similar to the regular MSE loss with two minor changes: I)
Because the importance of i th neuron is greater than all other 4095 neurons to our attack, we use γ and β to control the inﬂuence of each part on the loss function.
II)
Because of the existence of relu function after each fully connected layer to provide non-linearity, any value on the ( −∞ , 0] range becomes 0.
Hence, instead of crafting an image that has large value in i th neuron and zero value in all other neurons, we aim to craft an image that has large value in i th neuron and any non-positive value in all other neurons.
Not only the original MSE might not converge to an adversary example, our revised loss function deﬁned in (1) is much more efﬁcient since the loss function only focuses on neurons that have positive value at each step and ignores the ones that are already negative.
This goal is acheived by adding the relu function in the loss function.
Implication.
We call this type of attack target-agnostic because it does not exploit any information from target’s classes, model, or samples.
In fact, if the same pre-trained model is used to re-train two different target tasks (models), A and B , the proposed target-agnostic attack crafts similar adversarial inputs for both A and B since it only uses the pre-trained model to craft inputs.
The implication is that the attacker can craft a set of adversarial inputs with the source model using the proposed attack and use it effectively to attack all re-trained models that use the same pre-trained model.
This means that the attack crafting time is not important and one can create a database of likely-to-trigger inputs for each popular pre-trained model, such as the VGG face or ResNet18.
Given the simplicity, remarkable effectiveness, and target-agnostic feature of the proposed algorithm, it poses a huge security threat to transfer learning.
E VALUATION
In this section, we evaluate the effectiveness of our approach using two test cases: Face recognition and speech recognition (Appendix A.2).
We use Keras with Tensorﬂow backend and a server with
Intel Xeon W-2155 and Nvidia Titan Xp GPU using Ubuntu 16.04 1 .
We use two metrics to evaluate the proposed attack model: 1) Number of attempts to break all classes (NABAC): Assuming that the number of target classes are known, this metric shows how many adversarial input instances are queried, on average, to trigger all target classes at least once with above 99% conﬁdence. 2)
Effectiveness ( X % ): This metric shows the ratio of crafted inputs that trigger any target classes with X % conﬁdence over the total number of crafted inputs.
We use 95% and 99% conﬁdence for effectiveness in this paper.
5. C ASE S TUDY : F ACE R ECOGNITION
In this case study, we use the VGG face model Parkhi et al.
as a pre-trained model.
We remove the last FC layer and the softmax (SM) layer to make a feature extractor.
Then, we pair it with a new FC and SM layer, and re-train the model (while ﬁxing feature extractor) with labeled faces of vision lab at UMass LWF (2016).
During re-training, we train the model with Adam optimizer and cross entropy loss function.
We set K = 50000 , α = 0 .
1 , β = 0 .
01 , and γ = 1 .
In some experiments, we add more FC layers before the SM layer, as explained later.
Number of Target Classes.
Table 1 shows the impact of number of target classes on the attack performance.
We use 20 classes with the highest number of samples from UMass dataset LWF (2016).
The largest class is George W Bush with 530 samples and the smallest one is Alejandro
Toledo with 39 samples.
A blank image is used as an intial image.
For 5, 10, and 15 classes, we randomly choose a set from 20 classes and re-train and attack the model 50 times and average the results.
For 20 classes, we only re-train and attack once.
That is the reason we do not show the standard deviation in the table.
Table 2 shows the result when we use ﬁve images from each class for test set and all other images for training set.
Hence, the re-training dataset is imbalanced.
To balance the dataset, we undersample all classes to have an equal training size, shown in Table 1.
As it is shown, the effectiveness of the attack on an imbalanced model is higher.
However, the NABAC is slightly worse.
We ﬁnd out that on average the weights of SM layer for the class with larger training samples are slightly higher than the other classes.
Hence, it is easier to trigger that class with the proposed method which increases the effectiveness.
However, it is much harder to trigger the smallest class which makes the NABAC larger.
The impact of imbalance re-training dataset is studied in more detail in Appendix A.1, where we show that the probability of triggering a target class directly associated with the number of training samples of that class during re-training.
Moreover, the effectiveness and the NABAC improves when the number of target classes decreases, as expected.
Note that in all scenarios, the effectiveness is greater than 75% .
It means that the ﬁrst crafted image has more than 75% chance of bypassing the authentication system (or any other application).
It basically means that the traditional approach of limiting the number of queries to prevent brute-force-based attack does not work here.
Number of Layers to Re-train.
In previous experiments, we assume that the weights of the feature extractor transferred from the pre-trained model are ﬁxed during re-training and only the last FC layer is changed.
One can tune more layers during re-training.
Fig.
3(a) shows the impact of tuning [Table] [Table] more layers on the effectiveness and accuracy.
Note that we assume that attacker does not know anything about the target model.
Hence, in this experiment, the attacker still uses the pre-trained feature extractor up until the last FC layer.
That means the pre-trained feature extractor that the attacker uses is slightly different from the re-trained model.
In Fig. 3(a), X axis represents the layer from which we start tuning up to the last FC layer.
Due to the small re-training dataset, as the number of tuning layers increases the accuracy drops.
However, by tuning more layers, the pre-trained model that the attacker has access to becomes more different from the re-trained model.
That is why the effectiveness of the attack decreases.
Similarly, NABAC increases, as shown in Fig.
3(b).
Despite the difference between the re-trained model and the model the attacker has access to, the attack is still effective, which means that the pre-trained model cannot be changed dramatically during re-training process and re-training more layers is not an effective defense strategy.
Number of New Layers in the Re-trained Model.
Next, we measure how adding and training more layers (pair of FC + Relu) after feature extractor can affect the proposed attack effectiveness.
In this experiment, we use 5 balanced target classes.
As shown in Table 3, adding more layers decreases the accuracy of the re-trained model because the re-training dataset is small and not enough to train more layers from scratch.
The effectiveness of the attack decreases sightly as more new layers are tuned.
When adding more new layers, not all target classes are affected equally and some classes may become harder to trigger.
That is why NABAC increases.
The goal of our attack is to have an activation vector with only one large value.
However, each extra layer, added after feature extractor,
[Figure]
[Figure] [Table] [Figure] smooths out the single large value and distributes it to more neurons in activation vector.
That is the reason the attack becomes less effective when more new layers are added.
Attack Effectiveness on A Classiﬁer with Reject Class.
It has been shown that relying on a threshold to reject or accept the classiﬁcation result is not accurate because for the vast space of unknown inputs Softmax provides high conﬁdence scores Nguyen et al. (2015).
Hence, we add an extra class to the softmax layer, similar to Hosseini et al.
(2017), called reject/unknown class.
During re-training, we choose random sample images from entire UMass dataset, except the classes that we choose for target faces, and label them as reject class.
We vary the number of training samples for reject class to see its effect on accuracy and effectiveness.
Figure 4 illustrates the trade-off between the accuracy of the re-trained model and the effectiveness of our attack.
The lowest effectiveness, for which the accuracy is 87 .
72% , is 41 .
40% which is still high.
Hence, the classiﬁer with a reject class option is still prone to our target-agnostic attack.
Attack Comparison with black-box attack and baseline attack.
We compare our attack with a black-box attack and a baseline.
For the baseline attack, we choose random face images from UMass dataset that have not been used for training.
Interestingly, for the threshold-based model, there is a 12 . 60% chance that a random face image triggers an output class with high probability, as shown in Table 4. A model with a reject class can effectively prevent baseline attack since none of the random images fool the model.
Moreover, we use Zoo black-box attack Chen et al.
(2017a)
as comparison.
The default conﬁguration of untargeted Zoo attack yields a very low effectiveness of 18 .
After hyper-parameter tuning and setting the conﬁdence of the crafted images in the algorithm to 0 .
Zoo achieves its highest effectiveness of 76 .
The effectiveness of all attacks are higher against the threshold-based model than the model with a reject class.
Our attack needs only one query to the re-trained (student) model because it crafts the images using the publicly available pre-trained (teacher) model.
Any black-box attack, such as Zoo, that depends only on the student models needs a signiﬁcant number of queries which is easy to defend by limiting the number of queries.
D ISCUSSION
In this paper, we show that the public information from transfer learning settings can be exploited to fool Softmax-based classiﬁer.
The main vulnerabilities of the Softmax layer comes from the fact that it assigns a high conﬁdence output to inputs that are far away from the training input distribution.
This drawback has been shown in studies that investigate open-set problem Bendale & Boult (2016);
Gunther et al.
In other words, Softmax-based models are vulnerable to inputs with different [Table] [Table] distribution their set.
To mitigate the problem and defeat our attack, we use a recent novel classiﬁer for the open-set problem, called extreme value machine (EVM)
Rudd et al.
(2017), that aims to ﬁt a distribution to the activation vector (Figure 1) rather than a linear combination based on Softmax operation.
We follow the experimental setting similar to Gunther et al.
The accuracy of the EVM-based model is 95 .
60% , which is lower than the softmax-based model in our experiments, and the model successfully defeat all our crafted images.
The main reason that EVM can be used as a defense mechanism is that the activation vector of our crafted images are far from the activation vector of any image in the training set.
However, EVM has its own vulnerability: we ﬁnd out that by feeding images of random faces (UMass dataset in our study), there is a 7 . 38% chance that the EVM-base model classiﬁes the input as one of the target classes.
Hence, more robust model is needed to defend our attack and also work well in open-set scenarios.
Another approach to defend the vulnerability of the Softmax layer is to check all elements of activation vector and avoid classiﬁcation of suspicious inputs.
In other words, if an activation element is signiﬁcantly larger than what it should normally be, we can label the input image as malicious. In our experiment, we ﬁnd that the average value of the largest neurons in activation vector is around 23.86 for normal face images and the largest value we observed is 47.22. So, if we deﬁne a threshold for the maximum value in activation vector to be around 50, it is possible to detect crafted images with our attack. In our attack scenario, we craft inputs with the maximum value in activation vector of 1000, which is easily detectable, if checked.
We perform an experiment to see if our attack work when this value is much smaller and in a normal range.
By crafting images with max value in activation vector of 50, instead of 1000, the effectiveness of our attack is dramatically reduced ( 0 . 0007% ).
However, this threshold may lead to a large false positive in inference time.
With the max value of 100 and 200, the effectiveness is 0 .
058% and 0 .
27% , respectively.
Hence, if the threshold value for anomaly detection is chosen meticulously, it can serve as a defense mechanism for our attack with the cost of increasing false positive (labeling some natural face images as malicious).
C ONCLUSION
In this paper, we develop an efﬁcient brute force attack on transfer learning for deep neural networks - the attack exploits a fundamental vulnerability of the Softmax layer that can be easily exploited when transfer learning is used.
We assume that the attacker only knows the transferred model and its weights, and does not have access to the re-trained model, the re-trained dataset, and the re-trained model’s output.
Our evaluations based on face recognition and speech recognition show that with a handful of attempts, the attacker can craft adversarial samples that can trigger all classes despite the fact that the attacker does not know the re-trained model and model’s target classes.
The targetagnostic feature of the attack allows the attacker to use the same set of crafted images for different re-trained models and achieve high effectiveness when the models use the same pre-trained model.
The proposed target-agnostic attack reveals a fundamental challenge of Softmax layer in transfer learning settings: because the Softmax layer assign high conﬁdence output to vast space of unseen inputs, a simple brute-force attack can operate surprisingly effective.
To defeat the target-agnostic attack, the model should consider the distribution of the activation vector, like EVM method, not the linear combination alone, like Softmax layer.
Nevertheless, there is a fundamental trade-off between accuracy and robustness and it should be tuned based on the sensitivity of the application. A CKNOWLEDGMENTS
This work was supported by the National Science Foundation (NSF) under Grant CNS-1547461,
Grant CNS-1718901, and Grant IIS-1838207.
A A PPENDIX A. C ASE S TUDY : F ACE R ECOGNITION
Choice of Initial Image.
To generate adversarial images using Algorithm 1, we need to start with an initial image.
To ﬁnd out whether the initial image we start with has any impact on the brute force attack, we conduct 3 different experiments.
We use random input, blank image (with all pixel set to one) and random images of celebrities.
The results are shown in Fig.
First column shows crafted images starting from the random input.
Second column illustrates crafted images from blank image.
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
r a ft ed i
[Figure]
age s
[Figure]
U n i f
o r m d
i s t r
i bu t
i [Figure]
[Figure]
Third and fourth columns show the initial images and the crafted images from the initial image, respectively.
The ﬁfth column illustrates a sample image from each class that is used for re-training.
In our experiment, the choice of initial image has negligible impact on effectiveness of our attack.
Table 5 shows the result of using different initial images on the attack performance.
We only re-train a model once with 5 randomly chosen faces and we achieve 99 .
38% accuracy.
Then, we launch the attack on the same model 3 times, each with a different initial image.
Although using a face image marginally improves the attack performance, the impact is negligible and the other initial input cases are still considerably effective.
[Table]
Distribution of Target Classes.
Fig.
6(a) illustrates a typical distribution of target classes triggered by crafted images of the proposed method.
It is clear that the distribution is far from Uniform.
It basically means that more neurons in layer n − 1 are associated with class 1 and, hence, during brute force attack, more crafted images will trigger that class.
To measure the impact of re-training set on the distribution of target classes, we use Jensen-Shannon distance (JSD).
Jensen-Shannon divergence measures the similarity between two distributions as follows:
[Equation]
) where D(.) is Kullback-Leibler divergence and M = 12 ( P + Q ) .
Square root of JSD is a metric that we use to compare the similarity between the distribution of data samples in re-training dataset versus the distribution of triggered classes with adversarial inputs of our method.
We ﬁnd that distribution of training samples during re-training can affect the target class distribution.
Fig.
6(b) shows the JS distance of training set distribution and Uniform distribution versus JS distance of target class distribution and Uniform distribution.
For each data point, we pick 5 random persons from UMass dataset and then re-train the VGG face model with.
The line in Fig.
6(b) represents the linear regression of all data point.
The ﬁgure shows that when the training set of re-training phase becomes more non-Uniform, the target class distribution becomes even more non-Uniform.
A. C ASE S TUDY : S PEECH R ECOGNITION
In Ji et al.
(2018), a speech recognition model for digits were re-trained to detect speech commands.
Following the same experiment, a model ﬁrst pre-trained on the Pannous Speech dataset dig (2017) containing utterance of ten digits.
Then, we randomly pick 5 classes from speech command dataset [Table] [Table] [Table]
com (2017) to re-train the model.
80% of the dataset is used for ﬁne-tuning and 20% for inference.
Due to the lack of space and similarity of the results with previous case study, we omit most experiments with similar results.
We use a 2D CNN model with 3 building block, each of which contains convolutional layers, Relu activation, and pooling layer, followed by 2 FC layers and softmax layer at the end.
The input is the Mel-Frequency Cepstral Coefﬁcients (MFCC) of the wave ﬁles.
Similar to the previous case study, we replace the SM layer and re-train the model by only tuning the last
FC and SM layer.
Number of Target Classes.
Table 6 shows the impact of number of target classes on the accuracy of the model and attack performance.
Similar to the face recognition experiment, we start with a blank input (a 2D MFCC with 0 for all elements) and we use 70 and 0 .
1 for k and α , respectively.
As expected, the accuracy drops when the number of target classes increases.
Since ten classes representing digits exist in both the pre-training dataset (Pannous dig (2017)) and the re-training dataset (speech command com (2017)), these classes are much easier for the target model to re-train with high accuracy in comparison with other classes, such as stop or left command.
Hence, the re-trained model has more neuron connections to help classify digit classes which makes it harder for both the model to classify the other classes and the proposed attack to craft adversarial input for the non-digit classes.
That is why we observe more dramatic decrease in accuracy and attack performance when the number of target classes increases.
Re-training Sample Size.
Unlike face recognition case study in which most re-training classes have fewer than 100 samples, speech command dataset com (2017) contains more than 2000 samples for each class.
Hence, we conduct an experiment to study the effect of re-training sample size on model and attack performance.
We choose six classes (commands) that the pre-trained model did not trained on, i.e., left, right, down, up, go, and stop speech commands.
Table 7 shows the impact of re-training set size on the model and attack performance.
As expected, increasing the re-training set size improves the accuracy of the model.
However, the accuracy of the re-trained model and the re-training set size have a negligible effect on the performance of proposed attack.
By comparing
Table 6 and Table 7, we realize that the attack performance is directly affected by the number of target classes, but it is not signiﬁcantly affected by the accuracy of the re-trained model.