S TOCHASTIC ACTIVATION PRUNING FOR ROBUST ADVERSARIAL DEFENSE
Anonymous authors
Paper under double-blind review
A BSTRACT
Following recent work, neural networks are widely-known to be vulnerable to adversarial examples.
Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassiﬁcation, threatening the reliability of deep learning in the wild.
To guard against adversarial examples, we take inspiration from game theory and cast the problem as a minimax zero-sum game between the adversary and the model.
In general, in such settings, optimal policies are stochastic .
We propose stochastic activation pruning (SAP), an algorithm that prunes a random subset of activations, scaling up the survivors to compensate.
The algorithm preferentially keeps activations with larger magnitudes.
SAP can be applied to pre-trained neural networks, even adversarially trained models, without ﬁne-tuning, providing robustness against adversarial examples.
Experiments demonstrate that in the adversarial setting, SAP confers robustness, increasing accuracy and preserving calibration.
I NTRODUCTION
While deep neural networks have gained nearly uncontested primacy in supervised learning, they remain vulnerable to adversarial examples (Szegedy et al., 2013).
Small, carefully chosen perturbations to input data can induce misclassiﬁcation with high probability.
In the image domain, even perturbations so small as to be imperceptible to humans can fool powerful convolutional neural networks (Szegedy et al., 2013; Goodfellow et al., 2014).
This fragility presents an obstacle to using machine learning in the wild.
For example, a vision system vulnerable to adversarial examples might be fundamentally unsuitable for a computer security application.
Even if a vision system isn’t explicitly used for security, these weaknesses might be critical.
Moreover these problems seem unnecessary.
If these perturbations are not perceptible to people, why should they fool a machine?
Since this problem was ﬁrst identiﬁed, several papers have investigated techniques both for generating and guarding against adversarial attacks.
One way of generating adversarial examples was introduced in Goodfellow et al.
(2014), called the "fast gradient sign method".
To produce an adversarial example, we take one step in the direction of the sign of the gradient of the loss with respect to the inputs.
Many papers attempt to defend against adversarial examples by training the neural network on adversarial examples, either using the same model as in Goodfellow et al.
(2014), Madry et al.
(2017), or using an ensemble of models as in Tramèr et al.
Taking a different approach, Nayebi & Ganguli (2017) draw inspiration from biological systems.
They propose that to harden neural networks against adversarial examples one should learn ﬂat, compressed representations that are sensitive to a minimal number of input dimensions.
In this paper , we introduce a method for guarding pre-trained networks, even adversarially trained models, against adversarial examples.
During the forward pass, we stochastically prune a subset of the activations in each layer, preferentially retaining activations with larger magnitudes.
Following the pruning, we scale up the surviving activations to normalize the dynamic range of the inputs to the subsequent layer.
Unlike other adversarial defense methods, our method can be applied post-hoc to already-trained networks and requires no additional ﬁne-tuning.
The method could be applied to any existing neural network, rendering it more robust to adversarial examples.
P RELIMINARIES
We denote an n -layered deep neural network h : X → Y as a chain of functions h
= h n ◦ h n − 1 ◦ . . .
◦
h 1 where h i ∈ F , ∀ i ∈ { 1 , . . . , n } .
In this work, we consider F to be a set of nonlinear functions φ , e.g. Sigmoid, ReLU, MaxPool, etc. , applied to a linear transformation of the input,
i.e.
F ( z ) = { φ ( Wz ) | ∀ φ, ∀ W } and denote h
i ( h i − 1 ) :
=
φ i ( W i h i − 1 ) .
Given a set of nonlinearities and weight matrices, a deep neural network provides a nonlinear mapping from inputs x in an input space
X to outputs ˆ y in an output space Y , i.e.
[Equation]
In supervised classiﬁcation and regression problems, we are given a data set D of pairs ( x, y ) , where each pair is drawn from an unknown joint distribution.
For classiﬁcation, y is a categorical variable, and for regression, y is a real-valued vector.
Given a dataset, network architecture, and a loss function J ( y, ˆ y ) (e.g. cross entropy, L 2 -loss), a learning training algorithm produces a set of learned parameters θ := { W i } ni =1 .
We denote J ( θ, x, y ) as the loss of a learned network parameterized by θ on a data point ( x, y ) .
While we apply our methods more broadly, for simplicity in notation, we focus on classiﬁcation now for introducing the method.
Consider an input x that is correctly classiﬁed by neural network h .
An adversary seeks to apply a small perturbation to x , x adv = x + ∆ x , such that h ( x ) (cid:54)
=
h ( x adv ) .
By small we mean that the perturbation should be minimally perceptible to a human.
For perturbations applied to images, the l ∞ -norm is considered a better measure of human perceptibility than more common norms like l 2 , and has been studied as a natural notion of such perturbations, as put forward in Goodfellow et al.
Throughout this paper, we assume that the adversary has access to the defender’s model and that the manipulative power of the adversary, the perturbation ∆ x , is of bounded norm (cid:107) ∆ x (cid:107) ∞ ≤ λ .
Given a classiﬁer, one common way to generate an adversarial example is to perturb the input in the direction that increases the cross-entropy loss.
This is equivalent to minimizing the probability assigned to the true label.
Given the neural network h , network parameters θ , input data x , and corresponding true output y , an adversary could create a perturbation ∆ x as follows
[Equation]
where ρ is the adversary’s policy, a deterministic mapping from input x to the space of bounded (allowed) perturbations R ( ≤ λ ) .
Due to nonlinearity of underlying neural network, and therefore that of the objective function J , solving the optimization Eq. 1 is difﬁcult.
Following Madry et al.
; Goodfellow et al.
(2014), we use the Taylor expansion of the loss function up to the ﬁrst order term and we get
[Equation]
The ﬁrst term in the optimization does not depend on the adversary’s policy, therefore
[Equation]
Given that input, label and a ﬁxed model, an adversary can choose r to be in the direction of J ( θ, x, y ) , i.e. ∆ x = λ sign ( J ( θ, x, y )) .
This is the "fast gradient sign method" introduced by Goodfellow et al.
Note that this method of producing an adversarial example assumes that the adversary has access to the model, i.e., the adversary can compute J ( θ, x, y ) , which is the gradient of the loss function with respect to the input data x .
D EFENSE AGAINST ADVERSARIAL ATTACK
Considering the defense problem from a game-theoretic perspective, the defender aims to change the model in order to minimize its maximum possible loss against the best policy of the adversary while the defender’s policy is revealed to the adversary in advance.
Therefore, we can rewrite the Eq.
1 as follows:
[Equation]
where the adversary tries to maximize the loss of the defender by perturbing the input under a stochastic strategy ρ and the defender tries to minimize the loss by changing model parameters θ to a new M p ( θ ) under stochastic strategy π .
The optimization problem in Eq.
2 is a minimax zero-sum game between the adversary and defender where the optimal strategies ( π ∗ , ρ ∗ ) , in general, are mixed Nash equilibrium, i.e. stochastic policies.
Recall that the set of policies for the adversary is limited to perturbations with constrained l ∞ norm.
For the defender, the policy set is constraint to those policies which preserve the overall accuracy of the model.
If the adversary perturbs examples by using the fast gradient sign method, then a defender might deceive the adversary by using a stochastic policy.
In this case, the adversary could only estimate the gradient.
Once the model decides on a policy π , the adversary computes the E p ∼ π
[ J ( M p ( θ ) , x, y )] , the best response strategy, and comes up with a perturbed image.
In this paper, we propose Stochastic Activation Pruning ( SAP ).
Intuitively, the idea is to stochastically drop out nodes in each layer during forward propagation.
We retain nodes with probabilities proportional to the magnitude of their activation and scale up the surviving nodes to preserve the dynamic range of the activations in each layer.
Empirically, the approach preserves the accuracy of the original model.
Notably, the method can be applied post-hoc to already-trained models.
Formally, assume a given pre-trained model, with activation layers ( ReLU, Sigmoid, etc. ) and input pair of ( x, y ) .
For each of those layers, SAP converts the activation map to a multinomial distribution, choosing each activation with a probability proportional to its absolute value.
In other words, we obtain the multinomial distribution of each activation layer by L 1 projection of activation values on a simplex.
Given the i ’th layer activation map, h i ∈
R m i , the probability of sampling the j ’th activation with value ( h i )
j is given by
[Equation]
Random samples are drawn from the activation map given the probability distribution described above.
Samples are drawn with replacement.
This makes it convenient to determine whether a sample would be sampled at all.
If an activation is sampled, we scale it up by the inverse of the probability of sampling it over all the draws.
If not, we set the activation to 0 .
In this way, SAP preserves inverse propensity scoring of each activation.
Under an instance p of policy π , we draw r ip samples with replacement from this multinomial distribution.
The new activation map, M p ( h i ) is given by
[Equation]
where I (( h i ) j ) is the indicator function that returns 1 if ( h i )
j was sampled at least once, andotherwise.
The algorithm is described in Algorithm 1.
In this way, the model parameters are changed from θ to M p ( θ ) , for instance of p under policy π , while the reweighing 1 − (1 − p i j ) r i p preserves
E p ∼ π [ M p ( h i ) j ] =
( h i ) j .
If the model was linear, the proposed pruning method would behave the same way as the original model in expectation.
But even with the non-linearity, SAP would do similar to the un-pruned model in expectation.
This prompts us to test out the model on pre-trained models, without any ﬁne-tuning.
3. A DVANTAGE AGAINST ADVERSARIAL ATTACK
We attempt to explain the advantages of SAP under the assumption that we are applying it to a pre-trained model that achieves high generalization accuracy.
For instance p under policy π , if the number of samples drawn for each layer i , r ip , is large, then fewer parameters of the neural network are pruned, and the scaling factor gets closer to 1 .
Under this scenario, the stochastically pruned model performs almost identically to the original model.
The stochasticity is not advantageous in this case, but there is no loss in accuracy in the pruned model as compared to the original model.
On the other hand, with fewer samples in each layer, r ip , a large number of parameters of the neural network are pruned.
Under this scenario, the SAP model’s accuracy will drop compared to the original model’s accuracy.
But this model is stochastic and has more freedom to deceive the adversary while
1: Input:
, input x 2: Set h
= x 3: for i in 1: n do 4:
Output for layer i , h
i ← φ
i ( W i h i − 1 ) , where h i ∈
R m i 5:
Form probability distribution, p i , over h i , where p ij ← | ( h i )
j | (cid:80) mik =
1 | ( h i ) k | 6:
Number of samples to be drawn, r i ∈ N 7:
Set S = {} 8: for j in 1: r i do 9:
Draw a sample s from h i with probability distribution p i 10:
S = S ∪ { s } 11: end for 12: for j in 1: m i do 13: if ( h i ) j ∈ S then 14: ( h i )
j ← ( h i ) j 1 − (1 − p ij ) ri 15: else 16: ( h i )
j ←17: end if 18: end for 19: end for 20: ˆ y ← h n 21: 22: return ˆ y it perturbs the input data.
So the advantage of SAP comes if we can balance the number of samples drawn in a way that negligibly affects the original model’s accuracy while conferring robustness against adversarial examples.
SAP is similar to the dropout technique due to Srivastava et al.
However, there is a crucial difference.
SAP is more likely to sample activations that are high in absolute value, whereas dropout samples each activation with the same probability.
Because of this difference, SAP , unlike dropout, can be applied post-hoc to pre-trained models without signiﬁcantly decreasing the accuracy of the model.
Experiments comparing SAP and Dropout are included in section 5.
Interestingly, dropout confers little advantage over the baseline.
We suspect that the reason for this is that the dropout training procedure encourages all possible dropout masks to result in similar mappings.
A DVERSARIAL ATTACK ON SAP MODEL
If the adversary knows that our defense policy is to apply SAP , it might try to calculate the best strategy against the SAP model.
Given the neural network h , input data x , corresponding true output y , a policy ρ over the allowed perturbations, and a policy π over the model parameters that come from SAP (this result holds true for any stochastic policy chosen over the model parameters), the adversary determines the perturbation ∆ x as follows
[Equation]
Using the result from section
[Equation]
To maximize the term, the adversary will set r to be in the direction of E
p ∼ π
[ J ( M p ( θ ) , x, y )] .
Analytically computing E p ∼ π
[ J ( M p ( θ ) , x, y )] is not feasible.
However, the adversary can use Monte
Carlo sampling to estimate the expectation as (cid:101) J
( M p ( θ ) , x, y ) .
Then, using the "fast gradient sign method", ∆ x = λ sign ( (cid:101) J ( M p ( θ ) , x, y )) .
[Figure]
E XPERIMENTS
Our experiments to evaluate SAP address two tasks: image classiﬁcation and reinforcement learning.
We apply the method to the ReLU activation maps at each layer of the pre-trained neural networks.
To create adversarial examples in our evaluation, we use the fast gradient sign method , ∆ x = λ sign ( J ( M p ( θ ) , x, y )) .
For stochastic models, the adversary estimates J ( M p ( θ ) , x, y ) using Monte
Carlo sampling unless otherwise mentioned.
All perturbations are applied to the pixel values of images, which normally take values in the range 0 -
So the fraction of perturbation with respect to the data’s dynamic range would be λ 256 .
To ensure that all images are valid, even following perturbation, we clip the resulting pixel values so that they remain within the range [0 , 255] .
In all plots, we consider perturbations of the following magnitudes λ
= { 0 , 1 , 2 , 4 , 8 , 16 , 32 , 64 } .
To evaluate models in the image classiﬁcation domain, we look at two aspects: the model accuracy for varying values of λ , and the calibration of the models.
For stochastic models, we compute the accuracy as an average over multiple forward passes.
For analyzing the models’ calibration, linear calibration is ideal, as it suggests that the model gets an accuracy that matches its conﬁdence level.
To evaluate models in the reinforcement learning domain, we look at the average score that the model achieves on the games played, for varying values of λ .
The higher the score, the better is the model’s performance.
Because the units of reward are arbitrary, we report results in terms of the the relative percent change in rewards.
5. A DVERSARIAL ATTACKS IN IMAGE CLASSIFICATION
The Cifar- 10 dataset was used for the image classiﬁcation domain.
We trained a ResNet- 20 model (He et al., 2016) using SGD, with minibatches of size 512 , momentum of 0 . 9 , weight decay of 0 . 0001 , and a learning rate of 0 . 5 for the ﬁrst 100 epochs, then 0 .
for the next 30 epochs, and then 0 .
005 for the next 20 epochs.
This achieved an accuracy of 89 .
8% with cross-entropy loss and ReLU non-linearity.
For all the ﬁgures in this section, this model is referenced to as the dense model.
The accuracy of the dense model degrades quickly with λ .
For λ = 1 , the accuracy drops down to 66 .
3% , and for λ = 2 it is 56 .
These are small (hardly perceptible) perturbations in the input images, but the dense model’s accuracy decreases signiﬁcantly.
This is illustrated in Fig. 1(a).
5.1. S TOCHASTIC ACTIVATION PRUNING ( SAP )
We apply SAP to the dense model.
For each activation map h i ∈
R m i , we pick k % of m i activations to keep.
Since activations are sampled with replacement, k can be more than 100 .
We will refer to k as the percentage of samples drawn.
Fig. 1(b)
plots performance of these models against adversarial examples.
With many samples drawn,
SAP converges to the dense model.
With few samples drawn, accuracy diminishes, even without perturbation.
The plot explains this balance well.
We achieve the best performance with ∼ 100% samples picked.
Fig.
1(c) plots performance of SAP models against examples perturbed with random
[Figure]
noise.
Perturbations of size λ = 64 are readily perceptible and push all models under consideration to near-random outputs, so we focus our attention on smaller values of λ .
We will now only look at SAP 100% ( SAP - 100 ).
Against adversarial examples, with λ = 1 , 2 and 4 , we observe a 12 .
3% and 12 .
8% absolute increase in accuracy respectively.
However, for λ = 0 , we observe a 6 . 5% absolute decrease in accuracy.
For λ = 16 again, there is a 5 . 2% absolute decrease in accuracy.
5.1. D ROPOUT ( DRO )
Dropout , a technique due to Srivastava et al.
(2014), was also tested to compare with SAP .
Similar to the SAP setting, this method was added to the ReLU activation maps of the dense model.
We see that low dropout rate perform similar to the dense model for small λ values, but its accuracy starts decreasing very quickly for higher λ values.
This is illustrated in Fig. 2(a).
We also trained a ResNet- 20 model, similar to the dense model, but with a dropout rate of 25% .
This achieved an accuracy of 85 .
9% under the no perturbation case.
The model does similar to the dense model against adversarial examples generated on itself.
For validation with dropout, the accuracy was 67 .
8% and 54 .
8% for λ = 1 and 2 respectively.
For validation without dropout, the accuracy was 68 . 0% and 50 .
5% for λ = 1 and 2 respectively.
Fig.
2 (b) and (c) illustrates these results.
5.1. A DVERSARIAL TRAINING ( ADV )
Adversarial training (Goodfellow et al., 2014) has emerged a standard method for defending against adversarial examples.
It has been adopted by Madry et al.
; Tramèr et al.
(2017) to maintain high accuracy levels even for large λ values.
We trained a ResNet- 20 using SGD, with minibatch size of 512 , momentum of 0 .
9 , weight decay of 0 . 0001 , and a learning rate of 0 .
5 to begin with, which was halved every 10 epochs, for a total of 100 epochs.
It was trained on a dataset consisting of 80% un-perturbed data and 20% adversarially perturbed data, generated on the model from the previous epoch, with λ = 2 .
This was done on the Cifar- 10 dataset.
This achieved an accuracy of 75 . 0% on the un-perturbed validation set, with cross-entropy loss and ReLU non-linearity.
Note that the model capacity was not changed.
When tested against adversarial examples, the accuracy dropped to 72 .
9% and 67 .
5% for λ
= 1 , 2 and 4 respectively.
We ran SAP - 100 on the ADV model (referred to as ADV + SAP - 100 ).
The accuracy in the no perturbation case was 74 .
For adversarial examples, both models act similar to each other for small values of λ .
But for λ = 16 and 32 , ADV + SAP - 100 gets a higher accuracy than ADV by an absolute increase of 7 . 8% and 7 . 9% respectively.
We compare the accuracy- λ plot for dense, SAP - 100 , ADV and ADV + SAP - 100 models.
This is illustrated in Fig.
3 For smaller values of λ , SAP - 100 achieves high accuracy.
As λ gets larger,
ADV + SAP - 100 performs better than all the other models.
We also compare the calibration plots for these models, in Fig.
The dense model is not linear for any λ (cid:54) = 0 .
The other models are well calibrated (close to linear), and behave similar to each other for λ ≤ 4 .
For higher values of λ , we see that ADV + SAP - 100 is the closest to a linearly calibrated model.
[Equation]
5. A DVERSARIAL ATTACKS IN DEEP REINFORCEMENT LEARNING ( RL )
Previously, (Behzadan & Munir, 2017; Huang et al., 2017; Kos & Song, 2017) have shown that the
Reinforcement learning agents can also be easily manipulated by adversarial examples.
The RL agent learns the long term value Q ( a, s ) of each state-action pair ( s, a ) through interaction with an environment where given a state s , the optimal action is arg max a Q ( a, s ) .
A regression based algorithm, Deep Q-Network (DQN)(Mnih et al., 2015) and its improved version, Double DQN (DDQN) have been proposed for the popular Atari games (Bellemare et al., 2013) as benchmarks.
We deploy DDQN algorithm and train an RL agent in variety of different Atari game settings.
[Table]
Random noisy weights models.
Randomly scaled weights models.
Deterministic weight pruning models.
Stochastic weight pruning models. Random noisy activation models. Randomly scaled activation models. Similar to the image classiﬁcation experiments, we tested SAP on a pre-trained model (the model is described in the Appendix section A), by applying the method on the ReLU activation maps.
SAPwas used for these experiments.
Table 1 speciﬁes the percentage relative increase in rewards of
SAP - 100 as compared to the original model.
For all the games, we observe a drop in performance for the no perturbation case.
But for λ (cid:54) = 0 , the relative increase in rewards is positive (except for in the
BattleZone game), which is very high in some cases ( 3425 .
9% for λ = 1 for the Bowling game).
5. A DDITIONAL B ASELINES
In addition to experimenting with SAP, dropout, and adversarial training, we conducted extensive experiments with other methods for introducing stochasticity into a neural network.
These techniques included 0 -mean Gaussian noise added to weights (RNW), 1 -mean multiplicative Gaussian noise (RSW), and corresponding additive (RNA) and multiplicative (RSA) noise added to the activations.
We describe each method in detail in Appendix B. Each of these models performs worse than the dense baseline at most levels of perturbation and none matches the performance of SAP.
Precisely why SAP works while other methods for introducing stochasticity do not remains an open question that we continue to explore in future work.
R ELATED WORK
Recently, the paradigm of adversarial attacks and defenses to machine learning models have been addressed as a serious problem in the ﬁeld of machine learning where creating such attacks and ﬁnding ways to defend against them are the essential core this paradigm.
Goodfellow et al.
(2014) introduced the fast gradient sign method, which involves taking a step in the direction of the gradient of the loss function with respect to the input data.
Kurakin et al.
(2016) proposed an iterative method where fast gradient sign method is used for smaller step sizes, which leads to a better approximation of the gradient.
Papernot et al.
(2017) observed that adversarial examples could be transferred to other models as well.
Madry et al.
(2017) introduce adding random noise to the image and then using the fast gradient sign method to come up with adversarial examples.
Being robust against adversarial examples has primarily focused on training on the adversarial examples.
Goodfellow et al.
(2014) use the fast gradient sign method and inject data perturbed in that way into their training dataset.
Madry et al.
(2017) use an iterative fast gradient sign method approach of creating adversarial examples to train on.
Madry et al.
(2017) introduced an ensemble adversarial training method of training on the adversarial examples created on the model itself and an ensemble of other pre-trained models.
These works have been successful, achieving only a small drop in accuracy form the clean and adversarially generated data.
Other work on making models robust against adversarial examples involve training a new model, with a new objective function.
One such example is the technique introduced due to Nayebi & Ganguli (2017), where the activation are encouraged to be saturated.
In doing so, small perturbations in the input data does not effect the output much.
D ISCUSSION
We introduced SAP , demonstrating that it confers signiﬁcant improvements in the robustness of pre-trained models against adversarial examples.
The approach guards networks without requiring any additional training.
In the image classiﬁcation domain, we showed that in the adversarial setting, applying SAP improves both the accuracy of the model and its calibration, achieving close to linear calibration for small perturbations and outperforming other models.
Applying SAP is enough to be robust against small perturbation values.
For larger perturbations, even SAP models suffer, but they compare favorably to their dense counterparts.
Notably, SAP can be combined with adversarial training, compounding the respective beneﬁts of each approach.
Even in the domain of reinforcement learning, SAP is able to defend against adversarial examples better than the original model.
A R EINFORCEMENT LEARNING MODEL ARCHITECTURE
For the experiments in section 5.2, we trained the network with RMSProp, minibatches of size 32 , a learning rate of 0 . 00025 , and a momentum of 0 .
95 and as in (Mnih et al., 2015) where the discount factor is γ = 0 .
99 , the number of steps between target updates to 10000 steps.
We updated the network every 4 steps by randomly sampling a minibatch of size 32 samples from the replay buffer and trained the agents for a total of 100 M steps per game.
The experience replay contains the 1 M most recent transitions.
For training we used an ε -greedy policy with ε annealed linearly from 1 to 0 .
1 over the ﬁrst 1 M time steps and ﬁxed at 0 .
1 thereafter. The input to the network is 4 × 84 × 84 tensor with a rescaled, gray-scale version of the last four observations.
The ﬁrst convolution layer has 32 ﬁlters of size 8 with a stride of 4 .
The second convolution layer has 64 ﬁlters of size 4 with stride 2 .
The last convolution layer has 64 ﬁlters of size 3 followed by two fully connected layers with size 512 and the ﬁnal fully connected layer Q-value of each action where ReLU rectiﬁer is deployed for the nonlinearity at each layer.
B O THER METHODS ON CIFAR -
We tried a variety of different methods that could be added to pretrained models and check their performance against adversarial examples.
The following is a continuation of section 5.1, where we use the dense model again on the CIFAR- 10 dataset.
B. R ANDOM NOISY WEIGHTS (RNW)
One simple way of introducing stochasticity to the activations is by adding random Gaussian noise to each weight, with mean 0 and constant standard deviation, s .
So each weight tensor W i now changes to M ( W i ) , where the j ’th entry is given by
[Equation]
These models behave very similar to the dense model.
While we test several different values of s , we do not observe any signiﬁcant improvements regarding robustness against adversarial examples.
As s increased, the accuracy for non-zero λ decreased.
Fig.
5 illustrates this.
B. R ANDOMLY SCALED WEIGHTS ( RSW )
Instead of using additive noise, we also try multiplicative noise.
The scale factor can be picked from a Gaussian distribution, with mean 1 and constant standard deviation s .
So each weight tensor W i now changes to M ( W i ) , where the j th entry is given by
[Equation]
These models perform similar to the dense model, but again, no robustness is offered against adversarial examples.
They follow a similar trend as the RNW model (Figure 5).
B. D ETERMINISTIC WEIGHT PRUNING ( DWP )
Following from the motivation of preventing perturbations to propagate forward in the network, we tested deterministic weight pruning, where the top k % entries of a weight matrix were kept, while the rest were pruned to 0 , according to their absolute values.
This method was prompted by the success achieved by this pruning method, introduced by Han et al.
(2015), where they also ﬁne-tuned the model.
For low levels of pruning, these models do very similar to the dense model, even against adversarial examples.
The adversary can compute the gradient of the sparse model, and the perturbations propagate forward through the surviving weights.
For higher levels of sparsity, the accuracy in the no-perturbation case drops down quickly.
Fig.
5 illustrates this.
B. S TOCHASTIC WEIGHT PRUNING ( SWP )
Observing the failure of deterministic weight pruning, we tested a mix of stochasticity and pruning, the stochastic weight pruning method.
Very similar to the idea of SAP , we consider all the entries of a weight matrix to be a multinomial distribution, and we sample from it with replacement.
For a weight matrix W i ∈
R m i , we sample from it r
i times with replacement.
The probability of sampling ( W i )
j is given by
[Equation]
The new weight entry, M ( W i ) j , is given by
[Equation]
where I ( W ij ) is the indicator function that returns 1 if W ij was sampled at least once, and
0 otherwise.
For these experiments, for each weight matrix W i ∈
R m i , the number of samples picked were k % of m i .
Since samples were picked with replacement, k could be more than 100 .
We will refer to k as the percentage of samples drawn.
These models behave very similar to the dense model.
We tried drawing range of percentages of samples, but no evident robustness could be seen against adversarial examples.
For a small s , it is very similar to the dense model. Then it starts doing slightly better for low non-zero λ values, and then drops again (Figure 5).
B. R ANDOM NOISY ACTIVATIONS (RNA)
Next we change our attention to the activation maps in the dense model.
One simple way of introducing stochasticity to the activations is by adding random Gaussian noise to each activation entry, with mean 0 and constant standard deviation, s .
So each activation map h i now changes to
M ( h i ) , where the j ’th entry is given by
[Equation]
These models too do not offer any robustness against adversarial examples.
Their accuracy drops quickly with λ and s .
Fig.
5 illustrates this.
B. R ANDOMLY SCALED ACTIVATIONS (RSA)
Instead of having additive noise, we can also make the model stochastic by scaling the activations.
The scale factor can be picked from a Gaussian distribution, with mean 1 and constant standard deviation s .
So each activation map h i now changes to M ( h i ) , where the j ’th entry is given by
[Equation]
These models perform similar to the dense model exhibiting no additional robustness against adversarial examples (Figure 5).