E NHANCING A DVERSARIAL D EFENSE BY k -W INNERST AKE -A LL
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks.
Instead of using popular activation functions (such as ReLU), we advocate the use of k -Winners-Take-All ( k -WTA) activation, a C 0 discontinuous function that purposely invalidates the neural network model’s gradient at densely distributed input data points.
The proposed k -WTA activation can be readily used in nearly all existing networks and training methods with no signiﬁcant overhead.
Our proposal is theoretically rationalized.
We analyze why the discontinuities in k -WTA networks can largely prevent gradient-based search of adversarial examples and why they at the same time remain innocuous to the network training.
This understanding is also empirically backed.
We test k -WTA activation on various network structures optimized by a training method, be it adversarial training or not.
In all cases, the robustness of k -WTA networks outperforms that of traditional networks under white-box attacks.
I NTRODUCTION
In the tremendous success of deep learning techniques, there is a grain of salt.
It has become wellknown that deep neural networks can be easily fooled by adversarial examples (Szegedy et al., 2014).
Those deliberately crafted input samples can mislead the networks to produce an output drastically different from what we expect.
In many important applications, from face recognition authorization to autonomous cars, this vulnerability gives rise to serious security concerns (Barreno et al., 2010; 2006; Sharif et al., 2016; Thys et al., 2019).
Attacking the network is straightforward.
Provided a labeled data item ( x , y ) , the attacker ﬁnds a perturbation x (cid:48) perceptually similar to x but misleading enough to cause the network to output a label different from y .
By far, the most effective way of ﬁnding such a perturbation (or adversarial example) is by exploiting the gradient information of the
The defense, however, is challenging.
Recent studies showed that adversarial examples always exist if one tends to pursue a high classiﬁcation accuracy—adversarial robustness seems at odds with the accuracy (Tsipras et al., 2018; Shafahi et al., 2019a;
Su et al., 2018a;
Weng et al., 2018; Zhang et al., 2019).
This intrinsic difﬁculty of eliminating adversarial examples suggests an alternative path: can we design a network whose adversarial examples are evasive rather than eliminated?
Indeed, along with this thought is a series of works using obfuscated gradients as a defense mechanism (Athalye et al., 2018).
Those methods hide the network’s gradient information by artiﬁcially discretizing the input (Buckman et al., 2018;
Lin et al., 2019) or introducing certain randomness to the input (Xie et al., 2018a;
Guo et al., 2018) or the network structure (Dhillon et al., 2018;
Cohen et al., 2019) (see more discussion in Sec. 1.1).
Yet, the hidden gradient in those methods can still be approximated, and as recently pointed out by Athalye et al.
(2018), those methods remain vulnerable.
Technical contribution I)
Rather than obfuscating the network’s gradient, we make the gradient undeﬁned .
This is achieved by a simple change to the standard neural network structure: we advocate the use of a C 0 discontinuous activation function, namely the k -Winners-Take-All ( k -WTA) activation, to replace the popular activation functions such as rectiﬁed linear units (ReLU).
This is the only change we propose to a deep neural network.
All other components (such as BatchNorm, convolution, and pooling) as well as the training methods remain unaltered.
With no signiﬁcant overhead, k -WTA activation can be readily used in nearly all existing networks and training methods.
k -WTA activation takes as input the entire output of a layer, retains its k largest values and deactivates all others to zero.
As we will show in this paper, even an inﬁnitesimal perturbation to the input may cause a complete change to the network neurons’ activation pattern, thereby resulting in a large jump in the network’s output.
This means that, mathematically, if we use f ( x ; w ) to denote a k -WTA network taking an input x and parameterized by weights w , then the gradient ∇ x f ( x ; w ) at certain x is undeﬁned— f ( x ; w ) is C 0 discontinuous .
Technical contribution II) More intriguing than the mere replacement of the activation function is why k -WTA helps improve the adversarial robustness .
We offer our theoretical reasoning of its behavior from two perspectives.
On the one hand, we show that the discontinuities of f ( x ; w ) is densely distributed in the space of x .
Dense enough such that a tiny perturbation from any x almost always comes across some discontinuities, where the gradients are undeﬁned and thus the attacker’s search of adversarial examples becomes blinded (see Figure 1).
On the other hand, a paradox seemingly exists.
The discontinuities in the activation function also renders f ( x ; w ) discontinuous with respect to the network weights w (at certain w values).
But training the network relies on the presumption that the gradient with respect to the weights is almost always available.
Interestingly, we show that, under k -WTA activation, the discontinuities of f ( x ; w ) is rather sparse in the space of w , intuitively because the dimension of w (in parameter space) is much larger than the dimension of x (in data space).
Thus, the network can be trained successfully.
Summary of results.
We conducted extensive experiments on multiple datasets under different network architectures, including ResNet (He et al., 2016), DenseNet (Huang et al., 2017), and Wide
ResNet (Zagoruyko & Komodakis, 2016), that are optimized by regular training as well as various adversarial training methods (Madry et al., 2017; Zhang et al., 2019; Shafahi et al., 2019b).
In all these setups, we compare the robustness performance of using the proposed k -WTA activation with commonly used ReLU activation under several white-box attacks, including PGD (Kurakin et al., 2016), Deepfool (Moosavi-Dezfooli et al., 2016), C&W (Carlini & Wagner, 2017), MIM (Dong et al., 2018), and others.
In all tests, k -WTA networks outperform ReLU networks.
The use of k -WTA activation is motivated for defending against gradient-based adversarial attacks.
Our experiments suggest that the robustness improvement gained by simply switching to k -WTA activation is universal, not tied to speciﬁc network architectures or training methods.
To promote reproducible research, we will release our implementation of k -WTA networks, along with our experiment code, conﬁguration ﬁles and pre-trained models 1 .
1. R ELATED W ORK : O BFUSCATED G RADIENTS AS A D EFENSE M ECHANISM
Before delving into k -WTA details, we review prior adversarial defense methods that share the same philosophy with our method and highlight our advantages.
For a review of other attack and defense methods, we refer to Appendix A
Methods aiming for concealing the gradient information from the attacker has been termed as obfuscated gradients (Athalye et al., 2018) or gradient masking (Papernot et al., 2017; Tramèr et al., 2017) techniques.
One type of such methods is by exploiting randomness, either randomly transforming the input before feeding it to the network (Xie et al., 2018a; Guo et al., 2018) or introducing stochastic layers in the network (Dhillon et al., 2018).
However, the gradient information in these methods can be estimated by taking the average over multiple trials (Athalye et al., 2018; 2017).
As a result, these methods are vulnerable.
Another type of obfuscated gradient methods relies on the so-called shattered gradient (Athalye et al., 2018), which aims to make the network gradients nonexistent or incorrect to the attacker, by purposely discretizing the input (Buckman et al., 2018;
Ma et al., 2018) or artiﬁcially raising numerical instability for gradient evaluation (Song et al., 2018; Samangouei et al., 2018).
Unfortunately, these methods are also vulnerable.
As shown by Athalye et al.
, they can be compromised by backward pass differentiable approximation (BPDA).
Suppose f i ( x ) is a non-differentiable component of a network expressed by f = f 1 ◦ f 2 ◦ · · · ◦ f n .
The gradient ∇ x f can be estimated as long as one can ﬁnd a smooth delegate g that approximates well f i (i.e., g ( x ) ≈
f i ( x ) ).
[Figure]
In stark contrast to all those methods, a slight change of the k -WTA activation pattern in an earlier layer of a network can cause a radical reorganization of activation patterns in later layers (as shown in Sec. 3).
Thereby, k -WTA activation not just obfuscates the network’s gradients but destroys them at certain input samples, introducing discontinuities densely distributed in the input data space.
We are not aware of any possible smooth approximation of a k -WTA network to launch BPDA attacks.
k -W INNERS -T AKE -A LL A CTIVATION
The debut of the Winner-Takes-All (WTA) activation on the stage of neural networks dates back to 1980s, when Grossberg (1982) introduced shunting short-term memory equations in on-center off-surround networks and showed the ability to identify the largest of N real numbers.
Later, Majani et al.
(1989) generalized the WTA network to identify the K largest of N real numbers, and they termed the network as the K-Winners-Take-All (KWTA) network.
These early WTA-type activation functions output only boolean values, mainly motivated by the properties of biological neural circuits.
In particular, Maass (2000a;b) has proved that any boolean function can be computed by a single
KWTA unit.
Yet, the boolean nature of these activation functions differs starkly from the modern activation functions, including the one we use.
2. D EEP N EURAL N ETWORKS A CTIVATED BY k -W INNERS -T AKE -A LL
We propose to use k -Winners-Take-All ( k -WTA) activation, a natural generalization of the boolean
KWTA 2 (Majani et al., 1989).
k -WTA retains the k largest values of an N × 1 input vector and sets all others to be zero before feeding the vector to the next network layer, namely,
[Equation]
Here φ k : R N → R N is the k -WTA function (parameterized by an integer k ),
y ∈ R N is the input to the activation, and φ k ( y ) j denote the j -the element of the output φ k ( y )
(see the rightmost subﬁgure of Figure 2).
Note that if y has multiple elements that are equally k -th largest, we break the tie by retaining the element with smaller indices until the k slots are taken.
When using k -WTA activation, we need to choose k .
Yet it makes no sense to ﬁx k throughout all layers of the neural network, because these layers often have different output dimensions; a small k to one layer’s dimension can be relatively large to the other.
Instead of specifying k , we introduce a parameter γ ∈ (0 , 1) called sparsity ratio .
If a layer has an output dimension N , then its k -WTA activation has k = (cid:98) γ · N (cid:99) .
Even though the sparsity ratio can be set differently for different layers, in practice we found no clear gain from introducing such a variation.
Therefore, we use a ﬁxed γ —the only additional hyperparameter needed for the neural network.
In convolutional neural networks (CNN), the output of a layer is a C × H × W tensor.
C denotes the number of output channels; H and W indicate the feature resolution.
While there are multiple choices of applying k -WTA on the tensor—for example, one can apply k -WTA individually to each channel—empirically we found that the most effective (and conceptually the simplest) way is to treat the tensor as a long C · H · W × 1 vector input to the k -WTA activation.
Using k -WTA in this way is also backed by our theoretical understanding (see Sec. 3).
The runtime cost of computing a k -WTA activation is asymptotically O ( N ) , because ﬁnding k largest values in a list is asymptotically equivalent to ﬁnding the k -th largest value, which has an O ( N ) complexity (Cormen et al., 2009).
This cost is comparable to ReLU’s O ( N ) cost on a N -length vector.
Thus, replacing ReLU with k -WTA introduces no signiﬁcant overhead.
Remark: other WTA-type activations.
Relevant to k -WTA is the local Winner-Take-All (LWTA) activation (Srivastava et al., 2013; 2014), which divides each layer’s output values into local groups and applies WTA to each group individually.
LWTA is similar to max-pooling (Riesenhuber &
Poggio, 1999) for dividing the layer output and choosing group maximums.
But unlike ReLU and max-pooling being C 0 continuous, LWTA and our k -WTA are both discontinuous with respect to the input.
The differences among ReLU, max-pooling, LWTA, and k -WTA are illusrated in Figure 2.
LWTA is motivated toward preventing catastrophic forgetting (McCloskey & Cohen, 1989), whereas our use of k
-WTA is for defending against adversarial threat.
Both are discontinuous.
But it remains unclear what the LWTA’s discontinuity properties are and how its discontinuities affect the network training.
Our theoretical analysis (Sec. 3), in contrast, sheds some light on these fundamental questions about k -WTA, rationalizing its ability for improving adversarial robustness.
Indeed, our experiments conﬁrm that k -WTA outperforms LWTA in terms of robustness (see Appendix D.3).
WTA-type activation, albeit originated decades ago and widely studied in computational neuroscience (Douglas et al., 1989; Douglas & Martin, 2004), remains elusive in modern neural networks.
Perhaps this is because it has not demonstrated a considerable improvement to the network’s standard test accuracy, though it can offer an accuracy comparable to ReLU (Srivastava et al., 2013).
Our analysis and proposed use of k -WTA and its enabled improvement on adversarial defense may suggest a renaissance of studying WTA.
2. T RAINING k -WTA N ETWORKS
k -WTA networks require no special treatment in training.
Any optimization algorithm (such as stochastic gradient descent) for training ReLU networks can be directly used to train k -WTA networks.
Our experiments have found that when the sparsity ratio γ is relatively small ( ≤ 0 . 2 ), the network training converges slowly.
This is not a surprise.
A smaller γ activates fewer neurons, effectively reducing more of the layer width and in turn the network size, and the stripped “subnetwork” is much less expressive (Srivastava et al., 2013).
Since different training examples activate different subnetworks, collectively they make the training harder.
Nevertheless, we prefer a smaller γ .
As we will discuss in the next section, a smaller γ usually leads to better robustness against ﬁnding adversarial examples.
Therefore, to ease the training (when γ is small), we propose to use an iterative ﬁne-tuning approach.
Suppose the target sparsity ratio is γ 1 .
We ﬁrst train the network with a larger sparsity ratio γ 0 using the standard training process.
Then, we iteratively ﬁne tune the network.
In each iteration, we reduce its sparsity ratio by a small δ and train the network for two epochs.
The iteration repeats until γ 0 is reduced to γ 1 .
This incremental process introduces little training overhead, because the cost of each ﬁne tuning is negligible in comparison to training from scratch toward γ 0 .
We also note that this process is optional.
In practice we use it only when γ < 0 .
We show more experiments on the efﬁcacy of the incremental training in Appendix D.2.
U NDERSTANDING k -WTA D ISCONTINUITY
We now present our theoretical understanding of k -WTA’s discontinuity behavior in the context of deep neural networks, revealing some implication toward the network’s adversarial robustness.
Activation pattern.
To understand k -WTA’s discontinuity, consider one layer outputting values x , passed through a k -WTA activation, and followed by the next layer whose linear weight matrix is W (see adjacent ﬁgure).
Then, the value fed into the next activation can be expressed as W φ k ( x ) , where [Figure] φ k ( · ) is the k -WTA function deﬁned in (1).
Suppose the vector x has a length l .
We deﬁne the k -WTA’s activation pattern under the input x as
[Equation]
is one of the k largest values in
[Equation]
[Equation]
Here (and throughout this paper), we use [ l ] to denote the integer set { 1 , 2 , ..., l } .
Discontinuity.
The activation pattern A ( x ) is a key notion for analyzing k -WTA’s discontinuity behavior.
Even an inﬁnitesimal perturbation of x may change A ( x ) : some element i is removed from A ( x ) while another element j is added in.
Corresponding to this change, in the evaluation of
W φ k ( x ) , the contribution of W ’s column vector W i vanishes while another column W j suddenly takes effect.
It is this abrupt change that renders the result of W φ k ( x ) C 0 discontinuous.
Such a discontinuity jump can be arbitrarily large, because the column vectors W i and W j can be of any difference.
Once W is determined, the discontinuity jump then depends on the value of x i and x j .
As explained in Appendix B, when the discontinuity occurs, x i and x j have about the same value, depending on the choice of the sparsity ratio γ (recall Sec.
2.1)—the smaller the γ is, the larger the jump will be.
This relationship suggests that a smaller γ will make the search of adversarial examples harder.
Indeed, this is conﬁrmed through our experiments (see Appendix D.6).
Piecewise linearity.
Now, consider an n -layer k -WTA network, which can be expressed as f ( x ) =
W (1) · φ k ( W (2) · φ k ( · · · φ k ( W ( n ) x + b ( n ) ))
+ b (2) )
+ b (1) , where W ( i ) and b ( i ) are the i -th layer’s weight matrix and bias vector, respectively.
If the activation patterns of all layers are ﬁxed, then f ( x ) is a linear function.
When the activation pattern changes, f ( x ) switches from one linear function to another linear function.
Over the entire space of x , f ( x ) is piecewise linear.
The speciﬁc activation patterns of all layers deﬁne a speciﬁc linear piece of the function, or a linear region (following the notion introduced by Montufar et al.
Conventional ReLU (or hard tanh) networks also represent piecewise linear functions and their linear regions are joined together at their boundaries, whereas in k -WTA networks the linear regions are disconnected (see Figure 1).
Linear region density.
Next, we gain some insight on the distribution of those linear regions.
This is of our interest because if the linear regions are densely distributed, a small ∆ x perturbation from any data point x will likely cross the boundary of the linear region where x locates.
Whenever boundary crossing occurs, the gradient becomes undeﬁned (see Figure 3-a).
For the purpose of analysis, consider an input x passing through a layer followed by a k -WTA activation (see adjacent ﬁgure).
The output from the activation is φ k ( W x + b ) .
We would like to understand, when x is changed into x (cid:48) , how likely the activation pattern of φ k will change.
First, notice that if x (cid:48) and x satisfy x (cid:48) =
c · x with some c > 0 , the activation pattern remains unchanged.
Therefore, we introduce a notation d ( x , x (cid:48) ) that measures the “perpendicular” distance between x and x (cid:48) , one that satisﬁes x (cid:48)
=
c · ( x + d ( x , x (cid:48) ) x ⊥ ) for some scalar c , where x ⊥ is a unit vector perpendicular to x and on the plane spanned by x and x (cid:48) .
With this notion, and if the elements in W is initialized by sampling from N (0 , 1 l ) and b is initialized as zero, we ﬁnd the following property:
Theorem 1 (Dense discontinuities) .
Given any input x ∈ R m and some β , and
∀ x (cid:48) ∈
R m such that
[Equation]
≥ β , if the following condition l ≥ Ω (cid:18)(cid:18)
m γ · β (cid:19) · log (cid:18) m γ · β (cid:19)(cid:19) is satisﬁed, then with a probability at least 1 − · 2 − m , we have A ( W x + b ) (cid:54) =
A ( W x (cid:48)
+ b ) .
Here l is the width of the layer, and γ is again the sparsity ratio in k -WTA.
This theorem informs us that the larger the layer width l is, the smaller β —and thus the smaller perpendicular perturbation distance d ( x , x (cid:48) ) —is needed to trigger a change of the activation pattern, that is, as the layer width increases, the piecewise linear regions become ﬁner (see Appendix C for proof and more discussion).
This property also echos a similar trend in ReLU networks, as pointed out by Raghu et al.
Why is the k -WTA network trainable?
While k -WTA networks are highly discontinuous as revealed by Theorem 1 and our experiments (Figure 3-a), in practice we experience no difﬁculty on training these networks.
Our next theorem sheds some light on the reason behind the training success.
Theorem 2.
Consider N data points x 1 , x 2 , · · · , x N ∈ R m .
Suppose ∀ i (cid:54) =
j, x i (cid:107) x i (cid:107) 2 (cid:54) =
x j (cid:107) x j (cid:107) 2 .
If l is sufﬁciently large, then with a high probability, we have ∀ i (cid:54)
=
j, A ( W x i + b ) ∩ A ( W x j + b )
= ∅ .
This theorem is more formally stated in Theorem 10 in Appendix C together with a proof there.
Intuitively speaking, it states that if the network is sufﬁciently wide, then for any i (cid:54) =
j , activation pattern of input data x i is almost separated from that of x j .
Thus, the weights for predicting x i ’s and x j ’s labels can be optimized almost independently, without changing their individual activation patterns.
In practice, the activation patterns of x i and x j are not fully separated but weakly correlated.
During the optimization, the activation pattern of a data point x i may change, but the chance is relatively low—a similar behavior has also been found in ReLU networks (Li & Liang, 2018; Du et al., 2018; Allen-Zhu et al., 2019a;b; Song & Yang, 2019).
Further, notice that the training loss takes a summation over all training data points.
This means a weight update would change only a small set of activation patterns (since the chance of having the pattern changed is low); the discontinuous change on the loss value, after taking the summation, will be negligible (see Figure 3-c).
Thus, the discontinuities in k -WTA is not harmful to network training.
E XPERIMENTAL R ESULTS
We evaluate the robustness of k -WTA networks under adversarial attacks.
Our evaluation considers multiple training methods on different network architectures (see details below).
When reporting statistics, we use A rob to indicate the worst-case robustness accuracy on test data under all adversarial attacks we evaluated, and use A std to indicate the accuracy on the clean test data.
We use k -WTA- γ to represent k -WTA activation with sparsity ratio γ .
4. R OBUSTNESS UNDER W HITE - BOX A TTACKS
The rationale behind k -WTA activation is to destroy network gradients—information needed in white-box attacks.
We therefore evaluate k -WTA networks under multiple recently proposed whitebox attack methods, including Projected Gradient Descent (PGD) (Madry et al., 2017), Deepfool (Moosavi-Dezfooli et al., 2016), C&W attack (Carlini & Wagner, 2017), and Momentum Iterative
Method (MIM) (Dong et al., 2018).
Since k -WTA activation can be used in almost any training method, be it adversarial training or not, we also consider multiple training methods, including natural (non-adversarial) training, adversarial training (AT) (Madry et al., 2017), TRADES (Zhang et al., 2019) and free adversarial training (FAT) (Shafahi et al., 2019b).
In addition, we evaluate the robustness under transfer-based Black-box (BB) attacks (Papernot et al., 2017).
The black-box threat model requires no knowledge about network architecture and parameters.
Thus, we use a pre-trained VGG19 network (Simonyan & Zisserman, 2014) as the source model to generate adversarial examples using PGD.
As demonstrated by Su et al.
(2018b), VGG networks have the strongest transferability among different architectures.
In each setup, we compare the robust accuracy of k -WTA networks with standard ReLU networks on three datasets, CIFAR-10, SVHN, and MNIST.
Results on the former two are reported in Table 1, while the latter is reported in Appendix D.4.
We use ResNet-18 for CIFAR-10 and SVHN.
The perturbation range is 0.031 (CIFAR-10) and 0.047 (SVHN) for pixels ranging in [0 , 1] .
More detailed training and attacking settings are reported in Appendix D.1.
[Table]
The main takeaway from these experiments (in Table 1) is that k -WTA is able to universally improve the white-box robustness, regardless of the training methods.
The k -WTA robustness under black-box attacks is not always signiﬁcantly better than ReLU networks.
But black-box attacks, due to the lack of network information, are generally much harder than white-box attacks.
In this sense, white-box attacks make the networks more vulnerable, and k -WTA is able to improve a network’s worst-case robustness.
This improvement is not tied to any speciﬁc training method, achieved with no signiﬁcant overhead, just by a simple replacement of ReLU with k -WTA.
Athalye et al.
(2018) showed that gradient-based defenses may render the network more vulnerable under black-box attacks than under gradient-based white-box attacks.
However, we have not observed this behavior in k -WTA networks.
Even under the strongest black-box attack, i.e., by generating adversarial examples from an independently trained copy of the target network, gradient-based attacks are still stronger (with higher successful rate) than black-box attacks (see Appendix D.3).
Additional experiments include: 1) tests under transfer attacks across two independently trained k -WTA networks and across k -WTA and ReLU networks, 2) evaluation of k -WTA performance on different network architectures, and 3) comparison of k -WTA performance with the LWTA (recall
Sec. 2.1) performance.
See Appendix D.3 for details.
4. V ARYING SPARSITY RATIO γ AND MODEL ARCHITECTURE .
We further evaluate our method on various network architectures with different sparsity ratios γ .
Figure 4 shows the standard test accuracies and robust accuracies against PGD attacks while γ decreases.
To test on different network architectures, we apply k -WTA to ResNet18, DenseNet and Wide ResNet (WRN-22-12).
In each case, starting from γ = 0 .
2 , we decrease γ using incremental ﬁne-tuning.
We then evaluate the robust accuracy on CIFAR dataset, taking 20-iteration PGD attacks with a perturbation range (cid:15) = 0 .
31 for pixels ranging in [0 , 1] .
[Figure]
We ﬁnd that when γ is larger than ∼ 0 . 1 , reducing γ has little effect on the standard accuracy, but increases the robust accuracy.
When γ is smaller than ∼ 0 .
1 , reducing γ drastically lowers both the standard and robust accuracies.
The peaks in the A rob curves (Figure 4-right) are consistent with our theoretical understanding: Theorem 1 suggests that when l is ﬁxed, a smaller γ tends to sparsify the linear region boundaries, exposing more gradients to the attacker.
Meanwhile, as also discussed in
Sec. 3, a smaller γ leads to a larger discontinuity jump and thus tends to improve the robustness.
4. L OSS L ANDSCAPE IN G RADIENT -B ASED A TTACKS
We now empirically unveil why k -WTA is able to improve the network’s robustness (in addition to our theoretical analysis in Sec. 3).
Here we visualize the attacker’s loss landscape in gradient-based attacks in order to reveal the landscape change caused by k -WTA.
Similar to the analysis in Tramèr et al.
(2017), we plot the attack loss of a model with respect to its input on points x (cid:48)
=
x + (cid:15) 1 g 1 + (cid:15) 2 g 2 , where x is a test sample from CIFAR test set, g 1 is the direction of the loss gradient with respect to the input, g 2 is another random direction, (cid:15) 1 and (cid:15) 2 sweep in the range of [ − 0 .
04 , 0 . 04] , withsamples each.
This results in a 3D landscape plot with 2500 data points (Figure 5).
As shown in Figure 5, k -WTA models (with γ = 0 . 1 ) have a highly non-convex and non-smooth loss landscape.
Thus, the estimated gradient is hardly useful for adversarial searches.
This explains why k -WTA models can effectively resist gradient-based attacks.
In contrast, ReLU models have a much smoother loss surface, from which adversarial examples can be easily found using gradient descent.
Inspecting the range of loss values in Figure 5, we ﬁnd that adversarial training tends to compress the loss landscape’s dynamic range in both the gradient direction and the other random direction, making the dynamic range smaller than that of the models without adversarial training.
This phenomenon has been observed in ReLU networks (Madry et al., 2017; Tramèr et al., 2017).
Interestingly, k -WTA models manifest a similar behavior (Figure 5-a,b).
Moreover, we ﬁnd that in k -WTA models a larger γ leads to a smoother loss surface than a smaller γ (see Appendix D.6 for more details).
C ONCLUSION
This paper proposes to replace widely used activation functions with the k -WTA activation for improving the neural network’s robustness against adversarial attacks.
This is the only change we advocate.
The underlying idea is to embrace the discontinuities introduced by k -WTA functions to make the search for adversarial examples more challenging.
Our method comes almost for free, harmless to network training, and readily useful in the current paradigm of neural networks.
Document Enhancing Adversarial Defense by k -Winners-Take-All A O THER R ELATED W ORK
In this section, we brieﬂy review the key ideas of attacking neural network models and existing defense methods based on adversarial training.
Attack methods.
Recent years have seen adversarial attack studied extensively.
The proposed attack methods fall under two general categories, white-box and black-box attacks.
The white-box threat model assumes that the attacker knows the model’s structure and parameters fully.
This means that the attacker can exploit the model’s gradient (with respect to the input) to ﬁnd adversarial examples.
A baseline of such attacks is the Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2014), which constructs the adversarial example x (cid:48) of a given labeled data ( x , y ) using a gradient-based rule:
[Equation]
where f ( x ) denotes the neural network model’s output, L ( · ) is the loss function provided f ( x ) and input label y , and (cid:15) is the perturbation range for the allowed adversarial example.
Extending FGSM, Projected Gradient Descent (PGD) (Kurakin et al., 2016) utilizes local ﬁrstorder gradient of the network in a multi-step fashion, and is considered the “strongest” ﬁrst-order adversary (Madry et al., 2017).
In each step of PGD, the adversarial example is updated by a FGSM rule, namely,
[Equation]
(4) where x (cid:48) n is the adversarial examples after n steps and Π x ∈ ∆ (cid:15) ( x (cid:48) n ) projects x (cid:48)
n back into an allowed perturbation range ∆ (cid:15) (such as an (cid:15) ball of x under certain distance measure).
Other attacks include
Deepfool (Moosavi-Dezfooli et al., 2016), C&W (Carlini & Wagner, 2017) and momentum-based attack (Dong et al., 2018).
Those methods are all using ﬁrst-order gradient information to construct adversarial samples.
The black-box threat model is a strict subset of the white-box threat model.
It assumes that the attacker has no information about the model’s architecture or parameters.
Some black-box attack model allows the attacker to query the victim neural network to gather (or reverse-engineer) information.
By far the most successful black-box attack is transfer attack (Papernot et al., 2017; Tramèr et al., 2017).
The idea is to ﬁrst construct adversarial examples on an adversarially trained network and then attack the black-box network model use these samples.
There also exist some gradient-free black-box attack methods, such as boundary attack (Brendel et al., 2017; Chen & Jordan, 2019), one-pixel attack (Su et al., 2019) and local search attack (Narodytska & Kasiviswanathan, 2016).
Those methods rely on repeatedly evaluating the model and are not as effective as gradient-based white-box attacks.
Adversarial training.
Adversarial training (Goodfellow et al., 2014; Madry et al., 2017; Kurakin et al., 2016; Huang et al., 2015) is by far the most successful method against adversarial attacks.
It trains the network model with adversarial images generated during the training time.
Madry et al.
(2017) showed that adversarial training in essence solves the following min-max optimization problem:
[Equation]
(5) where ∆ (cid:15) is the set of allowed perturbations of training samples, and y denotes the true label of each training sample.
Recent works that achieve state-of-the-art adversarial robustness rely on adversarial training (Zhang et al., 2019;
Xie et al., 2018b).
However, adversarial training is notoriously slow because it requires ﬁnding adversarial samples on-the-ﬂy at each training epoch.
Its prohibitive cost makes adversarial training difﬁcult to scale to large datasets such as ImageNet (Deng et al., 2009) unless enormous computation resources are available.
Recently, Shafahi et al.
(2019b) revise the adversarial training algorithm to make it has similar training time as regular training, while keep the standard and robust accuracy comparable to standard adversarial training.
Regularization.
Another type of defense is based on regularizing the neural network, and many works of this type are combined with adversarial training.
For example, feature denoising (Xie et al., 2018b) adds several denoise blocks to the network structure and trains the network with adversarial training.
Zhang et al.
(Zhang et al., 2019) explicitly added a regularization term to balance the trade-off between standard accuracy and robustness, obtaining state-of-the-art robust accuracy on
CIFAR.
Some other regularization-based methods require no adversarial training.
For example, Ross & DoshiVelez (2017) proposed to regularize the gradient of the model with respect to its input; Zheng et al.
(2016) generated adversarial samples by adding random Gaussian noise to input data.
However, these methods are shown to be brittle under stronger iterative gradient-based attacks such as PGD (Zhang et al., 2019).
In contrast, as demonstrated in our experiments, our method without using adversarial training is able to greatly improve robustness under PGD and other attacks.
B D ISCONTINUITY J UMP OF W φ k ( x )
Consider a gradual and smooth change of the vector x .
For the ease of illustration, let us assume all the values in x are distinct.
Because every element in x changes smoothly, when the activation pattern A ( x ) changes, the k -th largest and k + 1 -th largest value in x must swap: the previously k -th largest value is removed from the activation pattern, while the previously k + 1 -th largest value is added in the activation pattern.
Let i and j denote the indices of the two values, that is, x i is previously the k -th largest and x j is previously the k + 1 -th largest.
When this swap happens, x i and x j must be inﬁnitesimally close to each other, and we use x ∗ to indicate their common value.
This swap affects the computation of W φ k ( x ) .
Before the swap happens, x i is in the activation pattern but x j is not, therefore W i takes effect but W j does not.
After the swap, W j takes effect while
W j is suppressed.
Therefore, the discontinuity jump due to this swap is ( W j − W i ) x ∗ .
When W is determined, the magnitude of the jump depends on x ∗ .
Recall that x
∗
is the k -th largest value in x when the swap happens.
Thus, it depends on k and in turn the sparsity ratio γ : the smaller the γ is, the smaller k is effectively used (for a ﬁxed vector length).
As a result, the k -th largest value becomes larger—when k = 1 , the largest value of x is used as x ∗ .
C T HEORETICAL P ROOFS
In this section, we will prove Theorem 1 and Theorem 2.
The formal version of the two theorems are
Theorem 9 and Theorem 10 respectively.
Notation.
We use [ n ] to denote the set { 1 , 2 , · · · , n } .
We use 1 ( E ) to indicate an indicator variable.
If the event E happens, the value of 1 ( E ) is 1 .
Otherwise the value of 1 ( E ) is 0 .
For a weight matrix
W , we use W i to denote the i -th row of W .
For a bias vector b , we use b i to denote the i -th entry of b .
In this section, we show some behaviors of the k -WTA activation function.
Recall that an n -layer neural network f ( x ) with k -WTA activation function can be seen as the following:
[Equation]
where W ( i ) is the weight matrix, b ( i ) is the bias vector of the i -th layer, and φ ( · ) is the k -WTA activation function, i.e., for an arbitrary vector y , φ k ( y ) is deﬁned as the following:
[Equation]
one of the top- k largest values,
[Equation]
otherwise.
For simplicity of the notation, if k is clear in the context, we will just use φ ( y ) for short.
Notice that if there is a tie in the above deﬁnition, we assume the entry with smaller index has larger value.
For a vector y ∈ R l , we deﬁne the activation pattern A ( y ) ⊆
[ l ] as
A ( y ) =
{ i ∈ [ l ] |
y
i is one of the top- k largest values } .
Notice that if the activation pattern A ( y ) is different from A ( y (cid:48) ) , then W · φ ( y ) and W · φ ( y (cid:48) ) will be in different linear region.
Actually, W · φ ( y ) may even represent a discontinuous function.
In the next section, we will show that when the network is much wider, the function may be more discontinuous with respect to the input.
C. D ISCONTINUITY WITH R ESPECT TO THE I NPUT
We only consider the activation pattern of the output of one layer.
We consider the behavior of the network after the initialization of the weight matrix and the bias vector.
By initialization, the entries of the weight matrix W are i.i.d.
random Gaussian variables, and the bias vector is zero.
We can show that if the weight matrix is sufﬁciently wide, then for any vector x , with high probability, for all vector x (cid:48) satisfying that the "perpendicular” distance between x and x (cid:48) is larger than a small threshold, the activation patterns of Wx and Wx (cid:48) are different.
Notice that the scaling of W does not change the activation pattern of Wx for any x , we can thus assume that each entry of W is a random variable with standard Gaussian distribution N (0 , 1) .
Before we prove Theorem 9, let us prove several useful lemmas.
The following several lemmas does not depend on the randomness of the weight matrix.
Lemma 1 (Inputs with the same activation pattern form a convex set) .
Given an arbitrary weight matrix W ∈ R l × m and an arbitrary bias vector b ∈ R l , for any x ∈ R m , the set of all the vectors x (
cid:48) ∈
R m satisfying A ( Wx (cid:48) + b ) =
A ( Wx + b ) is convex, i.e., the set
[Equation]
is convex.
Proof.
If A ( Wx (cid:48) + b ) =
A ( Wx + b ) , then x (cid:48) should satisfy:
[Equation]
Notice that the inequality W i x (cid:48)
+ b i ≥ ( or > ) W j x (cid:48)
+ b j denotes a half hyperplane
( W i − W j ) x (cid:48)
+ ( b i − b j ) ≥ ( or > ) 0 .
Thus, the set { x (cid:48) ∈ R m | A ( Wx + b )
=
A ( Wx (cid:48) + b ) } is convex since it is an intersection of half hyperplanes.
Lemma 2 (Different patterns of input points with small angle imply different patterns of input points with large angle) .
Let α ∈ (0 , 1) .
Given an arbitrary weight matrix W ∈ R l × m , a bias vector b = 0 , and a vector x ∈ R m with (cid:107) x (cid:107) 2 = 1 , if every vector x (cid:
) ∈
R m with (cid:107) x (cid:48) (cid:107) 2 = 1 and (cid:104) x, x (cid:48) (cid:105) =
α satisﬁes A ( Wx + b ) (cid:54) =
A ( Wx (cid:48) + b ) , then for any x (cid:48)(cid:48) ∈
R m with (cid:107) x
(cid:48)(cid:48) (cid:107) 2 = 1 and (cid:104) x, x (cid:48)(cid:48) (cid:105) <
α , it satisﬁes A ( Wx + b ) (cid:54) =
A ( Wx (cid:48)(cid:48) + b ) .
Proof.
We draw a line between x and x (cid:48)(cid:48) .
There must be a point x ∗ ∈ R m on the line and (cid:104) x, x (cid:48) (cid:105) =
α , where x (cid:48)
= x ∗ / (cid:107) x ∗ (cid:107) 2 .
Since b = 0 , we have A ( Wx ∗ + b ) =
A ( Wx (cid:48)
+ b ) (cid:54) =
A ( Wx + b ) .
Since x ∗ is on the line between x and x (cid:48)(cid:48) , we have A ( Wx (cid:48)(cid:48)
+ b ) (cid:54) =
A ( Wx + b ) by convexity (see Lemma 1).
Lemma 3 (A sufﬁcient condition for different patterns) .
Consider two vectors y ∈ R l and y (cid:48)
∈ R l .
If ∃ i ∈ A ( y ) , j ∈ [ l ] \ A ( y )
such that y (cid:48)
i < y (cid:48)
j , then A ( y ) (cid:54) =
A ( y (cid:48) ) .
Proof.
Suppose A ( y )
=
A ( y (cid:48) ) .
We have i ∈
A ( y (cid:48) ) .
It means that y (cid:48)
i is one of the top- k largest values among all entries of y (cid:48) .
Thus y (cid:48)
j is also one of the top- k largest values, and j should be in
A ( y (cid:48) ) which leads to a contradiction.
In the remaining parts, we will assume that each entry of the weight matrix W ∈ R l × m is a standard random Gaussian variable.
Lemma 4 (Upper bound of the entires of W ) .
Consider a matrix W ∈ R l × m where each entry is a random variable with standard Gaussian distribution N (0 , 1) .
With probability at least 0 .
99 , ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ ml .
Proof.
Consider a ﬁxed i ∈
[ l ] .
We have E [ (cid:107) W i (cid:107) 22 ] = m .
By Markov’s inequality, we have
Pr[ (cid:107) W i (cid:107) 22 > 100 ml ] ≤ 0 .
/l .
By taking union bound over all i ∈
[ l ] , with probability at least 0 .
99 , we have ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ ml .
Lemma 5 (Two vectors may have different activation patterns with a good probability) .
Consider a matrix W ∈ R l × m where each entry is a random variable with standard Gaussian distribution
N (0 , 1) .
Let γ ∈ (0 , 0 . 48) be the sparsity ratio of the activation, i.e., γ = k/l .
For any two vectors x, x (cid:48) ∈ R m with (cid:107) x (cid:107)
=
(cid:107) x (cid:48) (cid:107) 2 = 1 and (cid:104) x, x (cid:48) (cid:105) =
α for some arbitrary α ∈ (0 . 5 , 1) , with probability at least 1 − 2 − Θ((1 /α 2 − 1) γl ) , A ( Wx ) (cid:54) =
A ( Wx (cid:48) ) and ∃
i ∈
A ( Wx ) , j ∈ [ l ] \ A ( Wx )
such that
[Equation]
Proof.
Consider arbitrary two vectors x, x
(cid:48) ∈
R m with (cid:107) x (cid:107)
= (cid:107) x (cid:48) (cid:107) 2 = 1 and (cid:104) x, x (cid:48) (cid:105) =
α .
We can ﬁnd an orthogonal matrix Q ∈ R m × m such that ˜ x :=
Qx = (1 , 0 , 0 , · · · , 0) (cid:62) ∈ R m and ˜ x (cid:48) :
=
Qx (cid:48)
= ( α, √ 1 − α 2 , 0 , 0 , · · · , 0) (cid:62) ∈ R m .
Let ˜ W = WQ (cid:62) .
Then we have ˜ W ˜ x
= Wx and ˜ W ˜ x (cid:48) =
Wx (cid:48) .
Thus, we only need to analyze the activation patterns of ˜ W ˜ x and ˜ W ˜ x (cid:48) .
Since Q (cid:62) is an orthogonal matrix and each entry of W is an i.i.d.
random variable with standard Gaussian distribution N (0 , 1) , ˜ W = WQ (cid:62) is also a random matrix where each entry is an i.i.d.
random variable with standard Gaussian distribution N (0 , 1) .
Let the entries in the ﬁrst column of ˜ W be
X 1 , X 2 , · · · , X l and let the entries in the second column of ˜ W be Y 1 , Y 2 , · · · , Y l .
Then we have
[Equation]
We set ε = √ 1 − α 2 / (96 α ) and deﬁne R (cid:48) 1 < R 1 < R
[Equation]
R
[Equation]
2 as follows:
[Equation]
Since γ < 0 .
48 and ε ≤ 0 .
[Equation]
we have (1 + 2 ε ) γ < 0 .
It implies 0 < R (cid:48)
1 < R 1 < R 2 < R (cid:48) 2 .
Claim 3.
[Equation]
Proof.
By Equation (7) and Equation (10),
[Equation]
Due to the density function of standard Gaussian distribution
[Equation]
we have
[Equation]
Since R (cid:48) 2 ≥ R (cid:48) 1 ≥
[Equation]
we have ∀ t ∈
[ R (cid:48) 1 , R (cid:48) 2 ] , e − t 2 / 2 ≥ e − R (cid:48) 22 / 2 .
Thus,
[Equation]
By the tail bound of Gaussian distribution, we have
[Equation]
By combining with Equation (7), we have
[Equation]
which implies
[Equation]
where the last inequality follows from 1 − 2 ε ≥ 0 .
Claim 4.
[Equation]
Proof.
For i ∈ [ l ] , we have E [ 1 ( X
i ≥ R 2 )]
=
Pr[ X i ≥ R 2 ] =
(1 − ε ) γ by Equation (8).
By
Chernoff bound, we have
[Equation]
Since ε ≤ 0 .
[Equation]
We have E [ 1 ( X i ≥ R 1 )] =
Pr[ X i ≥ R 1 ] =
(1 + ε ) γ by Equation (9).
By Chernoff bound, we have
Pr
[Equation]
Thus,
Pr
[Equation]
We have E [ 1 ( R (cid:48) 2 ≥ X
i ≥ R 2 )]
=
Pr[ R (cid:48) 2 ≥
X
i ≥ R 2 ] = εγ by Equation (7) and Equation (8).
By
Chernoff bound, we have
[Equation]
Similarly, we have E [
1 ( R 1 ≥ X
i ≥ R (cid:48) 1 )]
=
Pr[ R 1 ≥ X
i ≥ R (cid:48) 1 ] = εγ by Equation (9) and
Equation (10).
By chernoff bound, we have
[Equation]
Equation (11) says that, with high probability, ∀ i ∈
[ l ] with X i ≥ R 2 , it has i ∈
A ( Wx ) .
Equation (12) says that, with high probability, ∀ i ∈
[ l ] with X i ≤ R 1 , it has i (cid:54)∈ A ( Wx ) .
Equation (14) (Equation (13)) says that, with high probability, there are many i ∈
[ l ] such that W i x ∈
[ R (cid:48) 1 , R 1 ] ( W i x ∈
[ R 2 , R (cid:48) 2 ] ).
Let E =
E 1 ∧ E 2 ∧ E 3 ∧ E 4 , where
• E 1 : (cid:80) li
=1 1 ( X i ≥ R 2 ) ≤ (1 − ε/ 2) γl , • E 2 : (cid:80) li =1 1 ( X i ≥ R 1 ) ≥ (1 + ε/ 2) γl ,
• E 3 : (cid:80) li
=1 1 ( R 1 ≥ X
i ≥ R (cid:48) 1 ) ≥ εγl/ 2 , • E 4 : (cid:80) li =1 1 ( R (cid:48) 2 ≥ X
i ≥ R 2 ) ≥ εγl/ 2 .
According to Equation (11), Equation (12), Equation (13) and Equation (14), the probability that E happens is at least
[Equation]
(15) by union bound over ¯ E 1 , ¯ E 2 , ¯ E 3 , ¯ E 4 .
Claim 5.
Condition on E , the probability that ∃ i ∈ [ l ] with X i ∈
[ R 2 , R (cid:48) 2 ] such that Y i < − α/ √ 1 − α 2 · 16 ε √ 2 π is at least
[Equation]
Proof.
For a ﬁxed i ∈ [ l ] ,
[Equation]
Thus, according to event E 4 , we have
Pr (cid:104) ∀ i with X
i ∈
[ R 2 , R (cid:48) 2 ] , Y i ≥ − α/ (cid:112) 1 − α 2 · 16 ε √
π
| E (cid:105) ≤ (cid:18) 16 ε ·
[Equation]
1 − α 2 +  (cid:19) εγl/.
Claim 6.
Condition on E , the probability that ∃ i ∈ [ l ] with X i ∈
[ R (cid:48) 1 , R 1 ] such that Y i ≥ 0 is at least 1 − (1 / 2) εγl/ 2 .
Proof.
For a ﬁxed i ∈ [ l ] , Pr[ Y i ≤ 0] = 1 / 2 .
Thus, according to event E 3 , we have
Pr [ ∀ i with X i ∈
[ R (cid:48) 1 , R 1 ] , Y i ≤ 0 | E ] ≤ (1 / 2) εγl/ 2 .
Condition on that E happens.
Because of E 1 , if X i ≥ R 2 , X i must be one of the top- k largest values.
Due to Equation (6), we have X i
=
W i x .
Thus, if X i ≥ R 2 , i ∈
A ( Wx ) .
By Claim 5, with probability at least
[Equation]
there is i ∈
A ( Wx ) such that
[Equation] [Equation]
where the ﬁrst step follows from Equation (6), the second step follows from Y i ≤ − α/ √ 1 − α 2 · 16 ε √ 2 π , and the last step follows from X i ∈
[ R 2 , R (cid:48) 2 ] .
Because of E 2 if X j ≤ R 1 , X j should not be one of the top- k largest values.
Due to Equation (6), we have X j = W j x .
Thus, if X j ≤ R 1 , j (cid:54)∈ A ( Wx ) .
By Claim C.1, with probability at least 1 − (1 /
[Equation] [Equation]
(18) there is j (cid:54)∈ A ( Wx ) such that
[Equation]
[Equation]
where the ﬁrst step follows from Equation (6), the second step follows from Y j ≥ 0 , and the last step follows from X j ∈ [ R (cid:48) 1 , R 1 ] .
By Equation (19) and Equation (17), ∃ i ∈
A ( Wx ) , j ∈ [ l ] \ A ( Wx ) ,
[Equation]
where the second step follows from Claim 3, the forth step follows from α ≥ 0 .
5 , and the last step follows from ε = √ 1 − α 2 / (96 α ) .
By Lemma 3, we can conclude A ( Wx ) (cid:54) =
A ( Wx (cid:48) ) .
By
Equation (15), Equation (16), Equation (18), and union bound, the overall probability is at least
[Equation]
where the ﬁrst and the last step follows from ε = √ 1 − α 2 / (96 α )
Next, we will use a tool called ε -net.
Deﬁnition 7 ( ε -Net) .
For a given set S , if there is a set N ⊆ S such that ∀ x ∈ S there exists a vector y ∈ N such that (cid:107) x − y (cid:107) 2 ≤ ε , then N is an ε -net of S .
There is a standard upper bound of the size of an ε -net of a unit norm ball.
Lemma 6 (Wojtaszczyk (1996) II.E, 10) .
Given a matrix U ∈ R m × d , let S =
{ Uy | (cid:107)
Uy (cid:107) 2 = 1 } .
For ε ∈ (0 , 1) , there is an ε -net N of S with |N| ≤ (1 + 1 /ε )
d .
Now we can extend above lemma to the following.
Lemma 7 ( ε -Net for the set of points with a certain angle) .
Given a vector x ∈ R m with (cid:107) x (cid:107) 2 =and a parameter α ∈ ( − 1 , 1) , let S =
{ x (cid:48) ∈
R m | (cid:107) x (cid:48) (cid:107) 2 = 1 , (cid:104) x, x (cid:48) (cid:105) = α } .
For ε ∈ (0 , 1) , there is an ε -net N of S with |N| ≤ (1 + 1 /ε ) m − 1 .
Proof.
Let U ∈ R m × ( m − 1) have orthonormal columns and Ux = 0 .
Then S can be represented as
[Equation]
Let
[Equation]
According to Lemma 6, there is an ε -net N (cid:48) of S (cid:48) with size |N (cid:48) | ≤ (1 + 1 /ε ) m − 1 .
We construct N as following:
[Equation]
It is obvious that |N| =
|N (cid:48) | ≤ (1 + 1 /ε ) m − 1 .
Next, we will show that N is indeed an ε -net of S .
Let x (cid:48) be an arbitrary vector from S .
Let x (cid:48)
=
α · x + √ 1 − α 2 · z for some z ∈ S (cid:48) .
There is a vector ( α · x + √ 1 − α 2 · z (cid:48) )
∈ N such that z (cid:48) ∈ N (cid:48) and (cid:107)
z − z (cid:48) (cid:107) 2 ≤ ε .
Thus, we have
[Equation]
Theorem 8 (Rotating a vector a little bit may change the activation pattern) .
Consider a weight matrix
W ∈ R l × m where each entry is an i.i.d.
sample drawn from the Gaussian distribution N (0 , 1 /l ) .
Let γ ∈ (0 , 0 . 48) be the sparsity ratio of the activation function, i.e., γ = k/l .
With probability at least 0 .
99 , it has ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ m .
Condition on that ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ m happens, then, for any x ∈ R m and α ∈ (0 . 5 , 1) , if l ≥ C · (cid:18) m
+ log(1 /δ )
γ · 1 − α(cid:19) · log (cid:18) m
+ log(1 /δ )
γ · 1 − α(cid:19) for a sufﬁciently large constant C , with probability at least 1 − δ · 2 − m , ∀ x (cid:48) ∈
R m with (cid:104) x,x (cid:48) (cid:105) (cid:107) x (cid:107) 2 (cid:107) x (cid:48) (cid:107) 2 ≤ α ,
A ( Wx ) (cid:54) =
A ( Wx (cid:48) ) .
Proof.
Notice that the scale of W does not affect the activation pattern of Wx for any x ∈ R m .
Thus, we assume that each entry of W is a standard Gaussian random variable in the remaining proof, and we will instead condition on ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ ml .
The scale of x or x (cid:48) will not affect (cid:104) x,x (cid:48) (cid:105) (cid:107) x (cid:107) 2 (cid:107) x (cid:48) (cid:107) 2 .
It will not affect the activation pattern either.
Thus, we assume (cid:107) x (cid:107)
= (cid:107) x (cid:48) (cid:107) 2 = 1 .
By Lemma 4, with probability at least 0 .
99 , we have ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ ml .
Let
S = { y ∈ R m |
[Equation]
y
[Equation] [Equation]
y
[Equation]
=
[Equation]
Set
[Equation]
By Lemma 7, there is an ε -net N of S such that
[Equation]
By Lemma 5, for any y ∈ N , with probability at least
[Equation]
∃ i ∈
A ( Wx ) , j ∈ [ l ] \ A ( Wx )
such that
[Equation]
By taking union bound over all y ∈ N , with probability at least 1 − |N|
[Equation] [Equation]
γl ) ≥ 1 − (cid:32)
[Equation]
≥ 1 − (cid:32)
[Equation] [Equation]
// C (cid:48) is a sufﬁciently large constant = 1 − (cid:32)
[Equation]
≥ 1 − δ · 2 − m
[Equation]
the following event E (cid:48) happens: ∀ y ∈ N , ∃ i ∈
A ( Wx ) , j ∈ [ l ] \ A ( Wx )
such that
[Equation]
In the remaining of the proof, we will condition on the event E (cid:48) .
Consider y (cid:48) ∈ S .
Since N is an ε -net of S , we can always ﬁnd a y ∈ N such that
[Equation]
Since event E (cid:48) happens
[Equation]
we can ﬁnd
i ∈
A ( Wx ) and j ∈ [ l ] \ A ( Wx )
such that
[Equation]
Then, we have
[Equation]
where the second step follows from W i y <
W j y − √
[Equation] [Equation]
y
√ ml ) .
By Lemma 3, we know that A ( Wx ) (cid:54) =
A ( Wy (cid:48) ) .
Thus, ∀ y (cid:48) ∈ R m with (cid:107) y (cid:48) (cid:107) 2 = 1 and (cid:104) x, y (cid:48) (cid:105) =
α , we have A ( Wx ) (cid:54) =
A ( Wy (cid:48) ) conditioned on E (cid:48) .
By Lemma 2, we can conclude that ∀ x (cid:48) ∈ R m with (cid:107) x (cid:48) (cid:107) 2 = 1 and (cid:104) x, x (cid:48) (cid:105) ≤
α , we have A ( Wx ) (cid:54) =
A ( Wx (cid:48) ) conditioned on E (cid:48) .
Theorem 9 (A formal version of Theorem 1) .
Consider a weight matrix W ∈ R l × m where each entry is an i.i.d.
sample drawn from the Gaussian distribution N (0 , 1 /l ) .
Let γ ∈ (0 , 0 . 48) be the sparsity ratio of the activation function, i.e., γ = k/l .
With probability at least 0 .
99 , it has ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ m .
Condition on that ∀ i ∈
[ l ] , (cid:107) W i (cid:107) 2 ≤ 10 √ m happens, then, for any x ∈ R m , if
[Equation]
for some β ∈ (0 , 1) and a sufﬁciently large constant C , with probability at least 1 − δ · 2 − m , ∀ x (cid:48) ∈
R m with (cid:107) ∆ x (cid:107) 22 / (cid:107) x (cid:107) 22 ≥ β , A ( Wx ) (cid:54) =
A ( Wx (cid:48) ) , where x (cid:48)
=
c · ( x + ∆ x ) for some scaler c , and ∆ x is perpendicular to x .
Proof.
If (cid:104) x, x (cid:48) (cid:105) ≤ 0 , then the statement follows from Theorem 8 directly.
In the following, we consider the case (cid:104) x, x (cid:48) (cid:105) > 0 .
If (cid:107) ∆ x (cid:107) 2 / (cid:107) x (cid:107) 22 ≥ β , (cid:104) x, x (cid:48) (cid:105)(cid:107) x (cid:107) 2 2 (cid:107) x (cid:48) (cid:107) 2=
[Equation]
2 (cid:107) x (cid:107)(cid:107) x (cid:107) 22 ( c 2 ( (cid:107) x
(cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
) = (cid:107) x (cid:107)(cid:107) x (cid:107)
+ (cid:107) ∆ x (cid:107)≤ (cid:107) x (cid:107)(cid:107) x (cid:107)
+ β (cid:107) x (cid:107) 22 ≤ 1 + β .
Thus, we have the bounds:
[Equation]
By Theorem 8, we conclude the proof.
Example 1.
Suppose that the training data contains N points x 1 , x 2 , · · · , x N ∈ R m ( m ≥ Ω(log N ) ), where each entry of x i for i ∈ [ N ] is an i.i.d.
Bernoulli random variable, i.e., each entry is 1 with some probability p ∈ (100 log( N ) /m, 0 . 5) and 0 otherwise.
Consider a weight matrix
W ∈ R l × m where each entry is an i.i.d.
sample drawn from the Gaussian distribution N (0 , 1 /l ) .
Let γ ∈ (0 , 0 . 48) be the sparsity ratio of the activation function, i.e., γ = k/l .
If l ≥ Ω( m/γ · log( m/γ )) , then with probability at least 0 .
9 , ∀ i, j ∈ [ N ] , the activation pattern of Wx i and Wx j are different, i.e., A ( Wx i ) (cid:54)
=
A ( Wx j ) .
Proof.
Firstly, let us bound (cid:107)
x
i (cid:107) 2 .
We have E [ (cid:107) x
i (cid:107) 22 ] =
E [ (cid:80) mt
=1 x i,t ] = pm .
By Bernstein inequality, we have
[Equation]
Thus, by taking union bound over all i ∈
[ N ] , with probability at least 0 . 99 , ∀ i ∈
[ N ] , √ 0 .
9 pm ≤ (cid:107) x i (cid:107) 2 ≤ √ 1 . 1 pm .
Next we consider (cid:104) x i , x j (cid:105) .
Notice that E [ (cid:104) x i , x j (cid:105) ]
=
E [ (cid:80) mt
=1 x i,t x j,t ] = p 2 m .
There are two cases.
Case 1 ( p 2 m > 20 log N ).
By Bernstein inequality, we have
Pr
[Equation]
By taking union bound over all pairs of i, j , with probability at least 0 . 99 , ∀ i (cid:54) =
j , (cid:104) x
i , x j (cid:105) ≤ 32 p 2 m .
Since (cid:107) x i (cid:107) 2 , (cid:107) x j (cid:107) 2 ≥ √ 0 .
9 pm , we have (cid:104) x i , x j (cid:105) (cid:107) x i (cid:107) 2 (cid:107) x j (cid:107) 2 ≤ 3 p 2 m/0 .
9 pm = 3 p ≤ 5 6 .
Case 2 ( p 2 m ≤ 20 log N ).
By Bernstein inequality, we have
Pr (cid:2)(cid:12)(cid:12) (cid:104) x i , x j (cid:105) − p 2 m (cid:12)(cid:12) > 10 log N (cid:3) ≤ 2 e −
[Equation]
10log N
[Equation]
2 / 2 p 2 m +13 · 10log N ≤ 0 .
01 /N 2 .
By taking union bound over all pairs of i, j , with probability at least 0 . 99 , ∀ i (cid:54) =
j, (cid:104) x
i , x j (cid:105) ≤ 10 log N ,
Since (cid:107) x i (cid:107) 2 , (cid:107) x j (cid:107) 2 ≥ √ 0 .
9 pm ≥ √ 90 log N , we have (cid:104) x
i , x j (cid:105) (cid:107) x i (cid:107) 2 (cid:107) x j (cid:107) 2 ≤ 10 log N 90 log N = 9 .
Thus, with probability at least 0 . 98 , we have ∀ i (cid:54) =
j , (cid:104) x
i , x j (cid:105) /
( (cid:107) x
i (cid:107) 2 (cid:107) x j (cid:107) 2 ) ≤ 5 / 6 .
By Theorem 8, with probability at least 0 . 99 , ∀ q ∈ [ l ] , (cid:107) W q (cid:107) 2 ≤ 10 √ m .
Condition on this event, and since ∀ i (cid:54) =
j we have (cid:104) x
i , x j (cid:105) /
( (cid:107) x
i (cid:107) 2 (cid:107) x j (cid:107) 2 ) ≤ 5 / 6 , by Theorem 8 again and union bound over all i ∈
[ N ] , with probability at least 0 . 99 , ∀ i (cid:54)
=
j, A ( Wx i ) (cid:54) =
A ( Wx j ) .
C.
D ISJOINTNESS OF A CTIVATION P ATTERNS OF D IFFERENT I NPUT P OINTS
Let X 1 , X 2 , · · · , X m be i.i.d.
random variables drawn from the standard Gaussian distribution
N (0 , 1) .
Let Z = (cid:80) mi
=
X 2 i .
We use the notation χ 2 m to denote the distribution of Z .
If m is clear in the context, we just use χ 2 for short.
Lemma 8 (A property of χ 2 distribution) .
Let Z be a random variable with χ 2 m m ( m ≥ 2) distribution.
Given arbitrary ε, η ∈ (0 , 1) , if R is sufﬁciently large then
Pr[ Z ≥ (1 + ε ) R ] / Pr[(1 + ε ) R ≥ Z ≥ R ] ≤ η.
Proof.
Let R be a sufﬁciently large number such that:
• e εR/ 2 ≥ 4 ε .
• e εR/ 8 ≥ R m/ 2 − 1 .
• e εR/ 4 ≥ 169 · 1 η .
Let ξ = ε/ 4 .
By the density function of χ 2 distribution, we have
[Equation]
and
Pr[
[Equation]
where Γ( · ) is the Gamma function, and for integer m/ 2 , Γ( m/ 2)
= ( m/ 2 − 1)( m/ 2 − 2) · · ·· 2 ·
[Equation]
= ( m/ 2 − 1)! .
By our choice of R , we have
[Equation]
where the ﬁrst step follows from ∀ t ≥ R , t m/ 2 − 1 ≥ 1 , and the third step follows from e
[Equation]
1+
[Equation]
R
[Equation]
e − εR/ 2 ≤ ξ.
We also have:
[Equation] [Equation]
where the ﬁrst step follos from ∀ t ≥ R, t m/ 2 − 1 ≤ e ξt/ 2 , and the third step follows from (1 − ξ )(1 + ε ) ≥ (1 + ε/ 2) .
Thus, we have
Pr[
[Equation]
Lemma 9.
Consider x, y, z ∈ R m .
If
[Equation]
y
[Equation]
≤ α,
[Equation]
x
[Equation]
z
[Equation]
(cid:107) x (cid:107) 2 (cid:107) z (cid:107) 2 ≥ β for some α, β ≥ 0 , then (cid:104)
y,z (cid:105) (cid:107) y (cid:107) 2 (cid:107) z (cid:107) 2 ≤ α + (cid:112) 1 − β 2 .
Furthermore, if β = 2+ α + √ 2 − α 2, then (cid:104) y,z (cid:105) (cid:107) y (cid:107) 2 (cid:107) z (cid:107) 2 ≤ (1 − ε α ) β , where ε α ∈ (0 , 1) only depends on α .
Proof.
Without loss of generality, we suppose (cid:107) x (cid:107)
= (cid:107) y (cid:107) 2 = (cid:107) z (cid:107) 2 = 1 .
We can decompose y as ax + y (cid:48) where y (cid:48) is perpendicular to x .
We can decompose z as b 1 x + b 2 y (cid:48) / (cid:107) y (cid:48) (cid:107)
+ z (cid:48) where z (cid:48) is perpendicular to both x and y (cid:48) .
Then we have:
[Equation]
where the last inequality follows from 0 ≤ b 1 ≤ 1 , a ≤ α , and b 2 ≤ (cid:112) 1 − b 21 ≤ (cid:112) 1 − β 2 , 0 ≤ (cid:107) y (cid:48) (cid:107) 2 ≤ 1 .
By solving β ≥ α + (cid:112) 1 − β 2 , we can get β ≥ α + √ 2 − α 2.
Thus, if we set β =
[Equation]
, β should be strictly larger than α + (cid:112) 1 − β 2 , and the gap only depends on α .
Lemma 10.
Give x ∈ R m , let
y ∈ R
m be a random vector, where each entry of y is an i.i.d.
sample drawn from the standard Gaussian distribution N (0 , 1) .
Given β ∈ (0 . 5 , 1) ,
Pr[ (cid:104) x, y (cid:105) / ( (cid:107) x (cid:107) 2 (cid:107) y (cid:107) 2 ) ≥ β ] ≥ 1 / (1 + 1 / (cid:112) 2(1 − β )) m .
Proof.
Without loss of generality, we can assume (cid:107) x (cid:107) 2 = 1 .
Let y (cid:48) =
y/ (cid:107) y
(cid:107) 2 .
Since each entry of y is an i.i.d.
Gaussian variable, y (cid:48) is a random vector drawn uniformly from a unit sphere.
Notice that if (cid:104) x, y (cid:48) (cid:105) ≥ β , then (cid:107) x − y (cid:48) (cid:107) 2 ≤ (cid:112) 2(1 − β ) .
Let C = { z ∈ R m | (cid:107) z (cid:107) 2 = 1 , (cid:107)
z − x (cid:107) 2 ≤ (cid:112) 2(1 − β ) } be a cap, and let S = { z ∈ R m | (cid:107) z (cid:107) 2 = 1 } be the unit sphere.
Then we have
Pr[ (cid:104) x, y (cid:48) (cid:105) ≥ β ] = area( C ) / area( S ) .
According to Lemma 6, there is an (cid:112) 2(1 − β )
-net N with |N| ≤ (1 + 1 / (cid:112) 2(1 − β )) m .
If we put a cap centered at each point in N , then the whole unit sphere will be covered.
Thus, we can conclude
Pr[
[Equation]
, y
[Equation]
(cid:105) ≥ β ] ≥ 1 / (1 + 1 / (cid:112)
[Equation]
(1 − β )) m .
Theorem 10 (A formal version of Theorem 2) .
Consider N data points x 1 , x 2 , · · · , x N ∈ R m and a weight matrix W ∈ R l × m where each entry of W is an i.i.d.
sample drawn from the Gaussian distribution N (0 , 1 /l ) .
Suppose ∀ i (cid:54) =
j ∈ [ N ] , (cid:104) x i , x j (cid:105) /
( (cid:107) x
i (cid:107) 2 (cid:107) x j (cid:107) 2 ) ≤ α for some α ∈ (0 . 5 , 1) .
Fix k ≥ 1 and δ ∈ (0 , 1) , if l is sufﬁciently large, then with probability at least 1 − δ , ∀ i, j ∈ [ N ] , A ( Wx i ) ∩
A ( Wx j ) = ∅ .
Proof.
Notice that the scale of W and x 1 , x 2 , · · · , x N do not affect either (cid:104) x i , x j (cid:105) /
( (cid:107) x
i (cid:107) 2 (cid:107) x j (cid:107) 2 ) or the activation pattern.
Thus, we can assume (cid:107) x 1 (cid:107)
=
(cid:107) x 2 (cid:107) 2 = · · · = (cid:107) x N (cid:107) 2 = 1 and each entry of
W is an i.i.d.
standard Gaussian random variable.
Let β =
+ α + √ 2 − α 2and ε
α be the same as mentioned in Lemma 9.
Set ε and β (cid:48) as
[Equation]
Now, set
[Equation]
=
[Equation]
and let R satisﬁes
[Equation]
According to Lemma 8, if l is sufﬁciently large, then R is sufﬁciently large such that
[Equation]
Notice that for t ∈ [ l ] , (cid:107) W t (cid:107) 2 2 is a random variable with χ 2 m distribution.
Thus, Pr[ (cid:107) W t (cid:107) 2 ≥ (1 + ε ) R ] =
δ/ 100 l .
By taking union bound over all t ∈ [ l ] , with probability at least 1 − δ/ 100 , ∀ t ∈ [ l ] , (cid:107) W t (cid:107) 2 ≤ (1 + ε ) R .
In the remaining of the proof, we will condition on that ∀ t ∈
[ l ] , (cid:107) W t (cid:107) 2 ≤ (1 + ε ) R .
Consider i, j ∈ [ N ] , t ∈ [ l ] , if W t x i > β (cid:48)
R , then we have
[Equation]
Due to Lemma 9, we have
[Equation]
Thus,
[Equation]
Notice that for i ∈ [ N ] , t ∈ [ l ] , we have
[Equation]
By Chernoff bound, with probability at least 1 − δ/ (100 N ) , l (cid:88) t = 1 ( W t x i > β
[Equation]
R ) ≥ k.
By taking union bound over i ∈ [ N ] , with probability at least 1 − δ/ 100 , ∀ i ∈ [ N ] , l (cid:88) t =
( W t x
[Equation]
> β
[Equation]
R ) ≥ k.
This implies that ∀ i ∈ [ N ] , if t ∈ A ( Wx i ) , then W t x i > β (cid:48) R .
Due to Equation (20), ∀ j ∈ [ N ] , we have W t x
j < β (cid:48) R which implies that t (cid:54)∈ A ( Wx j ) .
Thus, with probability at least 1 − δ/ 50 ≥ 1 − δ probability, ∀ i (cid:54) =
j , A
( Wx i ) ∩
A ( Wx j ) = ∅ .
Remark 1.
Consider any x 1 , x 2 , · · · , x N ∈ R m with (cid:107) x 1 (cid:107)
= (cid:107) x 2 (cid:107) 2 = · · · = (cid:107) x N (cid:107) 2 = 1 .
If ∀ i (cid:54) =
j ∈ [ N ] , (cid:104) x i , x j (cid:105) ≤ α for some α ∈ (0 . 5 , 1) , then | N | ≤ (1 + 2 / (cid:112) 2(1 − α )) m .
Proof.
Since (cid:104) x i , x j (cid:105) ≤ α , (cid:107) x
i − x j (cid:107)
= (cid:107) x
i (cid:107) 22 + (cid:107) x j (cid:107) 22 − 2 (cid:104) x
i , x j (cid:105) ≥ 2 − 2 α .
Let S be the unit sphere, i.e., S = { x ∈ R m | (cid:107) x (cid:107) 2 = 1 } .
Due to Lemma 6, there is a ( (cid:112) 2(1 − α ) / 2) -net
N of S with size at most |N| ≤ (1 + 2 / (cid:112) 2(1 − α )) m .
Consider x i , x j , and y ∈ N .
By triangle inequality, if (cid:107) x i − y (cid:107) 2 < (cid:112) 2(1 − α ) / 2 , then (cid:107) x
j − y (cid:107) 2 > (cid:112) 2(1 − α ) / 2 due to (cid:107) x
i − x j (cid:107) 2 ≥ (cid:112) 2(1 − α ) .
Since N is a net of S , for each x i , we can ﬁnd a y ∈ N such that (cid:107) x
i − y (cid:107) 2 < (cid:112) 2(1 − α ) / 2 .
Thus, we can conclude N ≤ |N| ≤ (1 + 2 / (cid:112) 2(1 − α )) m .
Theorem 11.
Consider N data points x 1 , x 2 , · · · , x N ∈ R m with their corresponding labels z 1 , z 2 , · · · , z N ∈ R and a weight matrix W ∈ R l × m where each entry of W is an i.i.d.
sample drawn from the Gaussian distribution N (0 , 1 /l ) .
Suppose ∀ i (cid:54) =
j ∈ [ N ] , (cid:104) x i , x j (cid:105) /
( (cid:107) x
i (cid:107) 2 (cid:107) x j (cid:107) 2 ) ≤ α for some α ∈ (0 . 5 , 1) .
Fix k ≥ 1 and δ ∈ (0 , 1) , if l is sufﬁciently large, then with probability at least 1 − δ , there exists a vector v ∈ R l such that
[Equation]
Proof.
Due to Theorem 10, with probability at least 1 − δ , ∀ i (cid:54) =
j , A
( Wx i ) ∩
A ( Wx j ) = ∅ .
Let t 1 , t 2 , · · · , t N ∈ [ l ] such that t i ∈
A ( Wx i ) .
Then t i (cid:54)∈ A ( Wx j ) for j (cid:54) = i .
For each entry v t , if t = t i for some i ∈
[ N ] , then set v t
=
z i / ( W t x i ) .
Then for i ∈ [ N ] , we have (cid:104) v,
φ k ( Wx i ) (cid:105) =
(cid:88) t ∈A ( Wx i ) v t · W t x i
=
z i / ( W t i x i ) · W t i x i
=
z i .
D A DDITIONAL E XPERIMENTAL R ESULTS
This section presents details of our experiment settings and additional results for evaluating and empirically understanding the robustness of k -WTA networks.
D. E XPERIMENT S ETTINGS
First, we describe the details of setting up the experiments described in Sec. 4.
To compare k -WTA networks with their ReLU counterparts, we replace all ReLU activations in a network with k -WTA activation, while retaining all other modules (such as BatchNorm, Convolution, and pooling).
To test on different network architectures, including ResNet18, DenseNet121, and Wide ResNet, we use the standard implementations that are publicly available 3 .
All experiments are conducted using PyTorch framework.
Training setups.
We follow the same training procedure on CIFAR-10 and SVHN datasets.
All the
ReLU networks are trained with stochastic gradient descent (SGD) method with momentum= 0 .
We use a learning rate 0.1 from the ﬁrst to 50-th epoch and 0.01 from 50-th to 80-th epoch.
To compare with ReLU networks, the k-WTA networks are trained in the same way as ReLU networks.
All networks are trained with a batch size of 256.
For k -WTA networks with a sparsity ratio γ = 0 .
1 , when adversarial training is not used, we train them incrementally (recall in Sec. 2.2).
starting with γ = 0 .
2 for 50 epochs with SGD (using learning rate=0.1, momentum=0.9) and then decreasing γ by 0.005 every 2 epochs until γ reaches 0 .
When adversarial training is enabled, we use untargeted PGD attack with 8 iterations to construct adversarial examples.
To train networks with TRADES (Zhang et al., 2019), we use the implementation of its original paper 4 with the parameter 1 /λ = 6 , a value that reportedly leads to the best robustness according to the paper.
To train networks with the free adversarial training method (Shafahi et al., 2019b), we implement the training algorithm by following the original paper.
We set the parameter m = 8 as suggested in the paper.
Attack setups.
All attacks are evaluated under the (cid:96) ∞ metric, with perturbation size (cid:15)
= 0.
(CIFAR-10) and 0.047 (SVHN) for pixels ranging in [0 , 1] .
We use Foolbox (Rauber et al., 2017), a third-party toolbox for evaluating adversarial robustness.
We use the following setups for generating adversarial examples in various attack methods: For PGD attack, we use 40 iterations with random start, the step size is 0.003.
For C&W attack, we set the binary search step to 5, maximum number of iterations to 20, learning rate to 0.01, and initial constant to 0.01.
For Deepfool, we use 20 steps and 10 sub-samples in its conﬁguration.
For momentum attack, we set the step size to 0.003 and number of iterations to 20.
All other parameters are set by
Foolbox to be its default values.
D. E FFICACY OF I NCREMENTAL T RAINING
We now report additional experiments to demonstrate the efﬁcacy of the incremental ﬁne-tuning method (described in Sec. 2.2).
As shown in Figure 6 and described its caption, models trained with incremental ﬁne-tuning (denoted as w/ FT in the plots’ legend) performs better in terms of both standard accuracy (denoted as std in the plots’ legend) and robust accuracy (denoted as Rob in the plots) when the k -WTA sparsity γ < 0 .
2 , suggesting that ﬁne-tuning is worthwhile when γ is small.
D. A DDITIONAL RESULTS ON CIFAR-
Tests on different network architectures.
We evaluate the robustness of k -WTA on different network architectures, including ResNet-18, DenseNet-121 and WideResNet-22-10.
The results are reported in Table 2, where similar to the notation used in Table 1 of the main text, A rob is calculated as the worst-case robustness, i.e., under the most effective attack among PGD, C&W, Deepfool and
MIM.
The training and attacking settings are same as other experiments described Sec.
D.1.
As shown in Table 2, while the standard and robustness accuracies, A std and A rob , vary on different network architectures, k -WTA networks consistently improves the worst-case robustness A rob over
ReLU networks, no matter what the network architecture and training method are used.
ResNet
Wide ResNet
[Figure]
u r ac y
A cc u r ac y
Sparsity Sparsity
[Table]
Comparison with LWTA.
We in addition compare k -WTA to LWTA activation (Srivastava et al., 2013; 2014).
For fair comparisons, we use the same sparsity ratio γ in both k -WTA and LWTA.
As shown in Table 2, on all network architectures and training methods we tested, k -WTA networks consistently have better robustness performance than LWTA networks (in terms of both A std and A rob ).
These results suggest that k -WTA is more suitable then LWTA for defending against adversarial attacks.
Transfer attack.
Since a k -WTA network is architecturally similar to its ReLU counterpart—with the only difference being the activation—we evaluate their robustness under (black-box) transfer
[Table] [Table]
attacks across k -WTA and ReLU networks.
To this end, we build a ReLU and a k -WTA-0.1 network on ResNet-
[Table]
and train both networks with natural (non-adversarial) training as well as adversarial training.
This gives us four different models denoted (in Table 3) as ReLU, k -WTA-0.1, ReLU (AT), and k -WTA-0.1 (AT).
We then launch transfer attacks across each pair of models.
We also consider by-far the strongest black-box attack (according to Papernot et al.
(2017)): for the same model, for example, a k -WTA-0.1 network optimized by adversarial training, we train two independent versions, each with a different random initialization
[Table]
and apply the transfer attacks across the two versions.
The results are reported in Table 3, where each row corresponds to a target (attacked) model, and each column corresponds to a source model from which the adversarial examples are generated.
On the diagonal line of Table 3, each entry corresponds to the robustness under aforementioned transfer attacks across the two versions of the same models.
The results suggest that 1) it is more difﬁcult to transfer attack k -WTA networks than ReLU networks using adversarial examples from other models, and 2) it is also more difﬁcult to use adversarial examples of a k -WTA network to attack other models.
In a sense, the adversarial examples of a k -WTA network tend to be “disjoint” from the adversarial examples of a ReLU network, despite their architectural similarity.
Inspecting the diagonal entries of Table 3, we also ﬁnd
that k -WTA networks are more robust than their ReLU counterparts under the strongest black-box attack (Papernot et al., 2017) (i.e., transfer attacks across two different versions of the same model).
D. MNIST R ESULTS
On MNIST dataset, we conduct experiments with an adversarial perturbation size (cid:15) =0.3 for pixels ranging in [0 , 1] .
We use Stochastic Gradient Descent (SGD) with learning rate=0.01 and momentum=0.9 to train a 3-layer CNN.
The training takes 20 epochs for all the methods we evaluate.
The robust accuracy are evaluated under PGD attacks that take 20 iterations with random initialization and a step size of 0.03.
The results are summarized in Table 4.
Again, k -WTA activation consistently improves robustness under all different training methods.
Even with natural (non-adversarial) training, the resulting k -WTA network still has 62.2% robust accuracy, signiﬁcantly outperforming ReLU network.
D. R OBUSTNESS WITH RESPECT TO N ATURAL P ERTURBATIONS
We also evaluate the robustness of k -WTA networks under (non-adversarial) natural perturbations.
We evaluated various types of perturbations, including adding Gaussian noise to the input image (std=0.05/0.1), random translation (maximum 5 pixel), random rotation (maximum 10 degrees) and color jittering (i.e., randomly changing the brightness, contrast and saturation of an image, with a maximum perturbation of 0.4), following Hendrycks & Dietterich (2019).
The results are summerized in Table 5, in which all models are ResNet18.
We found that under all the tested perturbations, the accuracy drops in k -WTA networks are no worse than those in ReLU networks.
Note that in these tests, all our models are trained with standard data
[Table]
augmentations (
[Table]
g., random crop and random ﬂip); they are not speciﬁcally trained to avert the tested perturbations.
We also highlight an interesting ﬁnding here.
Adding Gaussian noise leads to a large accuracy drop (e
[Table]
g., from 92.9% to 69.7%/27.0% as shown in the table) on naturally trained ReLU network, but in k -WTA networks (especially k -WTA-0.1), the corresponding accuracy drop is much smaller (e
[Table]
g., from 89.3% to 80.2%/50.9%).
We conjecture that the dense discontinuities in k -WTA networks (recall Figure 5) effectively add noise to the input distribution, thus making the model more robust against input noise.
D. L OSS L ANDSCAPE V ISUALIZATION
In addition to the experiments shown in Figure 5 and Sec.
4.3 of the main text
[Table]
we further we visualize the loss landscapes of k -WTA networks when different sparsity ratios γ are used.
The plots are shown in Figure 7, produced in the same way as Figure 5 described in Sec. 4.3.
As analyzed in Sec. 3, a larger γ tends to smooth the loss surface of the k -WTA network with respect to the input, while a smaller γ renders the loss surface more discontinuous and “spiky”.
In addition, adversarial training tends to reduce the range of the loss values—a similar phenomenon in ReLU networks has already been reported Madry et al.
; Tramèr et al.
(2017)—but that does not mean that the loss surface becomes smoother; the loss surface remains spiky.
[Figure]