Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
Abstract
Using weight decay to penalize the L2 norms of weights in neural networks has been a standard training practice to regularize the complexity of networks.
In this paper, we show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with positively homogeneous activation functions, such as linear, ReLU and max-pooling functions.
As a result of homogeneity, functions speciﬁed by the networks are invariant to the shifting of weight scales between layers.
The ineffective regularizers are sensitive to such shifting and thus poorly regularize the model capacity, leading to overﬁtting.
To address this shortcoming, we propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
The derived regularizer is an upper bound for the input gradient of the network so minimizing the improved regularizer also beneﬁts the adversarial robustness.
We demonstrate the efﬁcacy of our proposed regularizer on various datasets and neural network architectures at improving generalization and adversarial robustness.
1. Introduction
Weight decay (Krogh & Hertz, 1992) is a common regularization in training deep neural networks, i.e., to regularize the sum of squared L2 norms of the weights.
However, when the network has positively homogeneous activation functions (PHAFs), i e. φ ( ax ) =
aφ ( x ) for a > 0 , the scale of weights can be shifted between layers without changing the input-output function speciﬁed by the network.
Consider a 2-layer neural network with input x ∈ R d and weights
W 1 ∈ R d × h and w 2 ∈ R h .
Assuming the activation func- tion φ is positively homogeneous, e.g., ReLU or linear function, we have y = w T 2 φ ( W T 1 x ) =
a w T 2 φ ( 1 a W T 1 x ) for a > 0 .
This implies that the weight decay cannot well regularize the intrinsic norm of networks; we can change the weight decay term by shifting weight magnitudes between layers while maintaining the network function.
Here, we further consider a family of regularizers, which includes weight decay, and prove that they can always be minimized by properly shifting scales without changing the network function.
Since PHAFs are common in modern networks, overcoming this difﬁculty of weight decay, and its larger family of regularizers, is of great signiﬁcance.
In this paper, we propose to regularize the intrinsic norms of the networks using an improved norm regularizer, which is invariant to shifting of weight scales between layers.
Based on the homogeneity of activation functions, we extract the weight magnitudes η ( W i ) of all layers, measured by some pre-deﬁned norms (e.g., l p or spectral norm), resulting in a product of norms for the whole network, i.e., (cid:81)
Li =1 η ( W i ) .
Within each layer, this leaves a normalized weight
W i /η
( W i ) that should also be penalized to induce sparsity.
Thus we have two terms in the improved regularizer: the overall weight magnitude of the network, and the complexity of each layer.
We next show that besides the
WEight-Scale-Shift-Invariance (WEISSI) property, our regularizer improves adversarial robustness of neural networks.
Previous works (Simon-Gabriel et al., 2019; Ross & DoshiVelez, 2018; Cisse et al., 2017) have proved that penalizing the input gradient, or similarly the Lipschitz constant of a network, increases the adversarial robustness.
In this paper, we upper bound the norm of the input gradient by the weight energy term in our regularizer, which is previously shown to be related to the Lipschitz constant.
By deriving the input gradient, we also ﬁnd that sparsifying the activation maps has a positive effect on adversarial robustness.
Finally, we empirically prove the effectiveness of WEISSI regularizers on various neural architectures and datasets.
2. Preliminaries
Consider a neural network ˆ y = f
( x ; Θ ) : R D (cid:55)→ R P , where Θ are the weights and other trainable parameters.
The neural network has L hidden layers and the function is recursively composed as
[Equation]
We deﬁne the input x as h 0 for convenience.
Weight matrix
W l = [ w Tl, 1 ; · · · ; w Tl,H l ] ∈ R H l × H l [Equation] 1 , H l is the size of layer l , and the network as W =
( W 1 , · · · , W L +1 ) .
The activation functions φ ( · ) are assumed to be positively homogeneous, i.e., φ ( ax ) =
aφ ( x ) when a > 0 .
Many common activations in modern DNNs, such as ReLU, max pooling and average pooling, are positively homogeneous.
For simplicity, our analysis assumes that the output layer is linear – for networks with non-linear output layers, our analysis applies to the sub-network containing PHAFs.
The network is trained by minimizing a data loss term L data (e.g., MSE) and a regularization term L reg ,
[Equation]
and λ is a tradeoff parameter, and D = { ( x i , y i ) } i the data.
Notation.
We use (cid:107) · (cid:107) p to denote l p norm for vectors.
For matrices, (cid:107) · (cid:107) p,q is deﬁned as
[Equation]
When p = q , we use (cid:107) · (cid:107) p to represent matrix norm.
The spectral matrix norm is denoted as (cid:107) · (cid:107) σ .
Bold symbols are vectors or matrices while plain symbols represent scalars.
3. Weight Scale Shift and Ineffective Regularizers
Due to the positive homogeneity, a positive scalar can be pulled out of the activation function, i.e., φ ( ˜ W l h l − 1 +˜ b l ) =
γ l φ ( W l h l − 1 + b l ) , where γ l W l = ˜ W l , γ l b l = ˜
b l .
This factorization can be repeated from the ﬁrst layer to the output, yielding an equivalent network ˜ W =
( ˜ W 1 , · · · , ˜ W L +1 ) ,
[Equation]
Under the condition of the product term (cid:81) L +1 l =1 γ l = 1 , a “new” network with different weights is obtained, but with exactly the same mapping function as before.
In other words, the weight scale can be shifted between layers while keeping the network mapping function unchanged.
Such an equivalent transformation signals a problem with the commonly used weight decay regularizer.
Weight decay.
The weight decay (WD) regularizer is the sum of squared l 2 norms of the weights,
[Equation]
Note that in general, for two equivalent networks W , ˜ W ,
[Equation]
+
[Equation]
Thus, equivalent networks (with the same generalization error) have different regularization penalties.
As a result, the training with (3) will minimize the WD term by forming an equivalent network ˜ W to the current solution W ,
[Equation]
The minimum is found by Lagrange multipliers (see App. 1),
[Equation]
which occurs when the l 2 norms are the same for all layers,
[Equation]
Fig.
1 shows the equivalent-norm curves for a one hidden layer network.
All networks on one red line have the same mapping function and generalization error, while their L wd are different.
A consequence of this result is that any network with large weights in one layer, resulting in large lnorm, can be converted into an equivalent network with lower l 2 norm.
From another perspective, increasing the complexity of one layer will increase its l 2 norm, but this increase can be dampened by the other layers by shifting the weight scale around.
In particular, increasing a layer’s (cid:107) W l (cid:107) 2 by a factor of 2 will only increase the overall minimized l 2 norm L ∗ wd by an effective factor of 2 1 L +1 .
Crucially, this effective factor approaches 1 (i.e., no penalty increase) as the network depth L increases.
Thus, our key observation is that the weight decay regularizer is ineffective for deep networks because the penalty of increasing model complexity is dampened, while minimizing the loss in (3), allowing the model to more easily overﬁt.
In Appendix A.2, we give a more generalized ineffective regularization family.
4. WEISSI Regularizers
To overcome the problems of WD, we propose WEISSI regularizers to penalize the intrinsic norm of a DNN.
The disadvantage of WD stems from the fact that shifting the weight scales between layers will change the sum of the
L2 norm, allowing the regularizer to be artiﬁcially reduced.
Thus, to address this problem, an effective regularizer should
[Table]
be invariant to shifting of weight magnitudes between layers.
Our approach is to consider canonical forms of the
DNN weights, where the scale is factored from each layer according to a normalization function η ( W l ) ,
[Equation]
where possible choices of the normalizer are η ( W )
= (cid:107) W (cid:107) p,q .
Based on the canonical form, the new regularization term is separated into the overall weight energy over all layers, (cid:81) l η ( W l ) , and the complexity of individual layers, (cid:80) l g ( W l /η
( W l )) , where possible choices of g ( · ) are (cid:107) · (cid:107) p,q with different p or q to prevent the function from degenerating to constant 1.
Note that both terms are invariant to moving weight scales between layers, c.f. standard
WD.
Adding monotonically increasing functions on top of the weight energy term will not affect the invariance property, such as (cid:81)
l η ( W l ) n and (cid:80) l log η
( W l ) .
We choose two common norms, η ( · ) = (cid:107) · (cid:107) 2 and g ( · ) =
(cid:107) · (cid:107) 1 , and therefore the regularization term is
[Equation]
where ﬁrst term, L we , is deﬁned as the weight energy term, and the second term L wc as the layer complexity term.
For very deep neural networks, the weight energy term at initialization will explode to inﬁnity, so we take the logarithm.
The weight energy term is a measure of network capacity in (Neyshabur et al., 2015), supporting the efﬁcacy of the new regularizer from the perspective of learning theory.
The key difference here is that our work derives the WEISSI regularizers to effectively prevent overﬁtting during training, while (Neyshabur et al., 2015) gives a generalization bound based on the weight energy term in our WEISSI regularizer.
We analyze the effect of WEISSI on the optimization using the regularizer in (11) as an example.
Taking the gradient of L we with respect to the weight at l -th layer, we obtain d L we /d W l = 2 W l (cid:81) j (cid:54)
= l (cid:107) W j (cid:107) 22 .
In contrast to weight decay, whose gradient is d L wd /d W l = 2 W l , the new regularizer has an extra scalar term, i.e., the weight energy of all other layers.
Thus the gradient of L we is very large if
[Figure]
WDParsevalParseval
[Figure]
the network has a large product norm, and large penalties are thus incurred if the norm is increased.
For L wc , we observe that regularizing l 1 norm of the normalized weights induces sparse weights, which has been shown to improve adversarial robustness (Guo et al., 2018), and decrease the storage and computational time (Frankle & Carbin, 2019).
Adversarial Robustness.
The new regularizer has an additional advantage of alleviating the vulnerability to adversarial examples because the weight energy term is an upper bound of the input gradient.
Consider the gradient of the cross-entropy loss w.r.t.
the input x ,
[Equation]
where p j and ˆ y j are the j -th output of softmax function and corresponding logit, ˆ y gt is the logit of the ground-truth label.
For ReLU networks, we can derive an upper-bound to the input gradient norm as a function of the product norm (see
Appendix A.3 for derivation),
[Equation]
(13) where C is a constant related to sizes of layers.
Hence, regularizing the product term, as WEISSI does, can help control the sensitivity of a network to adversarial perturbations.
Previous works (Fazlyab et al., 2019; Guo et al., 2018) have made similar observations, but in here we derive the WEISSI regularizer from the perspective of weight scale shifting and improving adversarial robustness is a favorable property of our new regularizer.
In Appendix A.4, we discuss the situation in CNNs and ResNets.
5. Experiments
We show the effectiveness of WEISSI in various neural network architectures on standard image recognition datasets, using weight decay and Parseval Network (Cisse et al., 2017) as baselines.
More details about our experiment can be found in the Appendix B.
5.1. Networks without Residual Connections
We ﬁrst report the effectiveness of WEISSI in regularizing
DNNs without residual connections.
On MNIST (LeCun et al., 1998) dataset, we use two kinds of neural architec-
[Table]
[Table]
tures, a fully connected network (multilayer perceptron,
MLP) and a convolutional neural network (CNN).
The MLP has two fully connected hidden layers, and the CNN has two convolution layers.
On CIFAR10 (Krizhevsky et al., 2009) dataset, we use a variant of VGG16 (Simonyan &
Zisserman, 2014) as an exemplar of deep neural networks.
For shallow networks on MNIST, we use L we for the weight energy regularizer, and for VGG16 we use L ( CNN ) we (See
Appendix A.4).
Table 1 shows the performance of WD and WEISSI on
MNIST.
WEISSI has a clear advantage over WD in both neural architectures and both training settings.
We also observe that the variance of adversarial robustness (i.e., test accuracy on attacked images) when training with WEISSI is generally smaller than training with WD, indicating a lower sensitivity of WEISSI w.r.t.
random initializations.
Tableshows the performance of VGG16 trained on CIFAR10 using WEISSI and 2 baselines.
Although Parseval OC is better at defending small perturbation ( (cid:15) = 0 . 01 ) attacks in standard training, it will deteriorate dramatically when the attack has moderate perturbation scales ( (cid:15) = 0 .
When using adversarial training, Parseval OC shows a large decrease in both generalization and robustness, while WEISSI remains effective and has some advantage over WD on both clean and adversarial examples.
Finally, we note that adversarial training improves robustness of all architectures and regularizers when the training perturbation ∆ = 0 .
01 is closely matched to the attack perturbation (cid:15) ∈ { 0 . 01 , 0 . 03 } , with WEISSI showing the best performance overall.
5.2. Deep Residual Networks
We apply WEISSI with convex aggregation (WEISSI CA) to deep residual networks and demonstrate the advantage of WEISSI and WEISSI CA over several baselines.
We choose WRN-28-10 (Zagoruyko & Komodakis, 2016) as the architecture (28 layers and width 10).
Since the WRN has many convolution layers, we use L ( CNN ) we as the weight energy term.
Table 3 shows the performance of WEISSI (CA) and the baselines under standard and adversarial training.
For standard training, Parseval OC achieves the best generalization performance with WEISSI as the second.
Nonetheless,
WEISSI CA exhibits better adversarial robustness under both FGSM and PGD attacks, compared to other baselines.
WEISSI also shows similar robustness as WEISSI CA to
FGSM attacks, but performs poorly under PGD attacks, which shows that convex aggregation helps overall robustness.
For adversarial training, Parseval performs poorly in both generalization and robustness, while Parseval OC is good against small perturbation attacks ( (cid:15)
= 0 . 01 )
, but again fails when the perturbation is larger.
WEISSI and
WEISSI CA perform similarly and both have better adversarial robustness over WD and Parseval, especially for larger (cid:15) attacks.
In Fig.
2, we plot the histogram of l 2 gradient norm of 10,000 test images to show how WEISSI (CA) changes the input gradient norms compared to several baselines.
The number of gradient norms between 5.0 and 2.0 is negatively correlated with the adversarial robustness, since both
WEISSI and WEISSI CA have an obvious decrease in this region compared to 3 baselines.
Another ﬁnding is that Parseval network is good at suppressing relatively large input gradients, but this property does not seem to always help adversarial robustness.
Thus, we speculate that the key of adversarial robustness could also be suppressing small input gradient norms.
Appendix A.1 Minimized Weight Decay Regularization
We prove how to solve the optimization problem (7) in the main paper.
Recall that the problem is formulated as
[Equation]
Introduce the Lagrange multiplier λ and write the Lagrange function,
[Equation]
[Equation]
Take the gradient w.r.t.
γ l ∀ l and λ , we have,
[Equation]
Let the gradient be zeros, we have γ l = ( λ ) 1 / 2 / (2 (cid:107) W l (cid:107) 22 ) 1 / 2 and bring γ l back to (17), we have
[Equation]
[Equation]
So the optimal solution of γ l is ˆ γ l = ( (cid:81)
L +1 j =1 (cid:107) W j (cid:107) 22 ) 1 2( L +1) / (cid:107) W l (cid:107) 2 .
It is easy to check that the second-order gradient of the objective function w.r.t.
ˆ γ l is greater than 0, so we obtain a minimum solution.
Thus, the mimimized l 2 weight norm is
[Equation]
19) and the minimized weight decay is ( L + 1)( (cid:81)
L +1 l =1 (cid:107) W l (cid:107) 22 ) 1 / ( L +1) .
Appendix A.2 An Ineffective Regularization Family
We prove Theorem 3.1 in the main paper, which states as follows,
Theorem 3.1.
Assume we have a regularization function g : R m × n (cid:55)→ R , ∀ m, n ∈ N and a set of real matrices { W l } L +1 l =1 .
If 1) ∃ p > 0 , g ( γ ) =
γ p and 2) g ( γ W ) =
g ( γ ) g
( W ) , then the following equation holds.
[Equation]
proof.
Since we have g ( γ W ) =
g ( γ ) g
( W ) , the objective function can be written as
[Equation]
[Equation]
Following the previous section, we write the Lagrange funtion,
[Equation]
Then take the gradient of the Lagrange function,
[Equation]
Let the gradient be zero, we have
[Equation]
Then bring γ l back to (24), we have
[Equation]
Thus the optimal γ l the minimized g ( W l ) and objective function is
[Equation]
Appendix A.3 WEISSI and input gradient norm
Consider the gradient of the cross-entropy loss w.r.t.
the input x ,
[Equation]
(30) where p j and ˆ y j are the j -th output of softmax function and corresponding logit, ˆ y gt is the logit of the ground-truth label.
Then, the input gradient is
[Equation]
Taking the l 2 norm of input gradient, we have
[Equation]
because the norm is sub-multiplicative.
For ReLU network, we have (cid:107) J l (cid:107)
≤
H
l
(the size of layer l ), thus obtaining an upper-bound to the input gradient norm as a function of the product norm,
[Equation]
where C is a constant related to sizes of layers.
Appendix A.4 Convolution Neural Networks
A CNN uses structured convolution connections between layers, and the network function has a different expression from the fully connected case in (2).
The weight scale can be shifted among layers if we regard the convolution kernel as a weight matrix.
However, one problem with convolution layers is that the kernel parameters actually have more inﬂuence than weight parameters in fully connected layers, since they are actually applied M times (equal to output feature map size) due to the sliding of kernel over the feature map.
One solution is to multiply by M in the regularizer.
But the multipliers can be moved across layers due to the WEISSI property, see (11).
To address this shortcoming, for deep convolution network, we choose to sacriﬁce WEISSI property among all layers, and downscale the weight of the fully connected layers taking the square-root, L ( CNN ) we =
[Equation]
[Equation]
In this way, we maintain the WEISSI property among all convolution layers and give proper penalties for their kernel parameters.
From another perspective, the larger exponent on the CNN kernel weights suppresses parameters smaller than 1.0 and gives more penalty for those parameters larger than 1.0, inducing higher sparsity in the convolution kernels.
We consider the convolution layers in CNN and show that the conv layer is a special case of fully connected layer.
Assume the input z ∈ R D
× D is a 2D input with one channel (for simplicity) to the convolution layer W ∈ R d k × d k × 1 × H , where d k is the kernel size and H is the output channel number.
We ﬂatten the input z to a 1D vector ˜ z with zero paddings and transform the convolution kernels so that we can write the convolution operation as ˜ W ˜ z .
Assuming the stride is 1 and use zero paddings to keep the input and output has the same size, the ( i, j, m ) output element of the conv layer is
[Equation]
This can be seen as the product of a sparse vector consisting of 0 and W 1 , 1 ,m , . .
, W d k ,d k ,m and ˜ z , where the positions of kernel parameters are determined by the positions of z i,j so that we get the Equ. (35) from the vector product.
In this way, we can have a sparse matrix ˜ W consisting of elements in W where each column is a sparse vector and has the same effect as the convolution operation, and the output of the conv layer can be written in the form of matrix multiplication ˜ W ˜ z .
Notice that the kernel parameters in the matrix are multiplied several times, which is different from parameters in fully connected layers.
Speciﬁcally, the repeated computation time is the same as the output size.
If we calculate the l 2 norm for the transformed matrix, we have (cid:107) ˜ W (cid:107) 2 = D 2 (cid:107) W (cid:107) 2 .
This suggests a more reasonable weight decay scheme: multiply the repeated computation times for convolution kernels.
But the multiplication does not apply to our WEISSI regularizer, since the weight energy term is a product norm of all layers and the scale for one layer is equivalent for all layers.
That is why we propose to give larger exponents for conv kernel terms to give a proper regularization for conv layers.
Appendix A.5 Convex Aggregation in ResNet and Input Gradient Norm
We consider DNNs with residual connections (He et al., 2016), in particular Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016), and obtain the same WEISSI regularizer as that of the standard network.
Using our notation, a residual block in
WRN is,
[Equation]
36) where the weights can be in the form of fully connected or convolution layers, W (1) l
[Equation]
and we assume in each residual block the dimensions of input and output are the same for simplicity, i.e., H (2) l [Equation]
The scale in the weight matrices are pulled out of the residual blocks, i.e.,
[Equation]
Note that we assume the shortcut connection has a scalar multiplier, e.g., as in adaptive ResNet (Zhang et al., 2018), which can be re-scaled to keep the network’s function unchanged after the weight scale shifting.
Repeating this transformation we obtain a product of weight scales in front of the output of the network, thus we can also use the WEISSI regularizer (11) for
ResNet.
The Effect of Convex Aggregation.
We adopt the convex aggregation strategy when summing the residual connections and feature maps (Cisse et al., 2017; Zhang et al., 2018).
The residual block has extra weight scalars 0 ≥ α l ≥ 1 and 0 ≥ β l ≥ 1 , with α l + β l = 1 ,
[Equation]
(39) which account for the relative importance of the shortcut h l − 1 and feature map φ ( W (1) l φ ( h l − 1 )) .
Cisse et al.
(2017) explains the importance of convex aggregation by considering the Lipschitz constant of one aggregation node.
Here we explain the effectiveness of convex aggregation from a global perspective, i.e., the input gradients of the whole network.
We next show that convex aggregation removes an exponential term in an upper bound of the input gradient norm compared to standard ResNet, thus better constraining input gradients.
The gradient of logit y
i w.r.t input x is
[Equation]
where the intermediate term R L comprising residual blocks and aggregation scalars is our main interest.
Deﬁning the uniform upper bound for the norm of product weight terms,
[Equation]
where S tβ contains the β m weights for the t -th term in the product expansion of (46), we obtain an upper bound,
[Equation]
(43) since α l + β l = 1 .
We upper bound the input gradient as
[Equation] [Equation]
where H L is from the upper bound for (cid:107) J L +1 (cid:107) 2 .
In contrast, α l = β l = 1 for standard ResNet, and thus the upper bound in (51) for these networks is 2 L σ ( Θ , x ) , which contains an exponential term w.r.t network depth L .
Thus, convex aggregation avoids the exponential dependence of the input gradient norm on network depth in residual networks.
The gradient of logit y
i w.r.t input x is
[Equation]
[Equation]
where the intermediate term R L comprising residual blocks and aggregation scalars is our main interest.
We expand R L and obtain a summation of 2 L summands, where the t th summand includes the product (cid:81) l ∈S tα α l (cid:81) m ∈S tβ β m , and S tα , S tβ are index sets of α l and β m according to the expansion.
In the t -th summand, all α l and β m are from different multipliers/layers and |S tα | + |S tβ | =
L .
We write the expansion of R L as follows,
[Equation]
According to Minkowski’s inequality, we have (cid:107) R L (cid:107) 2 is upper bounded by [Equation]
[Equation]
The weights of a neural network cannot be inﬁnity, thus we denote the uniform upper bound for the norm of product weight terms as σ ( Θ , x ) , i.e.,
[Equation]
[Equation]
Thus, we have
[Equation]
since α l + β l = 1 .
We upper bound the input gradient as
[Equation]
where H L is from the upper bound for (cid:107) J L +1 (cid:107) 2 .
Appendix A.6 Related Work
We ﬁrst discuss several regularizers for neural networks and theoretical work on weight norm and generalization.
Weight decay.
Weight decay improves the generalization (Krogh & Hertz, 1992) of networks by penalizing large weight norms during training.
Many modern DNNs are trained with weight decay, including VGG (Simonyan & Zisserman, 2014),
ResNet (He et al., 2016) and Densenet (Huang et al., 2017).
We demonstrate that weight decay is unable to control the intrinsic weight norms of a neural network with positively homogeneous activation functions, especially when the number of layers in the network becomes large.
Since the neural networks that are widely used today often have ReLU activations and very deep architectures, this ineffectiveness of weight decay may cause severe problems in terms of generalization and adversarial robustness.
Our WEISSI regularizers mitigate this difﬁculty of weight decay and thus improve the generalization.
We further prove that WEISSI regularizers are preferable over weight decay in improving the adversarial robustness.
Dropout.
Another widely used regularizer for DNNs is Dropout (Srivastava et al., 2014), which prevents co-adaptation of neurons by randomly dropping neurons during training.
The implicit regularizer induced by Dropout for a single hidden-layer linear neural network is proven to constrain every hidden neuron’s input and output weight norms to be equal (Mianjy et al., 2018).
Such a constraint is equivalent to the square of path regularization (Neyshabur et al., 2015), which is invariant to weight scale shifting between layers.
In this sense, Dropout for a single hidden-layer linear neural network has a very similar effect to that of WEISSI regularizers.
WEISSI regularization extends the weight scale shift invariance property of
Dropout to multi-layer neural networks.
However, we emphasize that for multi-layer networks, it is complicated to analyze the implicit regularization of Dropout and its co-adaptation reduction property is generally accepted.
Thus, we consider
Dropout as a complementary component to WEISSI regularizer.
Spectral Norm Regularization.
Previous works have proposed to constrain the spectral norm of a neural network to control its sensitivity to input perturbations (Yoshida & Miyato, 2017; Cisse et al., 2017).
Spectral norm regularization (Yoshida & Miyato, 2017) replaces the squared L2 norm in weight decay with the squared spectral norm.
However, it is still not invariant to weight scale shifting, and thus spectral norm regularization has the same drawback as weight decay.
Parseval network (Cisse et al., 2017) proposes to constrain the Lipschitz constant of a neural network by introducing an orthogonal constraint on the weight matrix and a convex aggregation for residual connections.
Although Parseval network and WEISSI regularizers both constrain the Lipschitz constant of neural networks, their approaches are quite different: Parseval network constrains the Lipschitz constant of each individual unit of a network to be less than 1, while WEISSI penalizes an estimated upper bound for the Lipschitz constant for a whole network.
One advantage of WEISSI is that the regularizers can be easily plugged into the standard training pipeline, while Parseval network requires extra iterations to make sure the constraints are satisﬁed.
Our experiment shows that WEISSI is more effective at improving the adversarial robustness than Parseval network.
We also use the convex aggregation of Parseval network in the WEISSI regularized WideResNet (Zagoruyko &
Komodakis, 2016), and achieve better performance than using Parseval network, which demonstrates the superiority of
WEISSI over orthogonal weight constraints.
In addition, we provide a theoretical explanation on the efﬁcacy of convex aggregation in controlling vulnerability of networks by analyzing adversarial attacks.
Weight Normalization.
Similar to the WEISSI formulation, weight normalization (Salimans & Kingma, 2016) disentangles the magnitude and direction of weight vectors and proposes to optimize the two components for one weight to accelerate convergence.
Despite the similarity in this step, WEISSI regularization and weight normalization have different purposes:
WEISSI aims to control the complexity of neural networks to mitigate overﬁtting, while weight normalization is dedicated
[Figure]
[Figure]
Model
[Table]
to faster optimization.
As an alternative to batch normalization (Ioffe & Szegedy, 2015), weight normalization may also hurt adversarial robustness (Xie & Yuille, 2020) since they both make the optimizer too strong, causing overﬁtting.
Norm-based Generalization Error.
The proposed WEISSI regularizer is closely related to several learning theory works (Bartlett, 1998; Neyshabur et al., 2015; 2018).
(Bartlett, 1998) proves that the generalization performance of a DNN depends on the magnitude of weights instead of the number of weights.
(Neyshabur et al., 2015) derives a bound of the
Rademacher complexity as a function of the product of weight norms, which is equivalent to the weight energy term in
WEISSI regularizers.
(Neyshabur et al., 2018) extends the previous work by bounding the generalization error in the
PAC-Bayesian framework, where the bound is a function of the product of spectral weight norms.
We note that (Neyshabur et al., 2015; 2018)
both prove the invariance of a network function to weight scale shifting due to the homogeneity of ReLU activations, and use the product of some weight norms to bound generalization error.
The key observation in our paper is the incapability of weight decay in penalizing the intrinsic norm of a neural network, which is not considered in the previous papers.
We further propose WEISSI regularizers to more effectively control the network capacity, and thus enhance generalization accuracy and adversarial robustness.
Moreover, we provide various empirical results to support our claim.
In contrast, (Neyshabur et al., 2015; 2018) provide theoretical insights into the effect of the weight norm product in controlling the generalization error, without considering adversarial robustness or giving experimental evidence.
Appendix B.1 Experiment Settings
Dataset.
We consider the task of image recognition using deep neural networks.
Our experiment is run on 2 standard image recognition datasets, i.e., MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky et al., 2009).
[Table]
we
[Table]
wc
MLP 1e1e- CNN 1e1e-
VGG 1e3e-
WRN 1e1e-
WRN CA 1e1e-
λ we λ wc
MLP 1e1e-
CNN 1e1e-
VGG 1e3e-
WRN 1e1e-
WRN CA 1e1e-
Baselines.
Three regularization methods are used for comparison baselines:
(i) Weight Decay (WD) takes the sum of L2 norm of all layers as the regularization term, as in (6).
(ii) Parseval OC network (Cisse et al., 2017) enforces that the rows of weight matrices are orthogonal, and its largest singular value is close to 1.
(iii) Parseval network (Cisse et al., 2017) imposes convexity constraints in residual aggregation layers (as in Eq. 39), in addition to the orthogonal constraints.
Note that in our experiment we update all rows of each weight matrix instead of a subset of rows as in the original paper (Cisse et al., 2017).
For Parseval network, we use a parameterized function to ensure a valid convex aggregation, a = exp( l a ) /
(exp( l a ) + exp( l b )) and b = 1 − a , instead of using the simplex projection.
Training.
We minimize the softmax loss function using the momentum optimizer (Polyak, 1964) for all neural networks.
For training on CIFAR10, we use data augmentation of random cropping and left/right ﬂipping.
Pixel values are normalized to [0 , 1] .
The batch size is set to be 100 for all experiments.
Note that we do not use Batch Norm in our experiments because it makes the network invariant to the scale of weight norms due to the layerwise normalization, and it has been shown that Batch Norm hurts the adversarial robustness (Summers & Dinneen, 2020).
We trained WideResnet with Batch Norm on CIFAR
and it turns out although the accuracy is increased to 95.09%, it drops to 34.24% under FGSM attack when (cid:15)
= 0 .
For VGG network we use He initialization (He et al., 2015), while for WRN we use Fixup initialization (Zhang et al., 2019) to avoid gradient explosion (Zhang et al., 2019).
In addition to standard training, we also apply the adversarial training in (Madry et al., 2018), minimizing the loss:
[Equation]
where B ∆ is the set of allowed perturbations.
We choose the l ∞ -ball as the set of allowed perturbations and ∆ = 0 .
01 as the maximum l ∞ distance.
Evaluation.
We evaluate the generalization ability using the test accuracy on clean examples, and evaluate the adversarial robustness using the test accuracy on adversarial examples generated from two attacks, FGSM (Goodfellow et al., 2015) and
PGD (Madry et al., 2018) with 10 steps (denoted as PGD10).
For MNIST, we run 5 initializations and report the mean and standard deviation.
For CIFAR10, we run 3 trials and choose the model with of median test accuracy.
The experiment is implemented in Tensorﬂow.
For weight decay, we set the λ wd = 0 . 0001 .
For Parseval network, we set the same λ wd = 0 .
for the weight decay of output layer and use β parseval = 0 .
in the orthogonal constraint updates.
The hyperparameters in WEISSI are listed in Table 5.
Notice that in some neural networks like VGG and WRN, we use different hyperparameters in adversarial and standard training.
In MNIST experiment, MLP has two hidden layers with a width of 1024 and CNN has two convolution layers, where each layer has 128 ﬁlters of 3 × 3 size and the convolution stride is 1.
We train the network for 60 epochs with an initial learning rate of 0.1.
In CIFAR10 experiment, we use a variant of
VGG16 for 32 × 32-sized input images, where the convolution layers are the same as VGG
but we use one 512-width fully connected layer, and WideResnet-28-10.
In VGG training, we train the network for 300 epochs with an initial learning rate of 0.01 and dropout rate of 0.5.
The learning rate is decayed exponentially every 100 epchs with a decay rate 0.1.
In WRN training, we set the training epoch as 200, initial learning rate as 0.03 and use the same learning rate decay scheme as in
VGG.
Ablation study.
We investigate the effect of L we and L wc on adversarial robustness of WRN by removing either term in the regularization.
Fig.
4 shows a comparison between full WEISSI CA and only L we or L wc regularization when using
FGSM and PGD attacks with different perturbation scales.
WEISSI CA is robust to both attacks while single L we or
L wc can only defend against one attack.
Note that regularizing L we beneﬁts robustness to FGSM, while regularizing L wc increases robustness to PGD.
This suggests sparsity is crucial for defending against PGD and a smoother decision boundary is beneﬁcial to defending against FGSM.
Visualization.
We visualize the change of L we and L wc during WRN training under the 5 regularization schemes in Fig.
WEISSI achieves the lowest weight energy, while WEISSI CA shows a consistent decrease of weight complexity, indicating that the convex aggregation also helps train a more sparse network.
Training with a Robust Loss Function.
Besides adversarial training, we test the performance of WEISSI using standard training with a robust loss function, Max-Mahalanobis center (MMC) loss (Pang et al., 2020).
The regularization functions are the same as before.
Table 4 shows a comparison between the performance of WEISSI and WD for VGG16 and WRN using standard training with MMC.
In this case, WEISSI and WD achieve similar generalization performance, but WEISSI outperforms WD on the VGG network in terms of adversarial robustness.