S TABLE R ANK N ORMALIZATION FOR I MPROVED G ENERALIZATION IN N EURAL N ETWORKS AND GAN S
A BSTRACT
Exciting new work on generalization bounds for neural networks (NN) given by Bartlett et al.
; Neyshabur et al.
(2018) closely depend on two parameterdependant quantities a) the Lipschitz constant upper bound and b) the stable rank (a softer version of rank).
Even though these bounds typically have minimal practical utility, they facilitate questions on whether controlling such quantities together could improve the generalization behaviour of NNs in practice.
To this end, we propose stable rank normalization (SRN), a novel, provably optimal, and computationally efﬁcient weight-normalization scheme which minimizes the stable rank of a linear operator.
Surprisingly we ﬁnd that SRN, despite being non-convex , can be shown to have a unique optimal solution.
We provide extensive analyses across a wide variety of NNs (DenseNet, WideResNet, ResNet, Alexnet,
VGG), where applying SRN to their linear layers leads to improved classiﬁcation accuracy, while simultaneously showing improvements in generalization, evaluated empirically using shattering experiments (Zhang et al., 2016); and three measures of sample complexity by Bartlett et al.
(2017), Neyshabur et al.
(2018), & Wei &
Ma.
Additionally, we show that, when applied to the discriminator of GANs, it improves Inception, FID, and Neural divergence scores, while learning mappings with a low empirical Lipschitz constant.
I NTRODUCTION
Deep neural networks have shown astonishing ability to tackle a wide variety of problems and have shown a great ability to generalize.
Within this work we leverage very recent, and important, theoretical results on the generalization bounds of deep networks to yield a practical low cost method to normalize the weights within a network using a scheme which we call Stable Rank Normalization (SRN).
The motivation for SRN comes from the generalization bound for NNs given by Neyshabur et al.
and Bartlett et al.
(2017), O (cid:113)(cid:81) di (cid:107)
W i (cid:107) 22 (cid:80) di
=1 srank( W i ) 1 , that depend on two parameter-dependent quantities: a) the scale-dependent Lipschitz constant upper-bound (cid:81) di (cid:107) W
i
(cid:107)(product of spectral norms) and b) the sum of scale-independent stable ranks ( srank( W ) ).
Stable rank is a softer version of the rank operator and is deﬁned as the squared ratio of the Frobenius norm to the spectral norm.
Although these two terms appear frequently in these bounds, the empirical impact of simultaneously controlling them on the generalization behaviour of NNs has not been explored yet possibly because of the difﬁculties with optimizing stable rank.
This is precisely the goal of this work, and based on extensive experiments across a wide variety of NN architectures, we show that, indeed, controlling them simultaneously improves the generalization behaviour, while improving the classiﬁcation performance of NNs.
We observe improved training of Generative Adversarial
Networks (GAN) Goodfellow et al.
(2014) as well.
To this end, we propose Stable Rank Normalization (SRN) which allows us to simultaneously control the Lipschitz constant and the stable rank of a linear operator.
Note that the widely used Spectral
Normalization (SN) (Miyato et al., 2018) allows explicit control over the Lipschitz constant, however, as will be discussed in the paper, it does not have any impact on the stable rank.
We would like to emphasize that, as opposed to SN, the SRN solution is optimal and unique even in situations when it is non-convex.
It is one of those rare cases where an optimal solution to a provably non-convex problem could be obtained.
Computationally, our proposed SRN for NNs is no more complicated than SN, just requiring computation of the largest singular value which can be efﬁciently obtained using the power iteration method (Mises & Pollaczek-Geiringer, 1929).
Experiments Although SRN is in principle applicable to any problem involving a sequence of afﬁne transformations, considering recent interests, we show its effectiveness when applied to the linear layers of deep neural networks.
We perform extensive experiments on a wide variety of NN architectures (DenseNet, WideResNet, ResNet, Alexnet, VGG) for the analyses and show that, SRN, while providing the best classiﬁcation accuracy (compared against standard, or vanilla, training and SN), consistently shows improvement on the generalization behaviour.
We also experiment with GANs and show that, SRN prefers learning discriminators with low empirical Lipschitz while providing improved Inception, FID and Neural Divergence scores (Gulrajani et al., 2019).
We would like to note that although SN is being widely used for the training of GANs, its effect on the generalization behaviour over a wide variety of NNs has not yet been explored.
To the best of our knowledge, we are the ﬁrst to do so.
Contributions
• We propose SRN— a novel normalization scheme for simultaneously controlling the Lipschitz constant and the stable rank of a linear operator.
• Optimal and unique solution to the provably non-convex stable rank normalization problem.
• Efﬁcient and easy to implement SRN algorithm for NNs.
B ACKGROUND AND I NTUITIONS
Neural Networks Consider f θ : R m → R k to be a feed-forward multilayer NN parameterized by θ ∈ R n , each layer of which consists of a linear followed by a non-linear 2 mapping.
Let a l − 1 ∈ R n l − 1 be the input (or pre-activations) to the l -th layer, then the output (or activations) of this layer is represented as a l = φ l ( z l ) , where z l = W l a l − 1 + b l is the output of the linear (afﬁne) layer parameterized by the weights W l ∈ R n l − 1 × n l and biases b l ∈ R n l , and φ l ( . ) is the element-wise non-linear function applied to z l .
For classiﬁcation tasks, given a dataset with input-output pairs denoted as ( x ∈ R m , y ∈ { 0 , 1 } k ; (cid:80) j y j = 1) 3 , the parameter vector θ is learned using back-propagation to optimize the classiﬁcation loss ( e.g. , cross-entropy).
Singular Value Decomposition (SVD)
Given W ∈ R s × r with rank k ≤ min( s, r ) , we denote { σ i } ki =1 , { u i } ki =1 , and { v i } ki =1 as its singular values, left singular vectors, and right singular vectors, respectively.
Throughout this work, a set of singular values is assumed to be sorted σ 1 ≥ · · · ≥ σ k .
σ i ( W ) denotes the i -th singular value of the matrix W .
Using singular values, the matrix 2-norm (cid:107) W (cid:107) 2 and the Frobenius norm (cid:107) W (cid:107) F can be computed as σ 1 and (cid:112)(cid:80) i σ 2 i , respectively.
Stable Rank
Below we provide the formal deﬁnition and some properties of stable rank.
Deﬁnition 2.1.
The Stable Rank (Rudelson & Vershynin, 2007) of an arbitrary matrix W is deﬁned as srank
[Equation]
, where k is the rank of the matrix.
Stable rank is
• a soft version of the rank operator and, unlike rank, is less sensitive to small perturbations.
• differentiable as both Frobenius and Spectral norms are almost always differentiable.
• upper-bounded by the rank: srank
[Equation]
• invariant to scaling, implying, srank( W ) =
srank( W η ) , for any η ∈ R \ { 0 } .
Lipschitz Constant Here we describe the global and the local Lipschitz constants.
Brieﬂy, the Lipschitz constant is a quantiﬁcation of the sensitivity of the output with respect to the change in the input.
A function f : R m (cid:55)→ R k is globally L-Lipschitz continuous if ∃ L ∈ R
+ : (cid:107) f
( x i ) − f ( x j ) (cid:107) q ≤
L (cid:107) x
i − x j (cid:107) p , ∀ ( x i , x j ) ∈ R m , where (cid:107)·(cid:107) p and (cid:107)·(cid:107) q represents the norms in the input and the output metric spaces, respectively.
The global Lipschitz constant L g is:
[Equation]
The above deﬁnition of the Lipschitz constant depends on all pairs of inputs in the domain
R m × R m , (thus, global).
However, one can deﬁne the local Lipschitz constant based on the sensitivity of f in the vicinity of a given point x .
Precisely, at x , for an arbitrarily small δ > 0 , the local Lipschitz constant is computed on the open ball of radius δ centered at x .
Let h ∈ R m , (cid:107) h (cid:107) p < δ , then, similar to L g , the local Lipschitz constant of f at x , L l ( x ) , is greaterAmartya: shouldn’t this be less than or equal to? than or equal to sup h
[Equation]
[Equation]
Assuming f to be Fréchet differentiable, as h → 0 , using f ( x + h ) − f ( x ) ≈ J f ( x ) h , L l ( x ) is the matrix (operator) norm of the Jacobian (cid:16)
J f ( x ) =
∂f ( z ) ∂
z | x ∈ R k × m (cid:17) as follows:
[Equation]
A function is said to be locally Lipschitz with local Lipschitz constant L l if for all x ∈ R m he function is L l locally-Lipschitz at x .
Thus, L l = sup x ∈ R m L l ( x ) .
Notice that the Lipschitz constant (global or local), greatly depends on the chosen norms.
When p = q = 2 , the upperbound on the local
Lipschitz constant at x boils down to the 2-matrix norm (maximum singular value) of the Jacobian
J f ( x ) (see last equality of (2)).
With these preliminary deﬁnitions, in Section 3, we discuss more optimistic (or empirical) estimates of L l and L g , its link with generalization and then in Section 5,we show empirically the effect of SRN on them and on generalization.
The local Lipschitz upper-bound for Neural Networks As mentioned earlier (2), L l ( x ) = (cid:107) J f ( x ) (cid:107) p,q , where, in the case of NNs (proof along with why it is loose in Appendix C)
[Equation]
and L l = L l ( x ) (3)  W HY
Lipschitz alone is not sufﬁcient Although learning low Lipschitz functions has been shown to provide better generalization (Anthony & Bartlett, 2009; Bartlett et al., 2017; Neyshabur et al., 2018; 2015; Yoshida & Miyato, 2017; Gouk et al., 2018), enable stable training of GANs (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018) and help provide robustness against adversarial attacks (Cisse et al., 2017), controlling Lipschitz upper bound alone is not sufﬁcient to provide assurance on the generalization error.
One of the reasons is that it is scale-dependent, implying, for example, even though scaling an entire ReLU network would not alter the classiﬁcation behaviour, it can massively increase the Lipschitz constant and thus the theoretical generalization bounds.
This suggests that either the bound is of no practical utility, or at least one should regulate both—the
Lipschitz constant, and the stable rank (scale-independent)—in a hope to see improved generalization in practice.
Stable rank controls the noise-sensitivity As shown by Arora et al.
(2018), one of the critical properties of generalizable NNs is low noise sensitivity— the ability of a network to preferentially carry over the true signal in the data.
For a given noise distribution N , it can be quantiﬁed as Φ f θ , N = max
[Equation]
Φ f θ , N ( x ) , where Φ f θ , N ( x ) :
= E η ∼N (cid:34) (cid:107) f
θ ( x + η (cid:107) x (cid:107) ) − f θ ( x ) (cid:107)(cid:107) f θ ( x ) (cid:107)(cid:35) .
For a linear mapping with parameters W and the noise distribution being normal- N (0 , I ) , it can be shown that Φ f W , N ≥ srank( W ) (c.f. Proposition 3.1 in Arora et al.
Thus, decreasing the stable rank directly decreases the lower bound of the noise sensitivity.
In Figure 1, we show Φ f θ , N of a ResNet110 trained on CIFAR100.
Note that although the Lipschitz upper bound of SRN and SN are the same, SRN (algorithmic details in Section 4) is much less sensitive to noise as the constraints imposed enforce the stable rank to decrease to 30% of its original value, which in effect reduces the noise sensitivity.
Stable rank impacts empirical Lipschitz constant It is apparent that the Lipschitz constant upper bound ( (cid:81) di
(cid:107) W
i (cid:107) 2 ), along with being scale-dependent, also is data-independant and hence, provides a pessimistic estimate of the behaviour of a model on a particular task or dataset.
Considering this, a relatively optimistic estimate of the model’s behaviour would be an empirical estimate of the Lipschitz constant ( L e ) on a task-speciﬁc dataset 5 .
Note that local L e is just the norm of the Jacobian at a given point 6 .
Empirically, Novak et al.
(2018) provided results showing how local L e (in the vicinity of train data) is correlated with the generalization error of NNs.
This observation is further supported by the work of Wei & Ma; Nagarajan & Kolter (2019); Arora et al.
(2018) whereby the variants of L e are used to derive generalization bounds.
Thus, a tool that favours low
L e is likely to provide better generalization behaviour in practice.
To this end, we ﬁrst consider a simple two layer linear-NN example in Appendix B.2 and show that low rank mappings do favour low
L e .
Since direct minimization of rank for NNs is non-trivial, the expectation is that learning low stable rank (softer version of rank) might induce similar behaviour.
We experimentally validate this hypothesis by showing that, as we decrease the stable rank, the empirical Lipschitz decreases.
This shows SRN indeed prefers mappings with a low empirical Lipschitz constant.
S TABLE R ANK N ORMALIZATION
Here we provide a theoretically sound procedure to do SRN.
A big challenge in stable rank normalization comes from the fact that it is scale-invariant (refer Deﬁnition 2.1), thus, any normalization scheme that modiﬁes W = (cid:80)
i σ i u i v (cid:62)
i to (cid:99) W = (cid:80) i σ i η u
i v (cid:62) i will have no effect on the stable rank, making SRN non-trivial.
Examples of such schemes are SN (Miyato et al., 2018) where η = σ 1 , and Frobenius normalization where η = (cid:107) W (cid:107) F .
As will be shown, our approach to stable rank normalization is optimal and efﬁcient.
Note, the widely used SN (Miyato et al., 2018) is not optimal (proof in Appendix A.2).
The SRN Problem Statement Given a matrix W ∈ R m × n with rank p and spectral partitioning index k ( 0 ≤ k < p ), we formulate the SRN problem as:
[Equation]
where, 1 ≤ r < srank( W ) is the desired stable rank, λ i s and σ i s are the singular values of (cid:99) W k and
W , respectively.
The partitioning index k is used for the singular value (or the spectrum) preservation constraint .
It gives us the ﬂexibility to obtain (cid:99) W k such that its top k singular values are exactly the same as that of the original matrix.
Note, the problem statement is more general in the sense that putting k = 0 removes the spectrum preservation constraint.
The Solution to SRN The optimal unique solution to the above problem is provided in Theoremand proved in Appendix A.1.
Note, at k = 0 , the problem (4) is non-convex, otherwise convex.
Theorem 1.
Given a real matrix W ∈ R m × n with rank p , a target spectrum (or singular value) preservation index k (0 ≤ k < p ) , and a target stable rank of r (1 ≤ r < srank( W )) , the optimal solution (cid:99) W k to problem (4) is (cid:99) W
k
= γ 1 S 1 + γ 2 S 2 , where S 1 = (cid:80) max(1 ,k ) i
=
σ i u i v (cid:62) i and
S 2 = W − S 1 .
{ σ i } ki =1 , { u i } ki =1 and { v i } ki =1 are the top k singular values and vectors of W , and, depending on k , γ 1 and γ 2 are deﬁned below.
For simplicity, we ﬁrst deﬁne γ =
[Equation]
, then
a) If k =
(no spectrum preservation), the problem becomes non-convex, the optimal solution to which is obtained for γ 2 = γ + r − 1 r and
γ 1 = γ 2 γ , when r > 1 .
If r = 1 , then γ 2 =and γ 1 = 1 .
Since, in this case, (cid:107) S 1 (cid:107) 2F = σ 21 , γ = √ r −
[Equation] [Equation]
S 2 (cid:107) F
[Equation]
b) If k ≥ 1 , the problem is convex.
If r ≥ (cid:107) S 1 (cid:107) 2F σthe optimal solution is obtained for γ 1 = 1 , and γ 2 = γ and if not, the problem is not feasible.
c)
Also, (cid:13)(cid:13)(cid:13)(cid:99) W k − W (cid:13)(cid:13)(cid:13) F monotonically increases with k for k ≥ 1 .
Intuitively, Theorem 1 partitions the given matrix into two parts, depending on k , and then scales them differently in order to obtain the optimal solution.
The value of the partitioning index k is a design choice.
If there is no particular preference to k , then k = 0 provides the most optimal solution.
We provide a simple example to better understand this.
Given W = I 3 (rank = srank( W ) = 3), the objective is to project it to a new matrix with stable rank of 2 .
Solutions to this problem are:
[Equation]
Here, (cid:99) W 1 is obtained using the standard rank minimization (Eckart-Young-Mirsky (Eckart & Young, 1936)) while (cid:99) W 2 and (cid:99) W 2 are the solutions of Theorem 1 with k = 1 and k = 0 , respectively.
It is easy to verify that the stable rank of all the above solutions is 2 .
However, the Frobenius distance (lower the better) of these solutions from the original matrix follows the order (cid:13)(cid:13)(cid:13) W − (cid:99) W 1 (cid:13)(cid:13)(cid:13) F > [Equation] F > (cid:13)(cid:13) (cid:13) W − (cid:99) W 3 (cid:13)(cid:13) (cid:13) F .
As evident from the example, the solution to SRN, instead of completely removing a particular singular value, scales them (depending on k ) such that the new matrix has the desired stable rank.
Note that for (cid:99) W 1 (true for any k ≥ 1 ), the spectral norm of the original and the normalized matrices are the same, implying, γ 1 = 1 .
However, for k = 0 , the spectral norm of the optimal solution is greater than that of the original matrix.
It is easy to verify from Theorem 1 that as k increases, γ 2 decreases.
Thus, the amount of scaling required for the second partition S 2 is more aggressive.
In all situations, the following inequality holds: γ 2 ≤ 1 ≤ γ 1 .
Require:
W ∈ R m × n , r , k
≥1: S 1 ← 0 , β ← (cid:107) W (cid:107) 2F , η ← 0 , l ←2: for i ∈ { 1 , · · · , k } do 3: { u i , v i , σ i } ← SV D ( W , i ) 4: (cid:46) Power method to get i -th singular value 5: if r ≥ (cid:0) σ 2 i + η (cid:1) /σ 21 then 6:
S 1 ← S 1 + σ i u i v (cid:62) i 7: η ← η + σ 2 i , β ← β − σ 2 i 8: l ← l +9: else 10: break 11: end if 12: end for 13: η ← rσ 21 − η 14: return (cid:99) W l ← S 1 + (cid:113) ηβ ( W − S 1 ) , l
Require: W ∈ R m × n , r , learning rate α , minibatch dataset D 1: Initialize u ∈
R m with a random vector.
2: v ← W (cid:62) u (cid:107) W (cid:62) u (cid:107) , u ← W (cid:62) v (cid:107) W (cid:62) v (cid:107) 3: (cid:46) Perform power iteration 4: σ ( W )
=
u (cid:62) Wv 5: W f = W /σ
( W ) (cid:46)
Spectral Normalization 6: (cid:99) W =
W f − uv (cid:62) 7: if (cid:13)(cid:13)(cid:13)(cid:99) W (cid:13)(cid:13)(cid:13) F ≤ √ r − 1 then 8: return W f 9: end if 10: W f = uv (cid:62) + (cid:99) W √ r − 1 (cid:107) (cid:99) [Table]
Algorithm for Stable Rank Normalization We provide a general procedure in Algorithm 1 to solve the stable rank normalization problem for k ≥ 1 (the solution for k = 0 is straightforward from Theorem 1).
Claim 2 provides the properties of the algorithm.
The algorithm is constructed so that the prior knowledge of the rank of the matrix is not necessary.
Claim 2.
Given a matrix W , the desired stable rank r , and the partitioning index k ≥ 1 , Algorithmrequires computing the top l ( l ≤ k ) singular values and vectors of W .
It returns (cid:99) W l and the scalar l such that srank( (cid:99) W l ) = r , and the top l singular values of W and (cid:99) W l are the same.
If l = k , then the solution provided is the optimal solution to the problem (4) with all the constraints satisﬁed, otherwise, it returns the largest l up to which the spectrum is preserved.
Combining Stable Rank and Spectral Normalization for NNs Following the arguments provided in Section 1 and 3, for better generalizability, we propose to normalize both the stable rank and the spectral norm of each linear layer of a NN simultaneously.
To do so, we ﬁrst perform approximate
SN (Miyato et al., 2018), and then perform optimal SRN (using Algorithm 1).
We use k = 1 to ensure that the ﬁrst singular value (which is now normalized) is preserved.
Algorithm 2 provides a simpliﬁed procedure for the same for a given linear layer of a NN.
Note, the computational cost of this algorithm is exactly the same as that of SN , which is to compute the top singular value using the power iteration method.
E XPERIMENTS
Dataset and Architectures For classiﬁcation, we perform experiments on ResNet-110 (He et al., 2016), WideResNet-28-10 (Zagoruyko & Komodakis, 2016), DenseNet-100 (Huang et al., 2017),
VGG-19 (Simonyan & Zisserman, 2014), and AlexNet (Krizhevsky & Hinton, 2009) using the
CIFAR100 (Krizhevsky & Hinton, 2009) dataset.
In Figure 6, we use a different stopping criterion based on the training accuracy and show that SRN is consistent across other stopping criterion.
We present further experiments with CIFAR10 in Appendix D.1 in Figure 9 to 11.
We train them using standard training recipes with SGD, using a learning rate of 0 .
1 (except AlexNet where we use a learning rate of 0 . 01 ), and a momentum of 0 .
9 with a batch size of 128 (further details in
Appendix D).
For GAN experiments, we use CIFAR100, CIFAR10, and CelebA (Liu et al., 2015) datasets.
We show results on both, conditional and unconditional GANs.
Please refer to Appendix E.1 for further details about the training setup.
Choosing stable rank Given a matrix W ∈ R m × n , the desired stable rank r is controlled using a single hyperparameter c as r = c min( m, n ) , where c ∈ (0 , 1] .
For simplicity, we use the same c for all the linear layers.
It is trivial to note that if c = 1 , or for a given c , if srank( W ) ≤ r , then SRN boils [Figure]
down to SN. For classiﬁcation, we choose c = { 0 . 3 , 0 . 5 } , against (Vanilla) and training with SN.
For GAN experiments, we choose c = { 0 .
7 , 0 . 9 } , and compare SRN-GAN against SN-GAN (Miyato et al., 2018),
WGAN-GP (Gulrajani et al., 2017), and orthonormal regularization GAN (Ortho-GAN) (Brock et al., 2016).
Result Overview
• SRN improves classiﬁcation accuracy on a wide variety of architectures.
• Normalizing stable rank improves the learning capacity of spectrally normalized networks.
• SRN shows remarkably less memorization, even on settings very hard to generalize.
• SRN shows much improved generalization behaviour evaluated using recently proposed sample complexity measures.
• As we decrease the stable rank, the empirical Lipschitz of SRN-GAN decreases.
Proving our arguments provided in Section 3.
• SRN-GAN provides much improved Neural divergence score (ND)
(Gulrajani et al., 2019)
compared to SN-GAN, proving that it is robust to memorization in GANs as well.
• SRN-GAN also provide improved Inception and FID scores in all our experiments (except one where SN-GAN is better).
5. C LASSIFICATION E XPERIMENTS
We perform each experiment 5 times using a new random seed each time and report the mean, and the 75% conﬁdence interval for the test error in Figure 2.
These experiments show that the test accuracy of SRN, on a wide variety NNs, is always higher than the Vanilla and SN (except for
SRN-50 on Alexnet where SRN and SN are almost equal).
However, SN performs slightly worse than Vanilla for WideResNet-28 and ResNet110.
The fact that SRN does involve SN, combined with the above statement, indicate that even though SN reduced the learning capability of these networks, normalizing stable rank must have improved it signiﬁcantly in order for SRN to outperform Vanilla.
For example, in the case of ResNet110, SN is 71 .
5% accurate whereas SRN provides an accuracy of 73 .
In addition to this, we would like to note that even though SN is being used extensively for the training of GANs, it is not a popular choice when it comes to training standard NNs for classiﬁcation.
We suspect that this is because of the decrease in the capacity, which we have shown to be increased by the stable rank normalization, proving the worth of SRN for classiﬁcation tasks as well.
5. S TUDY OF G ENERALIZATION B EHAVIOUR
Our last set of experiments established that SRN provides improved classiﬁcation accuracies on various NNs.
Here we study the generalization behaviour of these models.
Quantifying generalization behaviour is non-trivial and there is no clear answer to it.
However, we utilize recent efforts that explore the theoretical understanding of generalization and use them to study it in practice.
Shattering Experiments To inspect the generalization behaviour in NNs we begin with the shattering experiment (Zhang et al., 2016).
It is a test of whether the network can ﬁt the training data well but not a label-randomized version of it (each image of the dataset is assigned a random label).
As there is no correlation of the labels with the data points P ( y | x ) is essentially uninformative because it is uniformly random.
Thus, the test accuracy on this task is almost 1% .
A high training accuracy — which indicates a high generalization gap (difference between train and test accuracy) can be achieved [Figure] [Figure] only by memorizing the train data 7 .
Figure 3 shows that SRN reduces memorization on random labels (thus, reduces the estimate of the Rademacher complexity (Zhang et al., 2016)).
Note, as shown in the classiﬁcation experiments, the same model was able to achieve the highest training accuracy when the labels were not randomized. [Table]
with and without weight decay.
(Higher is better.)
The clean test accuracy for this setting is shown in Appendix D.1.
We also look speciﬁcally at highly non-generalizable settings — low learning rate and without weight decay .
As shown in Table 1, SRN consistently achieves lower generalization error (by achieving a low train error)
both in the presence and the absence of weight decay 8 . Similar results are reported for Alexnet and WideResNet in Appendix D.1.
[Figure]
Empirical Evaluation of Generalization Behaviour When all the factors in training (eg. architecture, dataset, optimizer, among other) as in SRN vs SN vs Vanilla, are ﬁxed, and the only variability is in the normalization, the generalization error can be written as | Train Err − Test Err | ≤ (cid:101) O (cid:16)(cid:112) C alg / m (cid:17) where (cid:101) O ( · ) ignores the logarithmic terms, m is the number of samples in the dataset, and C alg denotes a measure of sample complexity for a given algorithm.
Lower the value of C alg , the better is the generalization.
Before we give various expressions for C alg , we ﬁrst deﬁne a common quantity in all these expressions, called the margin γ = f θ ( x ) [ y ] − max j (cid:54) =
y f θ ( x ) [ j ] .
It measures the gap between the output of the network on the correct label and the other labels.
Now we deﬁne three recently proposed sample complexity measures useful to quantify the generalization behaviour with further descriptions in Appendix D.1:
• Spec-Fro: (cid:81)
Li =1 (cid:107) W i (cid:107) 22 (cid:80) Li
=1 srank( W i ) /
γ 2 (Neyshabur et al., 2018).
• Spec-L1: (cid:81)
Li =1 (cid:107) W i (cid:107) 22 (cid:32)(cid:80)
Li =1 (cid:107) W
i (cid:107)
[Table]
• Jac-Norm: (cid:80) Li =
[Table] h
i [Table] [Table] J i (cid:107) 2 / γ (Wei & Ma), where h i is the i th hidden layer and J i = ∂γ∂h
i Histogram of the Empirical Lipschitz Constant (eLhist)
We evaluate above mentioned sample complexity measures on 10 , 000 points from the dataset and plot the distribution of the log using a histogram shown in Figure 4.
The more to the left the histogram, the better is the generalization capacity of the network.
For better clarity, we provide the 90 percentile for each of these histograms in Table 5 in Appendix D.1.
As the plots and the table show, both SRN and SN produces a much smaller quantity than a Vanilla network and in 7 out of the 9 cases, SRN is better than SN.
The difference between SRN and SN is much more signiﬁcant in the case of Jac-Norm.
As this depend on the empirical lipschitzness, it provides the empirical validation of our arguments in Section 3.
Above experiments indicate that SRN, while providing enough capacity for the standard classiﬁcation task, is remarkably less prone to memorization and provides improved generalization.
5. T RAINING OF G ENERATIVE A DVERSARIAL N ETWORKS (SRN-GAN)
In GANs, there is a natural tension between the capacity and the generalizability of the discriminator.
The capacity ensures that if the generated distribution and the data distribution are different, the discriminator has the ability to distinguish them.
At the same time, the discriminator has to be generalizable, implying, the class of hypothesis should be small enough to ensure that it is not just memorizing the dataset.
Based on these arguments, we use SRN in the discriminator of GAN which we call SRN-GAN, and compare it against SN-GAN, WGAN-GP, and orthonormal regularization based GAN (Ortho-GAN).
Along with providing results using evaluation metrics such as Inception score (IS) (Salimans et al., 2016) , FID (Heusel et al., 2017), and Neural divergence score (ND) (Gulrajani et al., 2019), we use histograms of the empirical now onwards, for the purpose of analyses.
For a given trained GAN (unconditional), we create 2 , 000 pairs of samples, where each pair ( x i , x j ) consists of x i (randomly sampled from the ‘real’ dataset) and x j (randomly sampled from the generator).
Each pair is then passed through the discriminator to compute (cid:107) f
( x i ) − f ( x j ) (cid:107) 2 / (cid:107) x
i − x j (cid:107) 2 , which we then use to create the histogram.
In the conditional setting, we sample a class from a discrete uniform distribution, and then follow the same approach as described for the unconditional setting. [Table]
Effect of Stable Rank on eLhist and Inception Score As shown in Figure 5a, lowering the value of c (aggressive reduction in the stable rank) moves the histogram towards zero, implying, lower empirical Lipschitz constant.
This validates our arguments provided in Section 3.
Lowering c also improves inception score, however, extreme reduction in the stable rank ( c = 0 . 1 )
dramatically collapses the histogram to zero and also drops the inception score signiﬁcantly.
This is due to the fact that at c = 0 . 1 , the capacity of the discriminator is reduced to the point that it is not able to learn to differentiate between the real and the fake samples anymore.
Table 2 and 3 show that SRN-GAN consistently provide better FID score and an extremely competitive inception score on CIFAR10 (both conditional and unconditional setting) and CIFAR100 (unconditional setting).
In Table 4, we compare the ND loss on CIFAR10 and CelebA datasets.
Note, ND has been looked as a metric more robust to memorization than FID and IS in recent works (Gulrajani et al., 2019; Arora & Zhang, 2017).
We report our exact setting to compute ND in Appendix E.1.
We essentially report the loss incurred by a fresh classiﬁer trained to discriminate the generator distribution and the data distribution.
Thus higher the loss, the better the generated images.
As evident, SRN-GAN provides better ND scores on both datasets.
For a qualitative analysis of the images, we compare generations in both conditional and unconditional setting in Appendix F. [Table]
[Table] Comparing different approaches In addition, in Figure 5b, we provide eLhist for comparing different approaches.
Random-GAN, as expected, has a low empirical Lipschitz constant and extremely poor inception score.
Unsurprisingly, WGAN-GP has a lower L e than Random-GAN, due to its explicit constraint on the Lipschitz constant, while providing a higher inception score.
On the other hand, SRNGAN, by virtue of its softer constraints on the Lipschitz constant, trades off a higher Lipschitz constant for a better inception score—highlighting the ﬂexibility provided by
SRN.
Additional experiments in Appendix E.2 show more detailed behaviour of GANs in regards to empirical lipschitz in a variety of settings.
C ONCLUSION
We proposed a new normalization (SRN) that allows us to constrain the stable rank of each afﬁne layer of a NN, which in turn learns a mapping with low empirical Lipschitz constant.
We also provide optimality guarantees of SRN.
On a variety of neural network architectures, we showed that SRN improves the generalization and memorization properties of a standard classiﬁer.
In addition, we show that SRN improves the training of GANs and provide better inception, FID, and ND scores.
A CKNOWLEDGEMENTS
The authors would like to thank Leonard Berrada and Pwan Kumar for helpful discussions.
AS acknowledges support from The Alan Turing Institute under the Turing Doctoral Studentship grant
TU/C/000023.
PHS and PD are supported by the ERC grant ERC-2012-AdG 321162-HELIOS,
EPSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1.
PHS and PD also acknowledges the Royal Academy of Engineering and FiveAI.
A T ECHNICAL P ROOFS
Here we provide an extensive proof of Theorem 1 (Appendix A.1).
We also provide the optimal solution to the spectral norm problem in Appendix A.2.
Auxiliary lemmas on which our proof depends are provided in Appendix A.3.
A.
Theorem 1.
Given a real matrix W ∈ R m × n with rank p , a target spectrum (or singular value) preservation index k (0 ≤ k < p ) , and a target stable rank of r (1 ≤ r < srank( W )) , the optimal solution (cid:99) W k to problem (4) is (cid:99) W
k
= γ 1 S 1 + γ 2 S 2 , where S 1 = (cid:80) max(1 ,k ) i
=
σ i u i v (cid:62) i and
S 2 = W − S 1 .
{ σ i } ki =1 , { u i } ki =1 and { v i } ki =1 are the top k singular values and vectors of W , and, depending on k , γ 1 and γ 2 are deﬁned below.
For simplicity, we ﬁrst deﬁne γ =
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
, then
a) If k =
(no spectrum preservation), the problem becomes non-convex, the optimal solution to which is obtained for γ 2 = γ + r − 1 r and
γ 1 = γ 2 γ , when r > 1 .
If r = 1 , then γ 2 =and γ 1 = 1 .
Since, in this case, (cid:107) S 1 (cid:107) 2F = σ 21 , γ = √ r − 1 σ 1 (cid:107) S 2 (cid:107) F .
b)
If k ≥ 1 , the problem is convex.
If r ≥ (cid:107) S 1 (cid:107) 2F σthe optimal solution is obtained for γ 1 = 1 , and γ 2 = γ and if not, the problem is not feasible.
c)
Also, (cid:13)(cid:13)(cid:13)(cid:99) W k − W (cid:13)(cid:13)(cid:13) F monotonically increases with k for k ≥ 1 .
Proof.
Here we provide the proof of Theorem 1 (in the main paper) for all the three cases with optimality and uniqueness guarantees.
Let (cid:99) W
k be the optimal solution to the problem for any of the two cases.
From Lemma 5, the SVD of W and (cid:99) W k can be written as W = U Σ V (cid:62) and (cid:99) W k
=
U Λ V (cid:62) , respectively.
Then, L = (cid:13)(cid:13)(cid:13) W − (cid:99) W k (cid:13)(cid:13)(cid:13) 2 F = (cid:104) Σ − Λ , Σ − Λ (cid:105) F .
From now onwards, we denote Σ and Λ as vectors consisting of the diagonal entries, and (cid:104) .
(cid:105) as the vector inner product.
Proof for Case (a): In this case, there is no constraint enforced to preserve any of the singular values of the given matrix while obtaining the new one.
The only constraint is that the new matrix should have the stable rank of r .
Let us assume Σ = ( σ 1 , · · · , σ p ) , Σ 2 = ( σ 2 , · · · , σ p ) , Λ = ( λ 1 , · · · , λ p ) and Λ
= ( λ 2 , · · · , λ p ) .
Using these notations, we can write L as:
[Equation]
Using the stable rank constraint srank( (cid:99) W k )
=
r , which is r = [Equation] +
[Equation]
Case for r > 1 If r > 1 we obtain the following equality constraint, making the problem nonconvex.
[Equation]
However, we will show that the solution we obtain is optimal and unique.
Substituting (7) into (6) we get
[Equation]
Setting ∂L ∂ Λ 2 = 0 to get the family of critical points
[Equation]
As the R.H.S. of 10 is independant of i , the above equality implies that all the critical points of (8) are a scalar multiple of Σ 2 , implying, Λ 2 = γ 2 Σ 2 .
Note that the domain of Λ 2 are all strictly positive vectors and thus, we can ignore the critical point at Λ 2 = 0 .
Substituting this into (9) we obtain
[Equation]
Using the fact that (cid:104) Σ 2 , Σ 2 (cid:105) = (cid:107) S 2 (cid:107) 2F in the above equality and with some algebraic manipulations, we obtain γ 2 = γ + r − 1 r where, γ = √ r − 1 σ 1 (cid:107) S 2 (cid:107) F .
Note, r ≥ 1 , γ ≥ 0 , and Σ ≥ 0 , implying, Λ 2 = γ 2 Σ 2 ≥ 0 .
Local minima: Now, we will show that Λ 2 is indeed a minima of (8).
To show this, we compute the hessian of L .
Recall that
[Equation]
Now we need to show that H at the solution Λ 2 is PSD
i
[Equation]
e. ∀ x ∈
[Equation]
[Equation]
x
[Equation]
H
[Equation]
Λ 2 ) x ≥
[Equation]
r
[Equation]
−
[Equation]
(cid:19) (cid:107)
[Equation]
(cid:107)
[Equation]
( c
[Equation]
r
[Equation]
(cid:18) 1 − γ (
[Equation]
1) (cid:19) (cid:107) x (cid:107)(
[Equation]
(cid:19) (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation] (cid:107) [Equation] Here ( a ) is due to the fact that the matrix ΛΛ (cid:62) is an outer product matrix and is hence PSD.
( b ) follows due to (7) and ( c ) follows by substituting λ 1 = γ 1 σ 1 and then the value of γ 1 .
Finally ( d ) follows as
[Equation]
is decreasing with respect to γ and we know that γ < 1 due to the assumption that srank( W ) < r .
Thus, we can substitute γ = 1 to ﬁnd the minimum value of the expression.
This concludes our proof that Λ 2 is indeed a local minima of L .
Uniqueness: The uniqueness of Λ 2 as a solution to (8) is shown in Lemma 6 and is also guaranteed by the fact that γ 2 has a unique value.
Using Λ 2 = γ 2 Σ 2 and λ 1 = γ 1 σ 1 in (7), we obtain a unique solution γ 1 = γ 2 γ .
Now, we need to show that it is also an unique solution to Theorem 1.
For all solutions to Theorem 1 that have singular vectors which are different than that of W , by Lemma 5, the matrix formed by replacing the singular vectors of the solution with that of W is also a solution.
Thus, if there were a solution with different singular values than (cid:99) W k , it should have appeared as a solution to (8).
However, we have shown that (8) has a unique solution.
Now, we need to show that among all matrices with the same singular values as that of (cid:99) W k , (cid:99) W k is strictly better in terms of (cid:13)(cid:13)(cid:13) W − (cid:99) W k (cid:13)(cid:13)(cid:13) .
This requires a further assumption that every non-zero singular value of Λ 2 has a multiplicity of 1 i.e. they are all distinctly unique.
Intuitively, this doesn’t allow to create a different matrix by simply interchanging the singular vectors associated with the equal singular values.
As the elements of Σ 2 are distinct, the elements of Λ 2 = γ 2 Σ 2 are also distinct and thus by the second part of Lemma 5, (cid:99) W k is strictly better, in terms of (cid:13)(cid:13)(cid:13) W − (cid:99) W k (cid:13)(cid:13)(cid:13) , than all matrices which have the same singular values as that of (cid:100) W k .
This concludes our discussion on the uniqueness of the solution.
Case for r = 1 : Substituting r = 1 in the constraint r = 1 + (cid:80) p
j =2 λ 2 j λwe get r − 1 = (cid:80) pj
[Equation]
j λ=[-
-]
[Equation]
= ⇒ p (cid:88)
j = λ [Equation] =
As it is a sum of squares, each of the individual elements is also zero i.e. λ j = 0 ∀ 2 ≤ j ≤ p .
Substituting this into (6), we get the following quadratic equation in λ
[Equation]
which is minimized at λ 1 = σ 1 , thus proving that γ 1 = 1 and γ 2 = 0 .
Proof for Case (b): In this case, the constraints are meant to preserve the top k singular values of the given matrix while obtaining the new one.
Let Σ
=
( σ 1 , · · · , σ k ) , Σ 2 = ( σ k +1 , · · · , σ p ) , Λ 1 = ( λ 1 , · · · , λ k ) , Λ 2 = ( λ k +1 , · · · , λ p ) .
Since satisfying all the constraints imply Σ 1 = Λ 1 , thus,
L :=
[Equation]
From the stable rank constraint srank( (cid:99) W k ) =
r , we have
[Equation]
The above equality constraint makes the problem non-convex.
Thus, we relax it to srank( (cid:99) W k ) ≤ r to make it a convex problem and show that the optimality is achieved with equality.
Let rσ 21 −(cid:104) Σ 1 , Σ 1 (cid:105)
=
η .
Then, the relaxed problem can be written as
[Equation]
We introduce the Lagrangian dual variables Γ ∈ R p − k and µ corresponding to the positivity and the stable rank constraints, respectively.
The Lagrangian can then be written as
[Equation]
Using the primal optimality condition ∂ L ∂ Λ 2 = 0 , we obtain
[Equation]
Using the above condition on Λ 2 with the constraint (cid:104) Λ 2 , Λ 2 (cid:105) ≤ η , combined with the stable rank constraint of the given matrix W that comes with the problem deﬁnition, srank( W ) > r (which implies (cid:104) Σ 2 , Σ 2 (cid:105) > η ), the following inequality must be satisﬁed for any Γ ≥
[Equation]
For the above inequality to satisfy, the dual variable µ must be greater than zero, implying, (cid:104) Λ 2 , Λ 2 (cid:105)− η must be zero for the complementary slackness to satisfy.
Using this with the optimality condition (14) we obtain
[Equation]
Substituting the above solution back into the primal optimality condition we get
[Equation]
Finally, we use the complimentary slackness condition Γ (cid:12) Λ 2 = 0 11 to get rid of the dual variable Γ as follows
[Equation]
It is easy to see that the above condition is satisﬁed only when Γ = 0 as Σ 2 ≥ 0 and η > 0 .
Therefore, using Γ = 0 in (16) we obtain the optimal solution of Λ 2 as [Equation]
[Equation]
Proof for Case (c): The monotonicity of (cid:13)(cid:13)(cid:13)(cid:99) W k − W (cid:13)(cid:13)(cid:13) F for k ≥ 1 is shown in Lemma 3.
Note that by the assumption that srank( W ) <
r , we can say that γ < 1 .
Therefore in all the cases γ 2 < 1 .
Let us look at the required conditions for γ 1 ≥ 1 to hold.
When k ≥ 1 , γ 1 = 1 holds.
When k = 0 , for γ 1 > 1 to be true, γ 2 < γ should hold, implying, ( γ − 1) <
r ( γ − 1) , which is always true as r > 1 (by the deﬁnition of stable rank).
Lemma 3.
For k ≥ 1 , the solution to the optimization problem (4) obtained using Theorem 1 is closest to the original matrix W in terms of Frobenius norm when only the spectral norm is preserved, implying, k = 1 .
Proof.
For a given matrix W and a partitioning index k ∈ { 1 , · · · , p } , let (cid:99) W
k
= S k
+ γ S k 2 be the matrix obtained using Theorem 1.
We use the superscript k along with S 1 and S 2 to denote that this refers to the particular solution of (cid:99) W k .
Plugging the value of γ and using the fact that (cid:13)(cid:13)
S k 2 (cid:13)(cid:13) F (cid:54) = 0 , we can write
[Equation]
Thus, (cid:13)(cid:13) (cid:13) W − (cid:99) W k
(cid:13)(cid:13) (cid:13) F can be written in a simpliﬁed form as f ( x ) = x − √ a + x 2 , where x = (cid:13) (cid:13)
S k 2 (cid:13) (cid:13) F and a = rσ 21 −(cid:107) W (cid:107) 2F .
Note, a ≤ 0 as 1 ≤ r ≤ srank( W ) , and a + x 2 ≥ 0 because of the condition in Theorem 1.
Under these settings, it is trivial to verify that f is a monotonically decreasing function of x .
Using the fact that as the partition index k increases, x decreases, it is straightforward to conclude that the minimum of f ( x ) is obtained at k = 1 .
A. P ROOF FOR O PTIMAL S PECTRAL N ORMALIZATION
The widely used spectral normalization (Miyato et al., 2018) where the given matrix W ∈ R m × n is divided by the maximum singular value is an approximation to the optimal solution of the spectral normalization problem deﬁned as
[Equation]
where σ ( (cid:99) W ) denotes the maximum singular value and s > 0 is a hyperparameter.
The optimal solution to this problem is shown in Algorithm 3.
In what follows we provide the optimality proof
Algorithm
Require:
W ∈ R m × n , s 1: W 1 ← 0 , p ← min( m n ) 2: for k ∈ { · · · , p do 3: power then [Equation]
[Equation]
7: else 8: break
9: end if 10: end for 11: return ← of Algorithm 3 for the sake of completeness.
Let SVD ( W ) =
U Σ V (cid:62) and let us assume that
Z = S Λ T (cid:62) is a solution to the problem 18.
Trivially, X = U Λ V (cid:62) also satisﬁes σ ( X ) ≤ s .
Now, (cid:107) W
[Equation]
(cid:107)
[Equation]
2F ≤
[Equation]
− Z (cid:107) 2F , where the last inequality directly comes from Lemma 4.
Thus the singular vectors of the optimal solution must be the same as that of
W .
This boils down to solving the following problem
[Equation]
Here, without loss of generality, we abuse notations by considering Λ and Σ to represent the diagonal vectors of the original diagonal matrices Λ and Σ , and Λ [ i ] as its i -th index.
It is trivial to see that the optimal solution with minimum Frobenius norm is achieved when
[Equation]
This is exactly what Algorithm 3 implements.
A. A UXILIARY L EMMAS
Lemma 4.
[Reproduced from Theorem 5 in Mirsky (1960)]
For any two matrices A , B ∈ R m × n with singular values as σ 1 ≥ · · · ≥ σ n and ρ 1 ≥ · · · ≥ ρ n , respectively
[Equation]
Proof.
Consider the following symmetric matrices
X = (cid:20)
A
A (cid:62) (cid:21) , Y = (cid:20)
B
B (cid:62) (cid:21) , Z = (cid:20)
A − B ( A − B ) (cid:62) (cid:21)
Let τ 1 ≥ · · · ≥ τ n be the singular values of Z .
Then the set of characteristic roots of X , Y and Z in descending order are { ρ 1 , · · · , ρ n , − ρ n , · · · , − ρ 1 } , { σ 1 , · · · , σ n , − σ n , · · · , − σ 1 } , and { τ 1 , · · · , τ n , − τ n , · · · , − τ 1 } , respectively.
By Lemma 2 in Wielandt (1955)
[Equation]
which implies that
[Equation]
Lemma 5.
Let A , B ∈ R m × n where SVD( A ) =
U Σ V (cid:62) and B is the solution to the following problem
[Equation]
Then, SVD ( B ) =
U Λ V (cid:62) where Λ is a diagonal matrix with non-negative entries.
Implying, A and
B will have the same singular vectors.
Proof.
Let us assume that Z = S Λ T (cid:62) is a solution to the problem 21 where S (cid:54) =
U and T (cid:54)
= V .
Trivially, X = U Λ V (cid:62) also lies in the feasible set as it satisﬁes srank( X ) =
r (note stable rank only depends on the singular values).
Using the fact that the Frobenius norm is invariant to unitary transformations, we can write (cid:107)
[Equation]
(cid:13)(cid:13) 2 F = (cid:107) (Σ − Λ) (cid:107) 2F .
Combining this with Lemma 4, we obtain (cid:107) A − X (cid:107) 2F
=
(cid:107) (Σ − Λ) (cid:107) 2F ≤ (cid:107)
A − Z (cid:107) 2F .
Since, S (cid:54) =
U and T (cid:54)
=
V , we can further change ≤ to a strict inequality < .
This completes the proof.
Generally speaking, the optimal solution to problem 21 with constraints depending only on the singular values ( e.g. stable rank in this case) will have the same singular vectors as that of the original matrix.
Further the inequality in (20) can be converted into a strict inequality if neither of A and B have repeated singular values.
Using that strict inequality, if both Σ and Λ have no repeated values, then B is the only solution to (21) that has the singular values of Λ .
Lemma 6.
Let y
= a x 1 + b ˆ x
and y
= a x 2 + b ˆ x 2 , where ˆ x 1 and ˆ x 2 denotes the unit vectors.
Then, y 1 =
y 2 if x 1 = x 2 .
B E MPIRICAL L IPSCHITZ CONSTANT B. R ELATING EMPIRICAL LOCAL AND GLOBAL L IPSCHITZ CONSTANTS
Proposition B.1.
Let f : R m (cid:55)→ R be a Fréchet differentiable function, D the dataset, and
Conv ( x i , x j ) denotes the convex combination of a pair of samples x i and x j , then ∀ p, q ∈ [1 , ∞ ] such that 1 p + 1 q
[Equation]
Proof.
Let f : R m → R be a differentiable function on an open set containing x i and x j such that x
i (cid:54) =
x j .
By applying fundamental theorem of calculus
[Equation]
The inequality (a) is due to Hölder’s inequality.
B. E FFECT OF R ANK ON THE E MPIRICAL L IPSCHITZ C ONSTANTS
Let f ( x ) =
W 2 W 1 x be a two-layer linear NN with weights W 1 and W 2 .
The Jacobian in this case is independent of x .
Thus, the local Lipschitz constant is the same for all x ∈ R m , implying, local
L e =
L l ( x ) =
L l = (cid:107) W 2 W 1 (cid:107) ≤ (cid:107) W 2 (cid:107) (cid:107) W 1 (cid:107) .
Note, in the case of 2-matrix norm reducing the rank will not affect the upperbound.
However, as will be discussed below, rank reduction greatly inﬂuences the global L e .
Let x i and x j be random pairs from D and ∆ x (cid:54) =
be the difference x
i − x j , then, the global L e is max
[Equation]
Let k 1 and k 2 be the ranks, and σ 1 ≥ · · · ≥ σ k 1 and λ 1 ≥ · · · ≥ λ kthe singular values of the matrices W 1 and W 2 , respectively.
Let P i =
u
i ¯ u (cid:62) i be the orthogonal projection matrix corresponding to u i and ¯ u i , the left and the right singular vectors of W 1 .
Similarly, we deﬁne Q i for W 2 corresponding to v i and ¯ v i .
Then, W 2 W 1 = (cid:80)
k 2 i =1 (cid:80) k 1 j =
λ i σ j Q
i P j .
The upperbound, λ 1 σ 1 , can be achieved if and only if ∆ x = ¯ u 1 (cid:107) ∆ x (cid:107) and u 1 = ¯ v 1 (a perfect alignment), which is highly unlikely.
In practice, not just the maximum singular values, as is the case with the
Lipschitz upper-bound, rather the combination of the projection matrices and the singular values play a crucial role in providing an estimate of global L e .
Thus, reducing the singular values, which is equivalent to minimizing the rank (or stable rank), will directly affect L e .
For example, assigning σ j = 0 , which in effect will reduce the rank of W 1 by one, will nullify its inﬂuence on all projections associated with P j .
Implying, all the k 2 projections σ j ( (cid:80) k
i
=
λ i Q i )
P j that would propagate the input via P j will be blocked.
This, in effect, will inﬂuence (cid:107) W 2 W 2 ∆ x (cid:107) ; hence the global L e .
In a more general setting, let k i be the rank of the
i -th linear layer, then, each singular value of a j -th layer can inﬂuence the maximum of (cid:81) j − 1 i =
1 k i (cid:81) li =
j +1 k
i many paths through which an input can be propagated.
Thus, mappings with low rank (stable) will greatly reduce the global L e .
Similar arguments can be drawn for local L e in the case of NN with non-linearity.
C T HE LOCAL L IPSCHITZ UPPER - BOUND FOR N EURAL N ETWORKS
As mentioned in Section 2, L l ( x ) = (cid:107) J f ( x ) (cid:107) p,q , where, in the case of NN, the Jacobian is:
[Equation]
Using ∂ z l ∂ a l − 1 = W l (afﬁne transformation), and applying submultiplicativity of the matrix norms:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Note, most commonly used activation functions φ ( . ) such as ReLU, sigmoid, tanh and maxout are known to have Lipschitz constant of 1 (if scaled appropriately) 12 , thus, the upper bound can further be written only using the operator norms of the intermediate matrices as
[Equation]
Furthermore L l ( x ) can be substituted by L l , the local Lipschitz constant, as the upper bound (Eq.
(24)) is independent of x .
Note that this is one of the main reasons why we consider the empirical Lipschitz to better reﬂect the true behaviour of the function as the NN is never exposed to the entire domain R m but only a small subset dependant on the data distribution.
The other reason why this upper bound is a bad estimate is that the inequality in Eq (23) is tight only when the partial derivatives are aligned, implying
[Equation]
(cid:96) ≤ l .
This problem has been referred to as the problem of mis-alignment and is similar to quantities like layer cushion in Arora et al.
D E XPERIMENTAL DETAILS
WideResNet-28-
We use a standard WideResNet with 28 layers and a growth factor of 10 .
In total, the network has 36,539,124 trainable parameters.
The network is the standard conﬁguration with batchnorm and ReLU activations and is trained with a weight decay of 1 e − 4 .
The learning rate was multiplied by 0 .
2 after 60 , 120 , and 160 epochs respectively.
ResNet-
The ResNet-110 is a standard 110 layered ResNet with batch Norm and ReLU and has 1 , 973 , 236 parameters.
The network is trained with SGD, an initial learning rate of 0 . 1 , which is multiplied 0 .
1 after 150 and 250 epochs respectively, a weight decay of 5 e − 4 and a momentum of 0 .
Densenet-
The DenseNet-100 is a standard 100 -layered densenet with Batchnorm and ReLU and has a total of 800 , 032 trainable parameters.
The network is trained with SGD, an initial learning rate of 0 . 1 , which is multiplied by 0 .
1 after 150 and 250 epochs respectively, a weight decay of 1 e − 4 , and a momentum of 0 .
VGG19 The VGG19 model is the standard 19-layered VGG model with Batchnorm and ReLU.
It has a total of 20 , 548 , 392 trainable parameters and is trained with SGD with a momentum of 0 .and
a weight decay of 5 e − 4 .
The initial learning rate is 0 . 1 and is multiplied by 0 .
1 after 150 and 250 epochs respectively.
For the shattering experiments, we used the same architecture and the same training recipe except the initial learning rate, which was deceased to 0 .
01 as the model failed to learn the random labels with a large learning rate.
AlexNet
The Alexnet model is the standard ALexNet model with 4 , 965 , 092 trainable parameters.
It was trained with SGD, with a momentum of 0 . 9 , with an initial learning rate is 0 . 01 , which is multiplied by 0 .
1 after 150 and 250 epochs respectively.
The optimizer was further augmented with a weight decay rate of 5 e − 4 .
Please refer to the next section for results on different learning rates and with and without weight decay.
D. A DDITIONAL E XPERIMENTS ON G ENERALIZATION
Stopping Criterion In this section we show that our regularizor performs consistently for a different stopping criterion.
In particular, we use the train accuracy as a stopping criterion.
For Resnet110,
WideResnet-28,Densenet-100, and VGG-19 we use a train accuracy of 99% as a stopping criterion and report the test accuracy when that train accuracy was achieved for the ﬁrst time.
For Alexnet, as
SRN-30 never achieves a train accuracy higher than 55% , we use 55% as the stopping criterion and plot the test accuracies in Figure 6.
Our results show that SRN-30 and SRN-50 outperform SN and vanilla consistently.
In Figure 9, we show similar plots for CIFAR10.
Complexity measures In this section, we provide more details about the various complexity measures we used in Figure 4.
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure]
• Spec-Fro: (cid:81)
Li =1 (cid:107) W i (cid:107) 22 (cid:80) Li
=1 srank( W i ) γ(Neyshabur et al., 2018).
This bound is the main motivation of this paper; the two quantities used to normalize the margin ( γ ) is the product of spectral norm i.e. (cid:81)
Li =1 (cid:107) W i (cid:107) 22 (or worst case
[Figure]
) and sum of stable rank i.e. , (cid:80) Li =1 srank( W i ) (or an approximate parameter count like rank of a matrix).
• Spec-L1: (cid:81)
Li =1 (cid:107) W i (cid:107)(cid:18) (cid:80) Li =1 (cid:107)
[Equation] W i [Equation]  (cid:19)γ, where (cid:107) .
(cid:107) 2 , 1 is the matrix 2-1 norm.
As showed by Bartlett et al.
(2017), Spec-L1 is the spectrally normalized margin, and unlike just the margin, is a good indicator of the generalization properties of a network.
• Jac-Norm: (cid:80)
Li =1 (cid:107) h
i (cid:107) 2 (cid:107) J i (cid:107) 2 γ (Wei & Ma), where h i is the i th hidden layer and J i = ∂γ∂h
i i.e. , the Jacobian of the margin with respect to the i th hidden layer (thus, a vector).
Note, Jac-Norm depends on the norm of the Jacobian (local empirical Lipschitz
and norm of the hidden layers - additional data-dependent terms compared to Spec-Fro and Spec-L1, thus captures a more realistic (and optimistic) generalization behaviour.
For better clarity regarding Figure 4, we provide the 90 percentile for each of these histograms in
Table 5.
As the plots and the table show, both SRN and SN produces a much smaller quantity than a
Vanilla network and in 7 out of the 9 cases, SRN is better than SN.
The difference between SRN and
SN is much more signiﬁcant in the case of Jac-Norm.
As this depend on the empirical lipschitzness, it provides the empirical validation of our arguments in Section 3.
[Table]
Alexnet experiments: Figure 7 shows the test error and generalization error of Alexnet trained with a large learning rate of 0 . 1 .
Note that, the model fails to learn completely without weight decay.
Generalisation Error decreases monotonically with decreasing c in the stable rank constraint error is the lowest for c = 0 .
The constraint becomes too aggressive for even c lower than that.
The slightly more interesting observation is that having a weight decay actually hurts generalization error while it has a slightly positive effect on test error.
[Figure]
Low Learning Rate Here, we train a WideResnet-28-10 with SRN, SN, and vanilla methods with an lr = 0 .
01 and weight decay of 5 × 10 − 4 on randomly labelled CIFAR100.
for 50 epochs.
The results are shown in Table 6 and it further supports that SRN is more robust to random noise than SN or vanilla methods. [Figure] [Table]
[Figure]
With and without weight decay In Figure 8a, we show the training error of Alexnet trained with
SGD with and without weight decay ( = 5 e − 4 ) with a learning rate of 0 . 01 .
Again, we see that a more aggressive stable rank constraint decreases ﬁtting the random data .
Similar results are seen for
ResNet-110 in Figure 8b.
[Figure]
[Figure] [Figure]
Low Learning Rate, with and without weight decay on clean CIFAR
In Appendix D.1, we show the test accuracies for the clean data with the same conﬁfuration as in Table 1.
This corresponds to the hihgly non-generelizable learning setting.
Vanilla
Spectral
StableStable-
W/o WD 69.2 ± 0.
69 ± 0.
69.1 ± 0.
69.3 ± 0.
With WD 70.4 ± 0.
71.35 ± 0.
70.6 ± 0. 70.6 ± 0.
CIFAR10 experiments In this section, we plot results on CIFAR10 trained using ResNet-110,
Desnenet100, WideResNet-28, and Alexnet.
In Figure 9, we plot the test accuracy on clean CIFARwith the training accuracy as the stopping criterion.
For all models other than Alexnet, we use 99% training accuracy as the criterion and for Alexnet we use 85% .
In Figure 10, we plot the test accuracy on clean CIFAR10 using the number of epochs as the stopping criterion.
The results here are consistent with those in the main paper in that SRN outperforms the vanilla and SN.
In Figure 11, we plot the training accuracy on CIFAR10 when the labels are randomized for Resnet100, and Alexnet.
SRN-50 and SRN-30 are much better than Vanilla and SN in this case.
[Figure]
E A DDITIONAL E XPERIMENTS ON GAN S E. GAN EXPERIMENTAL SETUP
Datasets and Network Architectures Each of the CIFAR datasets contain a total of 50 , 000 RGB images in the training set, where each image is of size 32 × 32 , and a further 10 , 000 RGB images of the same dimension in the test set.
The CelebA dataset contains more than 200K images scaled to a size of 64 × 64 .
The model architecture for both the generator and the discriminator was chosen to be a 32 layered ResNet (He et al., 2016) due to its previous superior performance in other works (Miyato et al., 2018).
We use Adam optimizer (Kingma & Ba, 2014) which depends on three main hyperparameters α - the initial learning rate, β 1 - the ﬁrst order moment decay rate and β 2 - the second order moment decay rate.
We cross-validate these parameters in the set α ∈ { 0 . 0002 , 0 . 0005 } , β 1 ∈ { 0 , 0 . 5 } , β 2 ∈ { 0 . 9 , 0 . 999 } and chose α = 0 . 0002 , β 1 = 0 . 0 and β 2 = 0 .
999 which performed consistently well in all of the experiments.
GAN objective functions In the case of conditional GANs (Mirza & Osindero, 2014), we used the conditional batch normalization (Dumoulin et al., 2017) to condition the generator and the projection discriminator (Miyato & Koyama, 2018) to condition the discriminator.
The dimension of the latent variable for the generator was set to 128 and was sampled from a zero mean and unit variance
Gaussian distribution.
For training the model, we used the hinge loss version of the adversarial loss (Lim & Ye, 2017; Tran et al., 2017) in all experiments except the experiments with WGAN-GP.
The hinge loss version was chosen as it has been shown to give consistently better performance in previous works (Zhang et al., 2018;
Miyato et al., 2018).
For training the WGAN-GP model, we used the original loss function as described in Gulrajani et al.
Evaluation Metrics We use Inception (Salimans et al., 2016) and Frechet Inception Distance (FID) (Heusel et al., 2017) scores for the evaluation of the generated samples.
For measuring the inception score, we generate 50 , 000 samples, as was recommended in Salimans et al.
For measuring FID, we use the same setting as Miyato et al.
(2018) where we sample 10 , 000 data points from the training set and compare its statistics with that of 5 , 000 generated samples.
In addition, we use a recent evaluation metric called Neural divergence score Gulrajani et al.
(2019) which is more robust to memorization.
The exact set-up for the same is discussed below.
In the case of conditional image generation, we also measure Intra-FID (Miyato et al., 2018), which is the mean of the FID of the generator, when it is conditioned over different classes.
Let FID( G , c ) be the FID of the generator G when it is conditioned on the class c ∈ C (where C is the set of classes), then,
Intra FID( G ) = 1 |C| FID( G , c )
Neural Divergence Setup We train a new classiﬁer inline with the architecture in Gulrajani et al.
It includes three convolution layers with 16, 32 and 64 channels, a kernel size of 5 ×and a stride of 2 .
Each of these layers are followed by a Swish activation (Ramachandran et al., 2018) and then ﬁnally a linear layer that gives a single output.
The network is initialized using normal distribution with zero mean and the standard deviation of 0 .
02 , and trained using Adam optimizer with α = 0 . 0002 , β 1 = 0 .
, β 2 = 0 .
9 for a total of 100 , 000 iterations with minibatch of 128 generated samples and 128 samples from the test set 13 .
We use the standard WGAN-GP loss function, log (1 + exp ( f ( x fake )))
+ log (1 + exp ( − x real )) , where f represents the network described above.
Finally, we generate 1 Million samples from the generator and report the average log (1 + exp ( f ( x fake ))) over these samples.
Higher average value implies better generation as the network in this case is unable to distinguish the generated and the real samples.
E. M ORE E MPIRICAL L IPSCHITZ PLOTS
For the purpose of analysis, Figure 13b and 14b shows eLhist for pairs where each sample either comes from the true data or from the generator, and we observe a similar trend.
To verify that same results hold in the conditional setup, we show comparisons for GANs with projection discriminator (Miyato & Koyama, 2018) in Figure 12, 13a and 14a, and observe a similar trend.
Further, to see the value of the local Lipschitzness in the vicinity of real and generated samples we also plot the norm of the
Jacobian in Figure 15 and 16 in Appendix E.2 and observe mostly a similar trend.
In Appendix E.
(Figure 17), we also show that the discriminator training of SRN-GAN is more stable than SN-GAN.
Conditional GANs
Figure 12 shows the eLihst of conditional GANs with projection discriminator (Miyato & Koyama, 2018).
Empirical Lipschitzness between real samples and between fake samples.
Figure 13 shows the histogram of eLhist of the discriminator for pairs of fake samples i.e. samples generated by the generator.
Figure 14 shows eLhist of the discriminator when samples came from the dataset.
[Figure]
[Figure]
Jacobian norm in the vicinity of the points Here we compare the Jacobian of the discriminator of the trained models in the vicinity of the samples from the generator and the real dataset.
This is a penalized measure in various algorithms Gulrajani et al.
; Petzka et al.
(2018) (often referred to as local perturbations) and was independently proposed by Kodali et al.
Figure and Figure 16 show the histogram of the norm of the Jacobian of the discriminator in the vicinity of the generated and the real samples, respectively.
To generate these plots, 2 , 000 samples were used from the respective distributions.
It is interesting to note that the norm is the same for the points in the vicinity of the real data points and the generated data points for the Stable Rank Normalization GAN (SRN-GAN) as well for WGAN-GP whereas it varies between fake and real samples for Spectral
Normalization GAN (SN-GAN).
[Figure]
[Figure]
E. T RAINING S TABILITY
Training Stability In Figure 17 we show the discriminator loss during the course of the training as an indicator of whether the generator gets sufﬁcient gradient during training or not.
These plots clearly suggest that the discriminator loss is more consistent for SRN than the SN.
F E XAMPLES OF G ENERATED I MAGES F. C ELEB A IMAGES
For these images, we generated 100 images from the respective models and hand-picked the 10 best images in terms of visual quality.
[Figure]
F. CIFAR10-C ONDITIONAL SRN-GAN
[Figure]