T RUTH OR BACKPROPAGANDA ? A N EMPIRICAL INVESTIGATION OF DEEP LEARNING THEORY
A BSTRACT
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike.
In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to ﬁnd examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) ﬁnd that rank does not correlate with generalization or robustness in a practical setting.
I NTRODUCTION
Modern deep learning methods are descendent from such long-studied ﬁelds as statistical learning, optimization and signal processing, all of which were built on mathematically rigorous foundations.
In statistical learning, principled kernel methods have vastly improved the performance of SVMs and
PCA (Suykens & Vandewalle, 1999; Sch¨olkopf et al., 1997), and boosting theory has enabled weak learners to generate strong classiﬁers (Schapire, 1990).
Optimizers in deep learning are borrowed from the ﬁeld of convex optimization , where momentum optimizers (Nesterov, 1983) and conjugate gradient methods provably solve ill-conditioned problems with high efﬁciency (Hestenes & Stiefel, 1952).
Deep learning harnesses foundational tools from these mature parent ﬁelds.
Despite its rigorous roots, deep learning has driven a wedge between theory and practice.
Recent theoretical work has certainly made impressive strides towards understanding optimization and generalization in neural networks.
But doing so has required researchers to make strong assumptions and study restricted model classes.
In this paper, we seek to understand whether deep learning theories accurately capture the behaviors and network properties that make realistic deep networks work.
Following a line of previous work, such as Swirszcz et al.
(2016), Zhang et al.
(2016), Balduzzi et al.
(2017) and Santurkar et al.
(2018), we put the assumptions and conclusions of deep learning theory to the test using experiments with both toy networks and realistic ones.
We focus on the following important theoretical issues:
• Local minima: Numerous theoretical works argue that all local minima of neural loss functions are globally optimal or that all local minima are nearly optimal.
In practice, we ﬁnd highly suboptimal local minima in realistic neural loss functions, and we discuss reasons why suboptimal local minima exist in the loss surfaces of deep neural networks in general.
• Weight decay and parameter norms: Research inspired by Tikhonov regularization suggests that low-norm minima generalize better, and for many, this is an intuitive justiﬁcation for simple regularizers like weight decay.
Yet for neural networks, it is not at all clear which form of (cid:96) 2 -regularization is optimal.
We show this by constructing a simple alternative: biasing solutions toward a non-zero norm still works and can even measurably improve performance for modern architectures.
• Neural tangent kernels and the wide-network limit: We investigate theoretical results concerning neural tangent kernels of realistic architectures.
While stochastic sampling of the tangent kernels suggests that theoretical results on tangent kernels of multi-layer networks may apply to some multi-layer networks and basic convolutional architectures, the predictions from theory do not hold for practical networks, and the trend even reverses for ResNet architectures. We show that the combination of skip connections and batch normalization is critical for this trend in ResNets.
• Rank: Generalization theory has provided guarantees for the performance of low-rank networks.
However, we ﬁnd that regularization which encourages high-rank weight matrices often outperforms that which promotes low-rank matrices.
This indicates that low-rank structure is not a signiﬁcant force behind generalization in practical networks.
We further investigate the adversarial robustness of low-rank networks, which are thought to be more resilient to attack, and we ﬁnd empirically that their robustness is often lower than the baseline or even a purposefully constructed high-rank network.
L OCAL MINIMA IN LOSS LANDSCAPES : D O SUBOPTIMAL MINIMA EXIST ?
It is generally accepted that “in practice, poor local minima are rarely a problem with large networks.”
(LeCun et al., 2015).
However, exact theoretical guarantees for this statement are elusive.
Various theoretical studies of local minima have investigated spin-glass models (Choromanska et al., 2014), deep linear models (Laurent & Brecht, 2018; Kawaguchi, 2016), parallel subnetworks (Haeffele & Vidal, 2017), and dense fully connected models (Nguyen et al., 2018) and have shown that either all local minima are global or all have a small optimality gap.
The apparent scarcity of poor local minima has lead practitioners to develop the intuition that bad local minima (“bad” meaning high loss value and suboptimal training performance)
are practically non-existent.
To further muddy the waters, some theoretical works prove the existence of local minima.
Such results exist for simple fully connected architectures (Swirszcz et al., 2016), single-layer networks (Liang et al., 2018; Yun et al., 2018), and two-layer ReLU networks (Safran & Shamir, 2017).
For example, (Yun et al., 2019) show that local minima exist in single-layer networks with univariate output and unique datapoints.
The crucial idea here is that all neurons are activated for all datapoints at the suboptimal local minima.
Unfortunately, these existing analyses of neural loss landscapes require strong assumptions (e.g. random training data, linear activation functions, fully connected layers, or extremely wide network widths) — so strong, in fact, that it is reasonable to question whether these results have any bearing on practical neural networks or describe the underlying cause of good optimization performance in real-world settings.
In this section, we investigate the existence of suboptimal local minima from a theoretical perspective and an empirical one.
If suboptimal local minima exist, they are certainly hard to ﬁnd by standard methods (otherwise training would not work).
Thus, we present simple theoretical results that inform us on how to construct non-trivial suboptimal local minima, concretely generalizing previous constructions, such as those by (Yun et al., 2019).
Using experimental methods inspired by theory, we easily ﬁnd suboptimal local minima in the loss landscapes of a range of classiﬁers.
Trivial local minima are easy to ﬁnd in ReLU networks – consider the case where bias values are sufﬁciently low so that the ReLUs are “dead” (i.e. inputs to ReLUs are strictly negative).
Such a point is trivially a local minimum.
Below, we make a more subtle observation that multilayer perceptrons (MLPs) must have non-trivial local minima, provided there exists a linear classifer that performs worse than the neural network (an assumption that holds for virtually any standard benchmark problem).
Speciﬁcally, we show that MLP loss functions contain local minima where they behave identically to a linear classiﬁer on the same data.
We now deﬁne a family of low-rank linear functions which represent an MLP.
Let “rank- s afﬁne function” denote an operator of the form G ( x ) =
A x + b with rank ( A ) = s .
Deﬁnition 2.1.
Consider a family of functions, { F φ : R m → R n } φ ∈ R P parameterized by φ.
We say this family has rank- s afﬁne expression if for all rank- s afﬁne functions G : R m → R n and ﬁnite subsets Ω ⊂ R m , there exists φ with F φ ( x ) =
G ( x ) , ∀ x ∈ Ω .
If s = min( n, m ) we say that this family has full afﬁne expression .
We investigate a family of L-layer MLPs with ReLU activation functions, { F φ : R m → R n } φ ∈ Φ , and parameter vectors φ , i.e., φ =
[Equation] [Equation]
1 , A
[Equation]
, b
[Equation]
[Equation]
) , F
[Equation]
x ) =
H L ( f ( H L − 1 ...
f ( H 1 ( x )))) , where f denotes the ReLU activation function and H i ( z )
=
A i z
+ b i .
Let A i ∈ R n i × n i − 1 , b i ∈ R n i with n 0 = m and n L = n .
Lemma 1.
Consider a family of L-layer multilayer perceptrons with ReLU activations { F φ : R m → R n } φ ∈ Φ , and let s =
min
i n i be the minimum layer width.
Such a family has rank- s afﬁne expression.
Proof.
The idea of the proof is to use the singular value decomposition of any rank- s afﬁne function to construct the MLP layers and pick a bias large enough for all activations to remain positive.
See
Appendix A.1.
The ability of MLPs to represent linear networks allows us to derive a theorem which implies that arbitrarily deep MLPs have local minima at which the performance of the underlying model on the training data is equal to that of a (potentially low-rank) linear model.
In other words, neural networks inherit the local minima of elementary linear models.
Theorem 1.
Consider a training set, { ( x i , y i ) } Ni =1 , a family { F φ } φ of MLPs with s =
min i n i being the smallest width.
Consider a parameterized afﬁne function G A, b solving min A, b L ( G A, b ; { ( x i , y i ) } Ni =1 ) , subject to rank ( A ) ≤ s, (1) for a continuous loss function L .
Then, for each local minimum, ( A (cid:48) , b (cid:48) ) , of the above training problem, there exists a local minimum, φ (cid:48) , of the MLP loss L ( F φ ; { ( x i , y i ) } Ni =1 ) with the property that F φ (cid:48)
( x i ) =
G A (cid:48) , b (cid:48)
( x i ) for i = 1 , 2 , ..., N .
Proof.
See appendix A.2.
The proof of the above theorem constructs a network in which all activations of all training examples are positive, generalizing previous constructions of this type such as Yun et al.
(2019) to more realistic architectures and settings.
Another paper has employed a similar construction concurrently to our own work (He et al., 2020).
We do expect that the general problem in expressivity occurs every time the support of the activations coincides for all training examples, as the latter reduces the deep network to an afﬁne linear function (on the training set), which relates to the discussion in Balduzzi et al.
We test this hypothesis below by initializing deep networks with biases of high variance.
Remark 2.1 (CNN and more expressive local minima) .
Note that the above constructions of Lemma 1 and Theorem 1 are not limited to MLPs and could be extended to convolutional neural networks with suitably restricted linear mappings G φ by using the convolution ﬁlters to represent identities and using the bias to avoid any negative activations on the training examples.
Moreover, shallower MLPs can similarly be embedded into deeper MLPs recursively by replicating the behavior of each linear layer of the shallow MLP with several layers of the deep MLP.
Linear classiﬁers, or even shallow MLPs, often have higher training loss than more expressive networks.
Thus, we can use the idea of Theorem 1 to ﬁnd various suboptimal local minima in the loss landscapes of neural networks.
We conﬁrm this with subsequent experiments.
We ﬁnd that initializing a network at a point that approximately conforms to Theorem 1 is enough to get trapped in a bad local minimum.
We verify this by training a linear classiﬁer on CIFAR-10 with [Table] weight decay, (which has a test accuracy of [Table]
[Table] of
[Table] [Table] to the logistic regression objective). We then initialize a multilayer network as described in Lemma 1 to approximate this linear classiﬁer and recompute these statistics on the full network (see Table 1).
When training with this initialization, the gradient norm drops futher, moving parameters even closer to the linear minimizer.
The ﬁnal training result still yields positive activations for the entire training dataset.
Moreover, any isolated local minimum of a linear network results in many local minima of an MLP
F φ (cid:48) , as the weights φ (cid:48) constructed in the proof of Theorem 1 can undergo transformations such as scaling, permutation, or even rotation without changing F φ (cid:48) as a function during inference, i.e.
F φ (cid:48)
( x ) =
F φ ( x ) for all x for an inﬁnite set of parameters φ , as soon as F has at least one hidden layer.
While our ﬁrst experiment initializes a deep MLP at a local minimum it inherited from a linear one to empirically illustrate our ﬁndings of Theorem 1, Table 1 also illustrates that similarly bad local minima are obtained when choosing large biases (third row) and choosing biases with large variance (fourth row) as conjectured above.
To signiﬁcantly reduce the bias, however, and still obtain a subpar optimum, we need to rerun the experiment with SGD without momentum, as shown in the last row, reﬂecting common intuition that momentum is helpful to move away from bad local optima. Remark 2.2 (Sharpness of sub-optimal local optima) .
An interesting additional property of minima found using the previously discussed initializations is that they are “sharp”.
Proponents of the sharpﬂat hypothesis for generalization have found that minimizers with poor generalization live in sharp attracting basins with low volume and thus low probability in parameter space (Keskar et al., 2016;
Huang et al., 2019), although care has to be taken to correctly measure sharpness (Dinh et al., 2017).
Accordingly, we ﬁnd that the maximum eigenvalue of the Hessian at each suboptimal local minimum is signiﬁcantly higher than those at near-global minima. For example, the maximum eigenvalue of the initialization by Lemma 1 in Table 1 is estimated as 113 , 598 . 85 after training, whereas that of the default initialization is only around 24 . 01 . While our analysis has focused on sub-par local optima in training instead of global minima with sub-par generalization, both the scarcity of local optima during normal training and the favorable generalization properties of neural networks seem to correlate with their sharpness. In light of our ﬁnding that neural networks trained with unconventional initialization reach suboptimal local minima, we conclude that poor local minima can readily be found with a poor choice of hyperparameters.
Suboptimal minima are less scarce than previously believed, and neural networks avoid these because good initializations and stochastic optimizers have been ﬁne-tuned over time.
Fortunately, promising theoretical directions may explain good optimization performance while remaining compatible with empirical observations.
The approach followed by Du et al.
(2019) analyzes the loss trajectory of SGD, showing that it avoids bad minima.
While this work assumes (unrealistically) large network widths, this theoretical direction is compatible with empirical studies, such as Goodfellow et al.
(2014), showing that the training trajectory of realistic deep networks does not encounter signiﬁcant local minima.
W EIGHT DECAY : A RE SMALL (cid:96) 2 - NORM SOLUTIONS BETTER ?
Classical learning theory advocates regularization for linear models, such as SVM and linear regression.
For SVM, (cid:96) 2 regularization endows linear classiﬁers with a wide-margin property (Cortes &
Vapnik, 1995), and recent work on neural networks has shown that minimum norm neural network interpolators beneﬁt from over-parametrization (Hastie et al., 2019) .
Following the long history of explicit parameter norm regularization for linear models, weight decay is used for training nearly all high performance neural networks (He et al., 2015a; Chollet, 2016; Huang et al., 2017; Sandler et al., 2018).
In combination with weight decay, all of these cutting-edge architectures also employ batch normalization after convolutional layers (Ioffe & Szegedy, 2015).
With that in mind, van Laarhoven (2017) shows that the regularizing effect of weight decay is counteracted by batch normalization, which removes the effect of shrinking weight matrices.
Zhang et al.
(2018) argue that the synergistic interaction between weight decay and batch norm arises because weight decay plays a large role in regulating the effective learning rate of networks, since scaling down the weights of convolutional layers ampliﬁes the effect of each optimization step, effectively increasing the learning rate.
Thus, weight decay increases the effective learning rate as the regularizer drags the parameters closer and closer towards the origin.
The authors also suggest that data augmentation and carefully chosen learning rate schedules are more powerful than explicit regularizers like weight decay.
Other work echos this sentiment and claims that weight decay and dropout have little effect on performance, especially when using data augmentation (Hern ´ andez-Garc ´ ıa & K ¨ onig, 2018).
Hoffer et al.
(2018) further study the relationship between weight decay and batch normalization, and they develop normalization with respect to other norms.
Shah et al.
(2018) instead suggest that minimum norm solutions may not generalize well in the over-parametrized setting.
We ﬁnd that the difference between performance of standard network architectures with and without weight decay is often statistically signiﬁcant, even with a high level of data augmentation, for example, horizontal ﬂips and random crops on CIFAR-10 (see Tables 2 and 3).
But is weight decay the most effective form of (cid:96) 2 regularization?
Furthermore, is the positive effect of weight decay because the regularizer promotes small norm solutions?
We generalize weight decay by biasing the (cid:96) 2 norm of the weight vector towards other values using the following regularizer, which we call norm-bias :
[Equation]
R 0 is equivalent to weight decay, but we ﬁnd that we can further improve performance by biasing the weights towards higher norms (see Tables 2 and 3).
In our experiments on CIFAR-10 and CIFAR-100, networks are trained using weight decay coefﬁcients from their respective original papers.
ResNet-18 and DenseNet are trained with µ 2 = 2500 and norm-bias coefﬁcient 0 .
005 , and MobileNetV2 is trained with µ 2 = 5000 and norm-bias coefﬁcient 0 .
µ is chosen heuristically by ﬁrst training a model with weight decay, recording the norm of the resulting parameter vector, and setting µ to be slightly higher than that norm in order to avoid norm-bias leading to a lower parameter norm than weight decay.
While we ﬁnd that weight decay improves results over a nonregularized baseline for all three models, we also ﬁnd that models trained with large norm bias (i.e., large µ ) outperform models trained with weight decay.
These results lend weight to the argument that explicit parameter norm regularization is in fact useful for training networks, even deep CNNs with batch normalization and data augmentation.
However, the fact that norm-biased networks can outperform networks trained with weight decay suggests that any beneﬁts of weight decay are unlikely to originate from the superiority of small-norm solutions.
To further investigate the effect of weight decay and parameter norm on generalization, we also consider models without batch norm.
In this case, weight decay directly penalizes the norm of the linear operators inside a network, since there are no batch norm coefﬁcients to compensate for the effect of shrinking weights.
Our goal is to determine whether small-norm solutions are superior in this setting where the norm of the parameter vector is more meaningful.
In our ﬁrst experiment without batch norm, we experience improved performance training an MLP with norm-bias (see Table 3).
In a state-of-the-art setting, we consider ResNet-20 with Fixup initialization, a ResNet variant that removes batch norm and instead uses a sophisticated initialization that solves the exploding gradient problem (Zhang et al., 2019).
We observe that weight decay substantially improves training over SGD with no explicit regularization — in fact, ResNets with this initialization scheme train quite poorly without explicit regularization and data normalization.
Still, we ﬁnd that norm-bias with µ 2 = 1000 and norm-bias coefﬁcient 0 .
achieves better results than weight decay (see Table 3).
This once again refutes the theory that small-norm parameters generalize better and brings into doubt any relationship between classical Tikhonov regularization and weight decay in neural networks.
See Appendix A.5 for a discussion concerning the ﬁnal parameter norms of Fixup networks as well as additional experiments on CIFAR-100, a harder image classiﬁcation dataset
[Table]
[Table]
[Table]
K ERNEL THEORY AND THE INFINITE - WIDTH LIMIT
In light of the recent surge of works discussing the properties of neural networks in the inﬁnitewidth limit, in particular, connections between inﬁnite-width deep neural networks and Gaussian processes, see Lee et al.
(2017), several interesting theoretical works have appeared.
The wide network limit and Gaussian process interpretations have inspired work on the neural tangent kernel (Jacot et al., 2018), while Lee et al.
and Bietti et al.
(2018) have used wide network assumptions to analyze the training dynamics of deep networks.
The connection of deep neural networks to kernel-based learning theory seems promising, but how closely do current architectures match the predictions made for simple networks in the large-width limit?
We focus on the Neural Tangent Kernel (NTK), developed in Jacot et al.
Theory dictates that, in the wide-network limit, the neural tangent kernel remains nearly constant as a network trains.
Furthermore, neural network training dynamics can be described as gradient descent on a convex functional, provided the NTK remains nearly constant during training (Lee et al., 2019).
In this section, we experimentally test the validity of these theoretical assumptions.
Fixing a network architecture, we use F to denote the function space parametrized by φ ∈ R p .
For the mapping F : R P → F , the NTK is deﬁned by
[Equation]
where the derivatives ∂ φ p F ( φ ) are evaluated at a particular choice of φ describing a neural network.
The NTK can be thought of as a similarity measure between images; given any two images as input, the NTK returns an n × n matrix, where n is the dimensionality of the feature embedding of the neural network.
We sample entries from the NTK by drawing a set of N images { x i } from a dataset, and computing the entries in the NTK corresponding to all pairs of images in our image set.
We do this for a random neural network f : R m → R n and computing the tensor Φ( φ )
∈ R N × N × n × n of all pairwise realizations, restricted to the given data:
[Equation]
By evaluating Equation 4 using automatic differentiation, we compute slices from the NTK before and after training for a large range of architectures and network widths.
We consider image classiﬁcation on CIFAR-10 and compare a two-layer MLP, a four-layer MLP, a simple 5-layer ConvNet, and a ResNet.
We draw 25 random images from CIFAR-10 to sample the NTK before and after training.
We measure the change in the NTK by computing the correlation coefﬁcient of the (vectorized) NTK before and after training.
We do this for many network widths, and see what happens in the wide network limit.
For MLPs we increase the width of the hidden layers, for the ConvNet (6-Layer, Convolutions, ReLU, MaxPooling), we increase the number of convolutional ﬁlters, for the ResNet we consider the WideResnet (Zagoruyko & Komodakis, 2016) architecture, where we increase its width parameter.
We initialize all models with uniform He initialization as discussed in
He et al.
(2015b), departing from speciﬁc Gaussian initializations in theoretical works to analyze the effects for modern architectures and methodologies. [Figure] l a t i v e N o r m o
f c
[-The results are visualized in Figure 1, where we plot parameters of the NTK for these different architectures, showing how the number of parameters impacts the relative change in the NTK ( || Φ 1 − Φ 
-]h a n g e i n N T K [Figure] 5 10k5 100k5 1M5 10M5 0 0.
[Figure]
C o rr e l a t i o n c o e ﬃ
c
{+i e n t (b) 5 1000 2 5 10k 2 5 100k 
+}
5 1M 2 5 10M 2 5 5 15 10 2100 2 52 5 10k5 100k
[Figure]
R e l a t i v e
N o r m o
f c
h a n g e i n N
T K (c)
[Figure]
[Figure]
[Figure]
5 1M
[Figure]
[Figure]
[Figure]
g
[Figure]
[Figure]
The results are visualized in Figure where number of parameters
{+1 − Φ 
+}
|| ||
Φ {+
+} || where Φ 0 / Φ /after training) and correlation coefﬁcient ( Cov(Φ 1 , Φ 0 ) /σ
(Φ 1 ) /σ (Φ 0 ) ).
Jacot et al.
(2018) predicts that the NTK should change very little during training in the inﬁnite-width limit.
At ﬁrst glance, it might seem that these expectations are hardly met for our (non-inﬁnite) experiments.
Figure 1a and Figure 1c show that the relative change in the NTK during training (and also the magnitude of the NTK) is rapidly increasing with width and remains large in magnitude for a whole range of widths of convolutional architectures.
The MLP architectures do show a trend toward small changes in the NTK, yet convergence to zero is slower in the 4-Layer case than in the 2-Layer case.
However, a closer look shows that almost all of the relative change in the NTK seen in Figure 1c is explained by a simple linear re-scaling of the NTK.
It should be noted that the scaling of the
NTK is strongly effected by the magnitude of parameters at initialization.
Within the NTK theory of Lee et al.
(2017), a linear rescaling of the NTK during training corresponds simply to a change in learning rate,
and so it makes more sense to measure similarity using a scale-invariant metric.
Measuring similarity between sub-sampled NTKs using the scale-invariant correlation coefﬁcient, as in Figure 1b, is more promising.
Surprisingly, we ﬁnd that, as predicted in Jacot et al.
(2018), the NTK changes very little (beyond a linear rescaling) for the wide ConvNet architectures.
For the dense networks, the predicted trend toward small changes in the NTK also holds for most of the evaluated widths, although there is a dropoff at the end which may be an artifact of the difﬁculty of training these wide networks on CIFAR-10.
For the Wide Residual Neural Networks, however, the general trend toward higher correlation in the wide network limit is completely reversed.
The correlation coefﬁcient decreases as network width increases, suggesting that the neural tangent kernel at initialization and after training becomes qualitatively more different as network width increases.
The reversal of the correlation trend seems to be a property which emerges from the interaction of batch normalization and skip connections.
Removing either of these features from the architecture leads to networks which have an almost constant correlation coefﬁcient for a wide range of network widths, see Figure 6 in the appendix, calling for the consideration of both properties in new formulations of the NTK.
In conclusion, we see that although the NTK trends towards stability as the width of simple architectures increases, the opposite holds for the highly performant Wide ResNet architecture. Even further, neither the removal of batch normalization or the removal of skip connections fully recover the positive NTK trend.
While we have hope that kernel-based theories of neural networks may yield guarantees for realistic (albeit wide) models in the future, current results do not sufﬁciently describe state-of-the-art architectures.
Moreover, the already good behavior of models with unstable NTKs is an indicator that good optimization and generalization behaviors do not fundamentally hinge on the stability of the NTK.
R ANK : D O NETWORKS WITH LOW - RANK LAYERS GENERALIZE BETTER ?
State-of-the-art neural networks are highly over-parameterized, and their large number of parameters is a problem both for learning theory and for practical use.
In the theoretical setting, rank has been used to tighten bounds on the generalization gap of neural networks.
Generalization bounds from Harvey et al.
(2017) are improved under conditions of low rank and high sparsity (Neyshabur et al., 2017) of parameter matrices, and the compressibility of low-rank matrices (and other lowdimensional structure) can be directly exploited to provide even stronger bounds (Arora et al., 2018).
Further studies show a tendency of stochastic gradient methods to ﬁnd low-rank solutions (Ji &
Telgarsky, 2018).
The tendency of SGD to ﬁnd low-rank operators, in conjunction with results showing generalization bounds for low-rank operators, might suggest that the low-rank nature of these operators is important for generalization.
Langenberg et al.
(2019) claim that low-rank networks, in addition to generalizing well to test data, are more robust to adversarial attacks.
Theoretical and empirical results from the aforementioned paper lead the authors to make two major claims.
First, the authors claim that networks which undergo adversarial training have low-rank and sparse matrices.
Second, they claim that networks with low-rank and sparse parameter matrices are more robust to adversarial attacks.
We ﬁnd in our experiments that neither claim holds up in practical settings, including ResNet-18 models trained on
CIFAR-10.
We test the generalization and robustness properties of neural networks with low-rank and highrank operators by promoting low-rank or high-rank parameter matrices in late epochs.
We employ the regularizer introduced in Sedghi et al.
(2018) to create the protocols RankMin, to ﬁnd low-rank parameters, and RankMax, to ﬁnd high-rank parameters. RankMin involves ﬁne-tuning a pre-trained [Table]
[Figure] [Figure] model by replacing linear operators with their low-rank approximations, retraining, and repeating this process.
Similarly, RankMax involves ﬁne-tuning a pre-trained model by clipping singular values from the SVD of parameter matrices in order to ﬁnd high-rank approximations.
We are able to manipulate the rank of matrices without strongly affecting the performance of the network.
We use both natural training and 7-step projected gradient descent (PGD) adversarial training routines (Madry et al., 2017).
The goal of the experiment is to observe how the rank of weight matrices impacts generalization and robustness.
We start by attacking naturally trained models with the standard PGD adversarial attack with (cid:15) = 8 / 255 .
Then, we move to the adversarial training setting and test the effect of manipulating rank on generalization and on robustness.
In order to compare our results with Langenberg et al.
(2019), we borrow the notion of effective rank, denoted by r ( W ) for some matrix W .
This continuous relaxation of rank is deﬁned as follows.
r ( W ) =
(cid:107) W (cid:107) ∗ (cid:107) W (cid:107) F where (cid:107) · (cid:107) ∗ , (cid:107) · (cid:107) 1 , and (cid:107) · (cid:107) F are the nuclear norm, the 1-norm, and the Frobenius norm, respectively.
Note that the singular values of convolution operators can be found quickly with a method from Sedghi et al.
and that method is used here.
[-14 Filter index
-]
In our experiments we investigate two architectures, ResNet-18 and ResNet-18 without skip connections.
We train on CIFAR-10 and CIFAR-100, both naturally and adversarially.
Table 4 shows that RankMin and RankMax achieve similar generalization on CIFAR-10.
More importantly, when adversarially training, a setting when robustness is undeniably the goal, we see the RankMax outperforms both RankMin and standard adversarial training in robust accuracy.
Figure 2 conﬁrms that these two training routines do, in fact, control effective rank.
Experiments with CIFAR-100 yield similar results and are presented in Appendix A.7.
It is clear that increasing rank using an analogue of rank minimizing algorithms does not harm performance.
Moreover, we observe that adversarial robustness does not imply low-rank operators, nor do low-rank operators imply robustness.
The ﬁndings in Ji & Telgarsky (2018) are corroborated here as the black dots in Figures 2 show that initializations are higher in rank than the trained models.
Our investigation into what useful intuition in practical cases can be gained from the theoretical work on the rank of CNNs and from the claims about adversarial robustness reveals that rank plays little to no role in the performance of CNNs in the practical setting of image classiﬁcation.
C ONCLUSION
This work highlights the gap between deep learning theory and observations in the real-world setting. We underscore the need to carefully examine the assumptions of theory and to move past the study of toy models, such as deep linear networks or single-layer MLPs, whose traits do not describe those of the practical realm.
First, we show that realistic neural networks on realistic learning problems contain suboptimal local minima. Second, we show that low-norm parameters may not be optimal for neural networks, and in fact, biasing parameters to a non-zero norm during training improves performance on several popular datasets and a wide range of networks.
Third, we show that the wide-network trends in the neural tangent kernel do not hold for ResNets and that the interaction between skip connections and batch normalization play a large role. Finally, we show that low-rank linear operators and robustness are not correlated, especially for adversarially trained models. A CKNOWLEDGMENTS
This work was supported by the AFOSR MURI Program, the National Science Foundation DMS directorate, and also the DARPA YFA and L2M programs. Additional funding was provided by the Sloan Foundation. A A PPENDIX A. P ROOF OF L EMMA
Lemma 1.
Consider a family of L-layer multilayer perceptrons with ReLU activations { F φ : R m → R n } and let s =
min
i n i be the minimum layer width.
Then this family has rank- s afﬁne expression.
Proof.
Let G be a rank- s afﬁne function, and Ω ⊂ R m be a ﬁnite set.
Let G ( x ) =
A x + b with
A = U Σ V being the singular value decomposition of A with U ∈ R n × s and V ∈ R s × m .
We deﬁne
[Equation] where 0 is a (possibly void) ( n 1 − s ) × m matrix of all zeros, and b 1 = c 1 for c = max x i ∈ Ω , 1 ≤ j ≤ n 1 | ( A 1 x i )
j | + 1 and 1 ∈ R n 1 being a vector of all ones.
We further choose
A l ∈ R n l × n l − 1 to have an s × s identity matrix in the upper left, and ﬁll all other entries with zeros.
This choice is possible since n l ≥ s for all l .
We deﬁne b l = [ 0 c 1 ] T ∈ R n l where 0 ∈ R 1 × s is a vector of all zeros and 1 ∈ R 1 × ( n l − s ) is a (possibly void) vector of all ones.
Finally, we choose A L =
[ U 0 ] , where now 0 is a (possibly void) n × ( n L − 1 − s ) matrix of all zeros, and b L = − cA L 1 + b for 1 ∈ R n L − 1 being a vector of all ones.
Then one readily checks that F φ ( x ) =
G ( x ) holds for all x ∈ Ω .
Note that all entries of all activations are greater or equal to c > 0 , such that no ReLU ever maps an entry to zero.
A.
P ROOF OF T HEOREM
Theorem 1.
Consider a training set, { ( x i , y i ) } Ni =1 , a family { F φ } of MLPs with s =
min i n i being the smallest width.
Consider the training of a rank- s linear classiﬁer G A, b , i.e.,
[Equation]
subject to rank [Equation]
[Equation]
for any continuous loss function L .
Then for each local minimum, ( A (cid:48) , b (cid:48) ) , of the above training problem, there exists a local minimum, φ (cid:48) , of L ( F φ ; { ( x i , y i ) } Ni =1 ) with the property that F φ (cid:48)
( x i ) =
G A (cid:48) , b (cid:48)
( x i ) for i = 1 , 2 , ..., N .
Proof.
Based on the deﬁnition of a local minimium, there exists an open ball D around ( A (cid:48) , b (cid:48) )
such that
[Equation]
First, we use the same construction as in the proof of Lemma 1 to ﬁnd a function F φ (cid:48) with F φ (cid:48) ( x i ) =
G A (cid:48) , b (cid:48)
( x i ) for all training example x i .
Because the mapping φ (cid:55)→ F φ ( x i ) is continuous (not only for the entire network F but also for all subnetworks), and because all activations of F φ (cid:48) are greater or equal to c > 0 , there exists an open ball B ( φ (cid:48) , δ 1 ) around φ (cid:48) such that the activations of F φ remain positive for all x i and all φ ∈ B ( φ (cid:48) , δ 1 ) .
Consequently, the restriction of F φ to the training set remains afﬁne linear for φ ∈ B ( φ (cid:48) , δ 1 ) .
In other words, for any φ ∈ B ( φ (cid:48) , δ 1 ) we can write
[Equation]
by deﬁning A ( φ ) =
A L A L − 1 . . .
A 1 and b ( φ ) =
(cid:80) Ll =
1 A L A L − 1 . . .
A l +1 b l .
Note that due to s = min
i n i , the resulting A ( φ ) satisﬁes rank ( A ( φ )) ≤ s .
After restricting φ to an open ball B ( φ (cid:48) , δ 2 ) , for δ 2 ≤ δ 1 sufﬁciently small, the above ( A ( φ ) , b ( φ )) satisfy ( A ( φ ) , b ( φ ))
∈ D for all φ ∈ B ( φ (cid:48) , δ 2 ) .
On this set, we, however, already know that the loss can only be greater or equal to L ( F φ (cid:48) ; { ( x i , y i ) } Ni =1 ) due to equation 6.
Thus, φ (cid:48) is a local minimum of the underlying loss function.
A. A DDITIONAL COMMENTS REGARDING T HEOREM
Note that our theoretical and experimental results do not contradict theoretical guarantees for deep linear networks (Kawaguchi, 2016; Laurent & Brecht, 2018) which show that all local minima are global.
A deep linear network with s = min( n, m ) is equivalent to a linear classiﬁer, and in this case, the local minima constructed by Theorem 1 are global.
However, this observation shows that Theorem 1 characterizes the gap between deep linear and deep nonlinear networks; the global minima predicted by linear network theories are inherited as (usually suboptimal) local minima when
ReLU’s are added.
Thus, linear networks do not accurately describe the distribution of minima in non-linear networks.
A. A DDITIONAL RESULTS FOR SUBOPTIMAL LOCAL OPTIMA
Table 5 shows more experiments.
As above in the previous experiment, we use gradient descent to train a full ResNet-18 architecture on CIFAR-10 until convergence from different initializations.
We ﬁnd that essentially the same results appear for the deeper architecture, initializing with very high bias leads to highly non-optimal solutions.
In this case even solutions that are equally bad as a zero-norm initialization.
Further results on CIFAR-100 are shown in Tables 6 and 7.
These experiments with MLP and
ResNet-18 show the same trends as explained above, thus conﬁrming that the results are not speciﬁc to the CIFAR-10 dataset.
A. D ETAILS CONCERNING LOW - NORM REGULARIZATION EXPERIMENTS
Our experiments comparing regularizers all run for 300 epochs with an initial learning rate of 0 .and
decreases by a factor of 10 at epochs 100, 175, 225, and 275.
We use the SGD optimizer with momentum 0 .
We also tried negative weight decay coefﬁcients, which leads to ResNet-18 CIFAR-10 performance above 90% while blowing up parameter norm, but this performance is still suboptimal and is not informative concerning the optimality of minimum norm solutions.
One might wonder if high norm-bias coefﬁcients lead to even lower parameter norm than low weight decay coefﬁcients. This [Table]
[Table] [Table] question may not be meaningful in the case of networks with batch normalization.
In the case of
ResNet-20 with Fixup, which does not contain running mean and standard deviation, the average parameter (cid:96) 2 norm after training with weight decay is 24 .
51 while that of models trained with normbias is 31 .
Below, we perform the same tests on CIFAR-100, a substantially more difﬁcult dataset.
Weight decay coefﬁcients are chosen to be ones used in the original paper for the corresponding architecture.
Norm-bias coefﬁcient/ µ 2 is chosen to be 8100 / 0 .
001 , and 2000 / 0 .for
ResNet-18, DenseNet-40, and ResNet-20 with Fixup, respectively, using the same heuristic as described in the main body.
[Table]
A. D ETAILS ON THE N EURAL T ANGENT K ERNEL EXPERIMENT
For further reference, we include details on the NTK sampling during training epochs in Figure 3.
We see that the parameter norm (Right) behaves normally (all of these experiments are trained with a standard weight decay parameter of 0 . 0005 ), yet the NTK norm (Left) rapidly increases.
Most of this increase, however is scaling of the kernel, as the correlation plot (Middle) is much less drastic.
We do see that most change happens in the very ﬁrst epochs of training, whereas the kernel only changes slowly later on.
[Figure]
a t i
[Figure]
t
[Figure]
i n i t i a l N T K
[Figure]
[Figure]
initialization. We expect this coefﬁcient to converge toward 1 in the inﬁnite-width limit for multi-layer networks. Also shown is the direct relative difference of the NTK norms, which behaves similarly to the normalized direct difference from ﬁgure 1.
[Figure] [Figure]
t i
[Figure]
f c
[Figure] g [Figure] i
[Figure]
A. D ETAILS ON R ANK M IN AND R ANK M AX
We employ routines to promote both low-rank and high-rank parameter matrices.
We do this by computing approximations to the linear operators at each layer.
Since convolutional layers are linear operations, we know that there is a matrix whose dimensions are the number of parameters in the input to the convolution and the number of parameters in the output of the convolution.
In order to compute low-rank approximations of these operators, one could write down the matrix corresponding to the convolution, and then compute a low-rank approximation using a singular value decomposition (SVD).
In order to make this problem computationally tractable we used the method for computing singular values of convolution operators derived in Sedghi et al.
We were then able to do low-rank approximation in the classical sense, by setting each singular value below some threshold to zero.
In order to compute high-rank operators, we clipped the singular values so that when mulitplying the SVD factors, we set each singular value to be equal to the minimum of some chosen constant and the true singular value.
It is important to note here that these approximations to the convolutional layers, when done naively, can return convolutions with larger ﬁlters.
To be precise, an n × n ﬁlter will map to a k × k ﬁlter through our rank modiﬁcations, where k ≥ n .
We follow the method in Sedghi et al.
(2018), where these ﬁlters are pruned back down by only using n × n entries in the output.
When naturally training ResNet-18 and Skipless ResNet-18 models, we train with a batch size of 128 for 200 epochs with the learning rate initiated to 0.01 and decreasing by a factor of 10 at epochs 100, 150, 175, and 190 (for both CIFAR-10 and CIFAR-100).
When adversarially training these two models on CIFAR-10 data, we use the same hyperparameters.
However, in order to adversarially train on CIFAR-100, we train ResNet-18 with a batch size of 256 for 300 epochs with an initial learning rate of 0.1 and a decrease by a factor of 10 at epochs 200 and 250.
For adversarially training Skipless ResNet-18 on CIFAR-100, we use a batch size of 256 for 350 epochs with an initial learning rate of 0.1 and a decrease by a factor of 10 at epochs 200, 250, and 300.
Adversarial training is done with an (cid:96) ∞ 7-step PGD attack with a step size of 2 / 255 , and (cid:15) = 8 / 255 .
For all of the training described above we augment the data with random crops and horizontal ﬂips.
During 15 additional epochs of training we manipulate the rank as follows.
RankMin and RankMax protocols are employed periodically in the last 15 epochs taking care to make sure that the loss remains small.
For these last epochs, the learning rate starts at 0.001 and decreases by a factor ofafter the third and ﬁfth epochs of the ﬁnal 15 epochs.
As shown in Table 10, we test the accuracy of each model on clean test data from the corresponding dataset, as well as on adversarial examples generated with 20-step PGD with (cid:15)
= 8 / 255 (with step size equal to 2 / 255 ) and (cid:15)
= 1 / 255 (with step size equal to .
When training multi-layer perceptrons on CIFAR-10, we train for 100 epochs with learning rate initialized to 0.01 and decreasing by a factor of 10 at epochs 60, 80 and 90.
Then, we train the network for 8 additional epochs, during which RankMin and RankMax networks undergo rank manipulation.
[Table]
[Table]
[Table]
Training
Accuracy (%)
Clean
Accuracy (%)
Robust (%) (cid:15) = 8 /
Robust (%) (cid:15) = 1 /
Naturally Trained 99.
RankMax 99.
RankMin 99.
Adversarially Trained 99.
RankMaxAdv 99.
RankMinAdv 99.
ResNet-18 w/o skip connections and CIFAR- [Table] method
TrainingAccuracy(%)
Clean
Accuracy (%)
Robust (%) (cid:15) = 8 /
Robust (%) (cid:15) = 1 /
Naturally Trained 99.
RankMax 99.
RankMin 99.
Adversarially Trained 99.
RankMaxAdv 99. RankMinAdv 99.