WHY FLATNESS DOES AND DOES NOT CORRELATE WITH GENERALIZATION FOR DEEP NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The intuition that local ﬂatness of the loss landscape is correlated with better generalization for deep neural networks (DNNs) has been explored for decades, spawning many different ﬂatness measures.
Recently, this link with generalization has been called into question by a demonstration that many measures of ﬂatness are vulnerable to parameter re-scaling which arbitrarily changes their value without changing neural network outputs.
Here we show that, in addition, some popular variants of SGD such as Adam and Entropy-SGD, can also break the ﬂatnessgeneralization correlation.
As an alternative to ﬂatness measures, we use a function based picture and propose using the log of Bayesian prior upon initialization, log P ( f ) , as a predictor of the generalization when a DNN converges on function f after training to zero error.
The prior is directly proportional to the Bayesian posterior for functions that give zero error on a training set.
For the case of image classiﬁcation, we show that log P ( f ) is a signiﬁcantly more robust predictor of generalization than ﬂatness measures are.
Whilst local ﬂatness measures fail under parameter re-scaling, the prior/posterior, which is global quantity, remains invariant under re-scaling.
Moreover, the correlation with generalization as a function of data complexity remains good for different variants of SGD
I NTRODUCTION
Among the most important theoretical questions in the ﬁeld of deep learning are: 1) What characterizes functions that exhibit good generalization?, and 2) Why do overparameterized deep neural networks (DNNs) converge to this small subset of functions that do not overﬁt?
Perhaps the most popular hypothesis is that good generalization performance is linked to ﬂat minima.
In pioneering works (Hinton & van Camp, 1993; Hochreiter & Schmidhuber, 1997), the minimum description length (MDL) principle (Rissanen, 1978) was invoked to argue that since ﬂatter minima require less information to describe, they should generalize better than sharp minima.
Most measures of ﬂatness approximate the local curvature of the loss surface, typically deﬁning ﬂatter minima to be those with smaller values of the Hessian eigenvalues (Keskar et al., 2016;
Wu et al., 2017; Zhang et al., 2018;
Sagun et al., 2016; Yao et al., 2018).
Another commonly held belief is that stochastic gradient descent (SGD) is itself biased towards ﬂatter minima, and that this inductive bias helps explain why DNNs generalize so well (Keskar et al., 2016; Jastrzebski et al., 2018;
Wu et al., 2017; Zhang et al., 2018; Yao et al., 2018; Wei & Schwab, 2019;
Maddox et al., 2020).
For example Keskar et al.
(2016) developed a measure of ﬂatness that they found correlated with improved generalization performance when decreasing batch size, suggesting that SGD is itself biased towards ﬂatter minima.
We note that others (Goyal et al., 2017;
Hoffer et al., 2017; Smith et al., 2017; Mingard et al., 2021a) have argued that the effect of batch size can be compensated by changes in learning rate, complicating some conclusions from Keskar et al.
Nevertheless, the argument that SGD is somehow itself biased towards ﬂat minima remains widespread in the literature.
In an important critique of local ﬂatness measures, Dinh et al.
(2017) pointed out that DNNs with
ReLU activation can be re-parameterized through a simple parameter-rescaling transformation.
[Equation]
where w 1 are the weights between an input layer and a single hidden layer, and w 2 are the weights between this hidden layer and the outputs.
This transformation can be extended to any architecture having at least one single rectiﬁed network layer.
The function that the DNN represents, and thus how it generalizes, is invariant under parameter-rescaling transformations, but the derivatives w.r.t.
parameters, and therefore many ﬂatness measures used in the literature, can be changed arbitrarily.
Ergo , the correlation between ﬂatness and generalization can be arbitrarily changed.
Several recent studies have attempted to ﬁnd "scale invariant" ﬂatness metrics (Petzka et al., 2019;
Rangamani et al., 2019; Tsuzuku et al., 2019).
The main idea is to multiply layer-wise Hessian eigenvalues by a factor of (cid:107) w i (cid:107) 2 , which renders the metric immune to layer-wise re-parameterization.
While these new metrics look promising experimentally, they are only scale-invariant when the scaling is layer-wise.
Other methods of rescaling (e.g. neuron-wise rescaling) can still change the metrics, so this general problem remains unsolved.
1. M AIN CONTRIBUTIONS
For a series of classic image classiﬁcation tasks (MNIST and CIFAR-10) we show that ﬂatness measures change substantially as a function of epochs.
Parameter re-scaling can arbitrarily change ﬂatness, but it quickly recovers to a more typical value under further training.
We also demonstrate that some variants of SGD exhibit signiﬁcantly worse correlation of ﬂatness with generalization than found for vanilla SGD.
In other words popular measures of ﬂatness sometimes do and sometimes do not correlate with generalization.
This mixed performance problematizes a widely held intuition that DNNs generalize well fundamentally because SGD or its variants are themselves biased towards ﬂat minima.
We next study the correlation of the Bayesian prior P ( f ) with the generalization performance of a DNN that converges to that function f .
This prior is the weighted probability of obtaining function f upon random sampling of parameters.
Motivated by a theoretical argument derived from a non-uniform convergence generalization bound, we show empirically that log P ( f ) correlates robustly with test error, even when local ﬂatness measures miserably fail, for example upon parameter re-scaling.
For discrete input/output problems (such as classiﬁcation), P ( f ) can also be interpreted as the weighted "volume" of parameters that map to f .
Intuitively, we expect local ﬂatness measures to typically be smaller (ﬂatter) for systems with larger volumes.
Nevertheless, there may also be regions of parameter space where local derivatives and ﬂatness measures vary substantially, even if on average they correlate with the volume.
Thus ﬂatness measures can be viewed as (imperfect) local measures of a more robust predictor of generalization, the volume/prior P ( f ) .
D EFINITIONS AND NOTATION 2. S UPERVISED LEARNING
For a typical supervised learning problem, the inputs live in an input domain X , and the outputs belong to an output space Y .
For a data distribution D on the set of input-output pairs X × Y , the training set S is a sample of m input-output pairs sampled i.i.d.
from D , S
=
( x i , y i ) } mi =1 ∼ D m , where x i ∈ X
and y i ∈ Y .
The output of a DNN on an input x
i is denoted as ˆ y i .
Typically a
DNN is parameterized by a vector w and trained by minimizing a loss function L :
Y × Y → R , which measures differences between the output ˆ y ∈ Y and the ground truth output y ∈ Y , by assigning a score L (ˆ y, y ) which is typically zero when they match, and positive when they don’t match.
Minimizing the loss typically involves using an optimization algorithm such as SGD on a training set S .
The generalization performance of the DNN, which is theoretically deﬁned over the underlying (typically unknown) data distribution D but is practically measured on a test set
E = { ( x (cid:48)
i , y (cid:48) i ) } | E | i =1 ∼ D | E | .
For classiﬁcation problems, the generalization error is practically measured as (cid:15) ( E ) =
| E | (cid:80) x (cid:48)
i ∈
E
[ ˆ y i (cid:54) =
y (cid:48) i ] , where 1 is the standard indicator function which is one when its input is true, and zero otherwise.
2. F LATNESS MEASURES
Perhaps the most natural way to measure the ﬂatness of minima is to consider the eigenvalue distribution of the Hessian H ij =
∂ 2 L ( w ) /∂w
i ∂w j once the learning process has converged (typically to a zero training error solution).
Here for simplicity we use L ( w ) instead of L (ˆ y, y ) as ˆ y is parameterized by w .
Sharp minima are characterized by a signiﬁcant number of large positive eigenvalues
λ i in the Hessian, while ﬂat minima are dominated by small eigenvalues.
Some care must be used in this interpretation because it is widely thought that DNNs converge to stationary points that are not true minima, leading to negative eigenvalues and complicating their use in measures of ﬂatness.
Typically, only a subset of the positive eigenvalues are used (Wu et al., 2017; Zhang et al., 2018).
Hessians are typically very expensive to calculate.
For this reason, Keskar et al.
(2016) introduced a computationally more tractable measure called "sharpness":
Deﬁnition 2.1 (Sharpness) .
Given parameters w (cid:48) within a box in parameter space C ζ with sides of length ζ > 0 , centered around a minimum of interest at parameters w , the sharpness of the loss L ( w ) at w is deﬁned as:
[Equation]
In the limit of small ζ , the sharpness relates to the spectral norm of the Hessian (Dinh et al., 2017):
[Equation]
The general concept of ﬂatness can be deﬁned as 1 /sharpness , and that is how we will interpret this measure in the rest of this paper.
2. F UNCTIONS AND THE B AYESIAN PRIOR
We ﬁrst clarify how we represent functions in the rest of paper using the notion of restriction of functions .
A more detailed explanation can be found in appendix C.
Here we use binary classiﬁcation as an example:
Deﬁnition 2.2 (Restriction of functions to C ) .
(Shalev-Shwartz & Ben-David, 2014) Consider a parameterized supervised model, and let the input space be X and the output space be Y , noting
Y = { 0 , 1 } in binary classiﬁcation setting.
The space of functions the model can express is a (potentially uncountably inﬁnite) set F ⊆ Y |X| .
Let C
= { c 1 , . .
, c m } ⊂ X .
The restriction of F to
C is the set of functions from C to Y that can be derived from functions in F :
[Equation]
where we represent each function from C to Y as a vector in Y | C | .
For example, for binary classiﬁcation, if we restrict the functions to S + E , then each function in
F S + E is represented as a binary string of length | S | + | E | .
In the rest of paper, we simply refer to "functions" when we actually mean the restriction of functions to S + E , except for the Boolean system in section 5.1 where no restriction is needed.
See appendix C for a thorough explanation.
For discrete functions, we next deﬁne the prior probability P ( f ) as
Deﬁnition 2.3 (Prior of a function) .
Given a prior parameter distribution P w ( w ) over the parameters, the prior of function f can be deﬁned as:
[Equation]
(2) where 1 is an indicator function: 1 [ arg ] =
if its argument is true or 0 otherwise; M is the parameterfunction map whose formal deﬁnition is in appendix B. Note that P ( f ) could also be interpreted as a weighted volume V ( f ) over parameter space.
If P w ( w ) is the distribution at initialization, the P ( f ) is the prior probability of obtaining the function at initialization.
We normally use this parameter distribution when interpreting P ( f ) .
illustrates how the relative sizes of the volumes of their basins of attraction V SGD ( f i correlate with the volumes V ( f i ) (or equivalently their priors P ( f i ) ) of the basins, and that, on average, larger
V ( f i ) or P ( f i ) implies ﬂatter functions, even if ﬂatness can vary locally
Note that the loss L ( w ) can vary within a region where the DNN achieves zero classiﬁcation error S
A schematic be in et al., but here it is more clear that local ﬂatness can be misleading
Remark.
Deﬁnition 2.3 works in the situation where the space X and Y are discrete, where P ( f ) has a prior probability mass interpretation.
This is enough for most image classiﬁcation tasks.
Nevertheless, we can easily extend this deﬁnition to the continuous setting, where we can also deﬁne a prior density over functions upon random initialization, with the help of Gaussian Process (GP) (Rasmussen, 2003).
For the GP prior see
appendix D. However, in this work, we focus exclusively on the classiﬁcation setting, with discrete inputs and outputs.
2. L INK BETWEEN THE PRIOR AND THE B AYESIAN POSTERIOR
Due to their high expressivity, DNNs are typically trained to zero training error on the training set S .
In this case the Bayesian picture simpliﬁes because if functions are conditioned on zero error on S , this leads to a simple 0 - 1 likelihood P ( S | f ) , indicating whether the data is consistent with the function.
Bayesian inference can be used to calculate a Bayesian posterior probability P B ( f | S ) for each f by conditioning on the data according to Bayes rule.
Formally, if S = { ( x i , y i ) } mi = corresponds to the set of training pairs, then
[Equation]
where P ( f ) is the Bayesian prior and P ( S ) is called the marginal likelihood or Bayesian evidence .
If we deﬁne, the training set neutral space N S as all parameters that lead to functions that give zero training error on S , then P ( S ) = (cid:82) N S P w ( w ) d w .
In other words, it is the total prior probability of all functions compatible with the training set S (Valle-Pérez et al., 2018; Mingard et al., 2021a).
Since P ( S ) is constant for a given S , P B ( f | S ) ∝ P ( f ) for all f consistent with that S .
T HE CORRELATION BETWEEN THE PRIOR AND GENERALIZATION
This link between the prior and the posterior is important, because it was empirically found in an extensive set of experiments by (Mingard et al., 2021a) that, for popular architectures and data sets,
[Equation]
where P SGD ( f | S ) is the probability that a DNN trained with SGD converges on function f , when trained to zero error on S .
In other words, to ﬁrst order, SGD appears to ﬁnd functions with a probability predicted by the Bayesian posterior, and thus with probabilities directly proportional to
P ( f ) .
The authors traced this behaviour to the geometry of the loss landscape, as follows.
Some general observations from algorithmic information theory (AIT) (Valle-Pérez et al., 2018) as well as direct calculations (Mingard et al., 2019) predict that the priors of functions should vary over many orders of magnitude.
When this is the case, it is reasonable to expect that the probabilities by which an optimizer ﬁnds different functions is affected by these large differences.
This is related to a mechanism identiﬁed previously in evolutionary dynamics, where it is called the arrival of the frequent (Schaper & Louis, 2014).
We illustrate this principle in ﬁg. 1 where we intuitively use the language of "volumes".
We expect that the relative sizes of the basins of attraction V SGD ( f ) , deﬁned as the set of initial parameters for which a DNN converges to a certain function f , is proportional, to ﬁrst order, to those of the priors P ( f ) (or equivalently the "volumes").
To second order there are, of course, many other features of a search method and a landscape that affect what functions a DNN converges on, but when the volumes/priors vary by so many orders of magnitude then we expect that to ﬁrst order P SGD ( f | S )
≈ P B ( f | S )
∝ P ( f ) =
V ( f ) .
Given that the P ( f ) of a function helps predict how likely SGD is to converge on that function, we can next ask how P ( f ) correlates with generalization.
Perhaps the simplest argument is that if DNNs trained to zero error are known to generalize well on unseen data, then the probability of converging on functions that generalize well must be high.
The P ( f ) of these functions must be larger than the priors of functions that do not generalize well.
Can we do better than this rather simplistic argument?
One way forward is empirical.
Mingard et al.
(2021a) showed that log ( P B ( f | S )) correlates quite tightly with generalization error.
These authors also made a theoretical argument based on the Poisson-Binomial nature of the error distribution to explain this log-linear relationship, but this approach needs further work.
One of the best overall performing predictors in the literature for generalization performance on classiﬁcation tasks is the marginal likelihood PAC-Bayes bound from Valle-Pérez et al.
Valle-Pérez & Louis (2020).
It is non-vacuous, relatively tight, and can capture important trends in generalization performance with training set size (learning curves), data complexity, and architecture choice (see also (Liu et al., 2021)).
However, the prediction uses the marginal likelihood P ( S ) deﬁned through a sum over all functions that produce zero error on the training set.
Here we are interested in the generalization properties of single functions.
One way around is to use a simple nonuniform bound which to the best of our knowledge was ﬁrst published in McAllester (1998) as a preliminary theory to the full PAC-Bayes theorems.
For any countable function space F , any distribution ˜ P , and for any selection of a training set S of size m under probability distribution D , it can be proven that for all functions f that give zero training error:
[Equation]
for δ ∈ (0 , 1)
Here we consider a space F S,E of functions with all possible outputs on the inputs of a speciﬁc E and zero error on a speciﬁc S ; (cid:15) S,E ( f ) is the error measured on E + S , which as the error on S is 0 , equals the error on the test set E .
This error will converge to the true generalization error on all possible inputs as | E | increases.
Valle-Pérez & Louis (2020) showed this bound has an optimal average generalization error when ˜ P ( f ) mimics the probability distribution over functions of the learning algorithm.
If P SGD ( f ) ≈ P B ( f | S ) ∝ P ( f ) , then the best performance of the bound is approximately when ˜ P ( f ) in eq.
(4) is the Bayesian prior P ( f ) .
Thus this upper bound on (cid:15) S,E ( f ) scales as − log ( P ( f )) .
F LATNESS , PRIORS AND GENERALIZATION
The intuition that larger P ( f ) correlates with greater ﬂatness is common in the literature, see e.g. Hochreiter & Schmidhuber (1997); Wu et al.
(2017), where the intuition is also expressed in terms of volumes.
If volume/ P ( f ) correlates with generalization, we expect ﬂatness should too.
Nevertheless, local ﬂatness may still vary signiﬁcantly across a volume.
For example Izmailov et al.
(2018) show explicitly that even in the same basin of attraction, there can be ﬂatter and sharper regions.
We illustrate this point schematically in ﬁg. 1, where one function clearly has a larger volume and on average smaller derivatives of the loss w.r.t.
the parameters than the others, and so is ﬂatter on average.
But, there are also local areas within the zero-error region where this correlation does not hold.
One of the main hypotheses we will test in this paper is that the correlation between ﬂatness and generalization can be broken even when the generalization-prior correlation remains robust.
E XPERIMENTAL R ESULTS 5. P RIOR / VOLUME - FLATNESS CORRELATION FOR B OOLEAN SYSTEM
We ﬁrst study a model system for Boolean functions of size n = 7 , which is small enough to directly measure the prior by sampling (Valle-Pérez et al., 2018).
There are 2 7 = 128 possible binary inputs.
Since each input is mapped to a single binary output, there are 2 128 = 3 .
4 × 10 34 possible functions f .
It is only practically possible to sample the prior P ( f ) because it is highly biased (Valle-Pérez et al., 2018; Mingard et al., 2019), meaning a subset of functions have priors much higher than average.
For a fully connected network (FCN) with two hidden layers of 40 ReLU units each (which was found to be sufﬁciently expressive to represent almost all possible functions) we empirically determined P ( f ) using 10 8 random samples of the weights w over an initial Gaussian parameter distribution P w ( w ) with standard deviation σ w = 1 . 0 and σ b = 0 . 1 .
We also trained our network with SGD using the same initialization and recorded the topmost commonly appearing output functions with zero training error on all 128 outputs, and then evaluated the sharpness/ﬂatness using deﬁnition 2.1 with an (cid:15) = 10 − 4 .
For the maximization process in calculating sharpness/ﬂatness, we ran SGD for 10 epochs and make sure the max value ceases to change.
As ﬁg. 2 demonstrates, the ﬂatness and prior correlate relatively well; ﬁg. S in the appendix shows a very similar correlation for the spectral norm of the Hessian.
Note that since we are studying the function on the complete input space, it is not meaningful to speak of correlation with generalization.
However, since for this system the prior P ( f ) is known to correlate with generalization (Mingard et al., 2021a), the correlation in ﬁg. 2 also implies that these ﬂatness measures will correlate with generalization, at least for these high P ( f ) functions.
5. P RIORS , FLATNESS AND GENERALIZATION FOR MNIST AND CIFAR-
We next study the correlation between generalization, ﬂatness and log P ( f ) on the real world datasets
MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky et al., 2009).
Because we need to run many different experiments, and measurements of the prior and ﬂatness are computationally expensive, we simplify the problem by binarizing MNIST (one class is 0-4, the other is 5-9) and CIFAR-
(we only study two categories out of ten: cars and cats).
Also, our training sets are relatively small (500/5000 for MNIST/CIFAR-10, respectively) but we have checked that our overall results are not affected by these more computationally convenient choices.
In appendix ﬁg.
S24 we show results for
MNIST with | S | = 10000 .
We use two DNN architectures: a relatively small vanilla two hidden-layer FCN with 784 inputs and 40 ReLU units in each hidden layer each, and also Resnet-50 (He et al., 2016), a 50-layer deep convolutional neural network, which is much closer to a state of the art (SOTA) system.
[Figure]
We measure the ﬂatness on cross-entropy (CE) loss at the epoch where SGD ﬁrst obtains zero training error.
Because the Hessian is so expensive to calculate, we mainly use the sharpness/ﬂatness measure (deﬁnition 2.1) which is proportional to the Frobenius norm of the Hessian.
The ﬁnal error is measured in the standard way, after applying a sigmoid to the last layer to binarize the outputs.
To measure the prior, we use the GPs to which these networks reduce in the limit of inﬁnite width (Lee et al., 2017; Matthews et al., 2018; Novak et al., 2018b).
As demonstrated in Mingard et al.
(2021a),
GPs can be used to approximate the Bayesian posteriors P B ( f | S ) for ﬁnite width networks.
For further details, we refer to the original papers above and to appendix D.
In order to generate functions f with zero error on the training set S , but with diverse generalization performance, we use the attack-set trick from Wu et al.
In addition to training on S , we add an attack set A made up of incorrectly labelled data.
We train on both S and A , so that the error on S is zero but the generalization performance on a test set E is reduced.
The larger A is w.r.t.
S , the worse the generalization performance.
As can be seen in ﬁg. 3(a)-(c), this process allows us to signiﬁcantly vary the generalization performance.
The correlation between log P ( f ) and generalization error is excellent over this range, as expected from our arguments in section 3.
Figs.3(d)-(f) show that the correlation between ﬂatness and generalization is much more scattered than for log P ( f ) .
In appendix G we also show the direct correlation between log P ( f ) and ﬂatness which closely resembles ﬁg. 3(d)-(f) because V ( f ) and (cid:15) correlate so tightly.
5. T HE EFFECT OF OPTIMIZER CHOICE ON FLATNESS
We then test the effect of changing the optimizer from the vanilla SGD we used in ﬁg. 3.
We use
Adam (Kingma & Ba, 2014), and entropy-SGD (Chaudhari et al., 2019) which includes an explicit term to maximize the ﬂatness.
Both SGD variants show good optimization performance for the standard default Tensorﬂow hyperparameters we use.
Their generalization performance, however, does not signiﬁcantly vary from plain SGD, and this is reﬂected in the priors of the functions that they ﬁnd.
More importantly, ﬁg. 4 shows that the generalization-ﬂatness correlation can be broken by
[Figure]
breakdown of correlation
Changing optimizers or changing hyperparameters can, of course, alter the generalization performance by small amounts, which may be critically important in practical applications.
Nevertheless, as demonstrated in Mingard et al.
(2021a), the overall effect of hyperparameter or optimizer changes is usually quite small on these scales.
The large differences in ﬂatness generated simply by changing the optimizer suggests that ﬂatness measures may not always reliably capture the effects of hyperparameter or optimizer changes.
Note that we ﬁnd less deterioration when comparing SGD to Adam for
Resnet50 on CIFAR-10, (ﬁg. S22).
The exact nature of these effects remains subtle.
5. T EMPORAL BEHAVIOR OF SHARPNESS AND log P ( f )
In the experiments above, the ﬂatness and log P ( f ) metrics are calculated at the epoch where the system ﬁrst reaches 100% training accuracy.
In ﬁg. 5, we measure the prior and the ﬂatness for each epoch for our FCN, trained on MNIST (with no attack set).
Zero training error is reached at epoch 140, and we overtrain for a further 1000 epochs.
From initialization, both the sharpness measure from
Deﬁnition 2.1, and log P ( f ) reduce until zero-training error is reached.
Subsequently, log P ( f ) stays constant, but the CE loss continues to decrease, as expected for such classiﬁcation problems.
This leads to a reduction in the sharpness measure (greater ﬂatness) even though the function, its prior, and the training error don’t change.
This demonstrates that ﬂatness is a relative concept that depends, for example, on the duration of training.
In ﬁgs.
S16 and S17 we show for an FCN on MNIST that the quality of ﬂatness-generalization correlations are largely unaffected by overtraining, for both SGD and Adam respectively, even though the absolute values of the sharpness change substantially.
One of the strong critiques of ﬂatness is that re-parameterizations such as the parameter-rescaling transformation deﬁned in eq.
(1) can arbitrarily change local ﬂatness measures (Dinh et al., 2017).
Fig.
5 shows that parameter-rescaling indeed leads to a spike in the sharpness measure (a strong reduction in ﬂatness).
As demonstrated in the inset, the prior is initially invariant upon parameterrescaling because f ( w ) is unchanged.
However, parameter-rescaling can drive the system to unusual parts of the volume with steep gradients in the loss function, which mean that SGD falls off the zero training error manifold.
log P ( f ) goes up because it is more likely to randomly fall onto large
V ( f ) functions.
However, the system soon relaxes to essentially the same function and log P ( f ) .
In ﬁg.
S11, we show that it is possible to obtain a spike in the sharpness measure without the prior changing.
In each case, the sharpness measure rapidly decays after the spike, suggesting that parameter-rescaling brings the system into a parameter region that is "unnatural".
D ISCUSSION AND FUTURE WORK
The notion that ﬂatness correlates with generalization is widely believed in the community, but the evidential basis for this hypothesis has always been mixed.
Here we performed extensive empirical work showing that ﬂatness can indeed correlate with generalization.
However, this correlation is not always tight, and can be easily broken by changing the optimizer, or by parameter-rescaling.
By contrast, the P ( f ) which is directly proportional to the Bayesian posterior P B ( f | S ) for functions that give zero error on the training set, is a much more robust predictor of generalization.
While the generalization performance of a DNN can be successfully predicted by the marginal likelihood PAC-Bayes bound (Valle-Pérez et al., 2018; Valle-Pérez & Louis, 2020), no such tight bound exists (to our knowledge) linking generalization and the Bayesian prior or posterior at the level of individual functions.
Further theoretical work in this direction is needed.
Moreover, it is natural to further extend current work towards linking ﬂatness and the prior to other quantities which correlate with generalization such as frequency (Rahaman et al., 2018;
Xu et al., 2019), or the sensitivity to changes in the inputs (Arpit et al., 2017; Novak et al., 2018a).
Improvements to the
GP approximations we use are an important technical goal.
P ( f ) can be expensive to calculate, so ﬁnding reliable local approximations related to ﬂatness may still be a worthy endeavour.
Finally, our main result – that log P ( f ) correlates so well with generalization – still requires a proper theoretical underpinning, notwithstanding the bound in eq.(4).
Such explanations will need to include not just the networks and the algorithms, but also the data (Zdeborová, 2020).
We refer readers to appendix A for more discusion on related works.
A M ORE RELATED WORK A. P RELIMINARIES : TWO KINDS OF QUESTIONS GENERALIZATION AND TWO TYPES OF INDUCTIVE BIAS
In this supplementary section we expand on our briefer discussion of related work in the Introduction of the main paper.
The question of why and how DNNs generalize in the overparameterized regime has generated a vast literature.
To organize our discussion, we follow (Mingard et al., 2021a) and ﬁrst distinguish two kinds of questions about generalization in overparameterized DNNs:
The question of over-parameterized generalization : Why do DNNs generalize at all in the overparameter- ized regime, where classical learning theory suggests they should heavily overﬁt. 2)
The question of ﬁne-tuned generalization : Given that a DNN already generalizes reasonably well, how can detailed architecture choice, optimizer choice, and hyperparameter tuning further improve generalization?
Question 2) is the main focus of a large tranche of the literature on generalization, and for good reason.
In order to build SOTA DNNs, even a few percent accuracy improvement (taking image classiﬁcation as an example) is important in practice.
Improved generalization performance can be achieved in many ways, including local adjustments of the DNNs structure (e.g. convolutional layers, pooling layers, shortcut connections etc.), hyperparameter tuning (learning rate, batch size etc.), or choosing different optimizers (e.g. vanilla SGD versus entropySGD (Chaudhari et al., 2019) or Adam Kingma & Ba (2014).
In this paper, however, we are primarily interested in question 1).
As pointed out, for example famously in (Zhang et al., 2016), but also by many researchers before that 1 , DNNs can be proven to be highly expressive, so that the number of hypotheses that can ﬁt a training data set S , but generalize poorly, is typically many orders of magnitude larger than the number that can actually generalize.
And yet in practice DNNs do not tend to overﬁt much, and can generalize well, which implies that DNNs must have some kind of inductive bias (Shalev-Shwartz & Ben-David, 2014) toward hypotheses that generalise well on unseen data.
Following the framework of (Mingard et al., 2021a), we use the language of functions (rather than that of hypotheses, see also appendix B.) to distinguish two major potential types of inductive bias.
A)
The inductive bias upon upon random sampling of parameters over a parameter distribution P w ( w ) .
In other words, given a DNN architecture, loss function etc.
and a measure over parameters P w ( w ) (which can be taken to be the initial parameter distribution for an optimiser, but is more general), this bias occurs when certain types of functions more likely to appear upon random sampling of parameters than others.
This inductive bias can be expressed in terms of a prior over functions P ( f ) , or in terms of a posterior P B ( f | S ) when the functions are conditioned, for example, on obtaining zero error on training set S .
B)
The inductive bias induced by optimizers during a training procedure.
In other words, given an inductive bias upon initialization (from A) , does the training procedure induce a further inductive bias on what functions a DNN expresses?
One way of measuring this second form of inductive bias is to calculate the probability P opt ( f | S ) that an DNN trained to zero error on training set S with optimizer opt (typically a variant of SGD) expresses function f , and to then compare it to the Bayesian posterior probability P B ( f | S ) that this function obtains upon random sampling of parameters Mingard et al.
(2021a).
In principle P B ( f | S ) expresses the inductive bias of type A), so any differences between P opt ( f | S ) and P B ( f | S ) could be due to inductive biases of type B).
These two sources of inductive bias can be relevant to both questions above about generalization.
We emphasise that our taxonomy of two questions about generalization, and two types of inductive bias is just one way of parsing these issues.
We make these ﬁrst order distinctions to help clarify our discussion of the literature, and are aware that there are other ways of teasing out these distinctions.
A. R ELATED WORK ON FLATNESS
The concept "ﬂatness" of the loss function of DNNs can be traced back to Hinton & van Camp (1993) and Hochreiter & Schmidhuber (1997).
Although these authors did not provide a completely formal mathematical deﬁnition of ﬂatness, Hochreiter & Schmidhuber (1997) described ﬂat minima as "a large connected region in parameter space where the loss remains approximately constant", which requires lower precision to specify than sharp minima.
They linked this idea to the minimum description length (MDL) principle (Rissanen, 1978), which says that the best performing model is the one with shortest description length, to argue that ﬂatter minima should generalize better than sharp minima.
More generally, ﬂatness can be interpreted as a complexity control of the hypotheses class introduced by algorithmic choices.
The ﬁrst thing to note is that ﬂatness is a property of the functions that a DNN converges on.
In other words, the basic argument above is that ﬂatter functions will generalize better, which can be relevant to both questions 1) and 2) above.
It is a different question to ask whether a certain way of ﬁnding functions (say by optimising a DNN to zero error on a training set) will generate an inductive bias towards ﬂatter functions.
In Hochreiter & Schmidhuber (1997), the authors proposed an algorithm to bias towards ﬂatter minima by minimizing the training loss while maximizing the log volume of a connected region of the parameter space.
This idea is similar to the recent suggestion of entropy-SGD Chaudhari et al.
(2019), where the authors also introduced an extra regularization to bias the optimizer into wider valleys by maximizing the "local entropy".
In an inﬂuential paper, Keskar et al.
(2016) reported that the solutions found by SGD with small batch sizes generalize better than those found with larger batch sizes, and showed that this behaviour correlated with a measure of "sharpness" (sensitivity of the training loss to perturbations in the parameters).
Sharpness can be viewed as a measure which is the inverse of the ﬂatness introduced by Hinton & van Camp (1993) and Hochreiter & Schmidhuber (1997).
This work helped to popularise the notion that SGD itself plays an important role in providing inductive bias, since differences in generalization performance and in sharpness correlated with batch size.
In follow-on papers others have showed that the correlation with batch size is more complex, as some of the improvements can be mimicked by changing learning rates or number of optimization steps for example, see (Hoffer et al., 2017;
Goyal et al., 2017; Smith et al., 2017; Neyshabur et al., 2017b).
Nevertheless, these changes in generalization as a function of optimizer hyperparameters are important things to understand because they are fundamentally type B inductive bias.
Because the changes in generalization performance in these papers tend to be relatively small, they mainly impinge on question 2) for ﬁne-tuned generalization.
Whether these observed effects are relevant for question 1) is unclear from this literature.
Another strand of work on ﬂatness has been through the lens of generalization bounds.
For example, Neyshabur et al.
(2017b) showed that sharpness by itself is not sufﬁcient for ensuring generalization, but can be combined, through PAC-Bayes analysis, with the norm of the weights to obtain an appropriate complexity measure.
The connection between sharpness and the PAC-Bayes framework was also investigated by Dziugaite & Roy (2017), who numerically optimized the overall PAC-Bayes generalization bound over a series of multivariate Gaussian distributions (different choices of perturbations and priors) which describe the KL-divergence term appearing in the second term in the combined generalization bound by Neyshabur et al.
(2017b).
For more discussion of this literature on bounds and ﬂatness, see also the recent review Valle-Pérez & Louis (2020).
Rahaman et al.
(2018) also draw a connection to ﬂatness through the lens of Fourier analysis, showing that DNNs typically learn low frequency components faster than high frequency components.
This frequency argument is related to the input-output sensitivity picture, which is systematically investigated in Novak et al.
(2018a).
There is also another wide-spread belief that SGD trained DNNs are implicitly biased towards having small parameters norms or large margin, intuitively inspired by classical ridge regression and SVMs.
Bartlett et al.
(2017) presented a margin-based generalization bound that depends on spectral and L 2 , 1 norm of the layer-wise weight matrices of DNNs.
Neyshabur et al.
(2017a) later proved a similar spectral-normalized margin bound using PAC-Bayesian approach rather than the complex covering number argument used in Bartlett et al.
Liao et al.
(2018) further strengthen the theoretical arguments that an appropriate measure of complexity for
DNNs should be based on a product norm by showing the linear relationship between training/testing CE loss of normalized networks.
Jiang et al.
(2018) also empirically studied the role of margin bounds.
In a recent important large-scale empirical work on different complexity measures by Jiang et al.
(2019), 40 different complexity measures are tested when varying 7 different hyperparameter types over two image classiﬁcation datasets.
They do not introduce random labels so that data complexity is not thoroughly investigated.
Among these measures, the authors found that sharpness-based measures outperform their peers, and in particular outperform norm-based measures.
It is worth noting that their deﬁnition of "worst case" sharpness is similar to deﬁnition 2.1 but normalized by weights, so they are not directly comparable.
In fact, their deﬁnition of worst case sharpness in the PAC-Bayes picture is more close to the works by Petzka et al.
; Rangamani et al.
; Tsuzuku et al.
(2019) which focus on ﬁnding scale-invariant ﬂatness measure.
Indeed enhanced performance are reported in these works.
However, these measures are only scale-invariant when the scaling is layer-wise.
Other methods of re-scaling (e.g. neuron-wise re-scaling) can still change the metrics.
Moreover, the scope of Jiang et al.
(2019) is concentrated on the practical side (e.g. inductive bias of type B) and does not consider data complexity, which we believe is a key ingredient to understanding the inductive bias needed to explain question 1) on generalization.
Finally, in another inﬂuential paper, Dinh et al.
(2017) showed that many measures of ﬂatness, including the sharpness used in Keskar et al.
(2016), can be made to vary arbitrarily by re-scale parameters while keeping the function unchanged.
This work has called into question the use of local ﬂatness measures as reliable guides to generalization, and stimulated a lot of follow on studies, including the present paper where we explicitly study how parameter-rescaling affects measures of ﬂatness as a function of epochs.
A. R ELATED WORK ON THE INFINITE - WIDTH LIMIT
A series of important recent extensions of the seminal proof in Neal (1994) - that a single-layer DNN with random iid weights is equivalent to a GP (Mackay, 1998) in the inﬁnite-width limit - to multiple layers and architectures (NNGPs) have recently appeared (Lee et al., 2017; Matthews et al., 2018; Novak et al., 2018b; Garriga-Alonso et al., 2019; Yang, 2019).
These studies on NNGPs have used this correspondence to effectively perform a very good approximation to exact Bayesian inference in DNNs.
When they have compared NNGPs to SGD-trained DNNs the generalization performances have generally shown a remarkably close agreement.
These facts require rethinking the role SGD plays in question 1) about generalization, given that NNGPs can already generalize remarkably well without SGD at all.
A. R ELATIONSHIP TO PREVIOUS PAPERS USING THE FUNCTION PICTURE
The work in this paper builds on a series of recent papers that have explored the function based picture in random neural networks.
We brieﬂy review these works to clarify their connection to the current paper.
Firstly, in (Valle-Pérez et al., 2018), the authors demonstrated empirically that upon random sampling of parameters, DNNs are highly biased towards functions with low complexity.
This behaviour does not depend very much on P w ( w ) for a range of initial distributions typically used in the literature.
Note that this behaviour does start to deviate from what was found in (Valle-Pérez et al., 2018), when the system enters a chaotic phase, which can be reached with for tanh or erf non-linearities and for P w ( w ) with a relatively large variance Yang & Salman (2019).
They show more speciﬁcally that the bias towards simple functions is consistent with the "simplicity bias" from Dingle et al.
(2018; 2020), which was inspired by the coding theorem from algorithmic information theory (AIT)
(Li & Vitanyi, 2008), ﬁrst derived by Levin (1974) .
The idea of simplicity bias in
DNNs states that if the parameter-function map is sufﬁciently biased, then the probability of the DNN producing a function f on input data drops exponentially with increasing Kolmogorov complexity K ( f ) of the function f .
In other words, high P ( f ) functions have low K ( f ) , and high K ( f ) functions have low P ( f ) .
A key insight from (Dingle et al., 2018; 2020) is that K ( f ) can be approximated by an appropriate measure ˜ K ( f ) and still be used to make predictions on P ( f ) , even if the true K ( f ) is formally incomputable.
Recently Mingard et al.
(2019) and De Palma et al.
(2018) gave two separate non-AIT based theoretical justiﬁcations for the existence of simplicity bias in DNNs.
In other words, this line of work suggests that DNNs have an intrinsic bias towards simple functions upon random sampling of parameters, and in our taxonomy, that is bias of type A).
If simplicity bias in DNNs matches "natural" data distributions, then, at least upon random sampling of parameters, this should help facilitate good generalization.
Indeed, it has been shown that data such as MNIST or CIFAR-10 is relatively simple (Lin et al., 2017; Goldt et al., 2019; Spigler et al., 2019), suggesting that an inductive bias toward simplicity will assist with good generalization.
A second paper upon which the current one builds is (Mingard et al., 2021a), where extensive empirical test (for a range of architectures (FCN, CNN, LSTM), datasets (MNIST, Fashion-MNIST, CIFAR-10,ionosphere, IMDb moviereview dataset), and SGD variants (vanilla SGD, Adam, Adagrad, RMSprop, Adadelta), as well as for different batch sizes and learning rates) were done of the hypothesis that:
[Equation]
Here P opt ( f | S ) is the probability that an optimiser (SGD or one of its variants) converges upon a function f after training to zero training error on a training set S .
By training over many different parameter initializations,
P opt ( f | S ) can be calculated.
Similarly, the Bayesian posterior probability P B ( f | S ) is deﬁned as the probability that upon random sampling of parameters, a DNN expresses function f , conditioned on zero error on S .
The functions were, as in the current paper, a restriction to a given training set S and test set E .
Since the systems always had zero error on the training set, functions could be compared by what they produced on the test set (for example, the set of labels on the images for image classiﬁcation).
It was found that the hypothesis (A.4) held remarkably well to ﬁrst order, for a wide range of systems.
At ﬁrst sight this similarity is surprising, given that the procedures to generate P opt ( f | S ) (training with an optimiser such as SGD) is completely different from those for P
B ( f | S ) (where GP techniques and direct sampling were used), which knows nothing of optimisers at all.
The fact that these two probabilities are so similar suggests that any inductive bias of type B, which would be a bias beyond what is already present in P B ( f | S ) , is relatively small.
While this conclusion does not imply that there are no induced biases of type B), and clearly there are since hyperparameter tuning affects ﬁne-tuned generalization, it does suggest that the main source of inductive bias needed to explain 1), the question of why DNNs generalize in the ﬁrst place, is found in the inductive biases of type A), which are already there in P B ( f | S ) .
In (Mingard et al., 2021a), the authors propose that, for highly biased priors P ( f ) , that SGD is dominated by the large differences in basin size for the different functions f , and so ﬁnds functions with probabilities dominated by the initial distribution.
A similar effect was seen in evolutionary systems Schaper & Louis (2014); Dingle et al.
(2015) where it was called the arrival of the frequent.
In addition, in (Mingard et al., 2021a), the authors observed for one system that − log( P B ( f | S )) scaled linearly with the generalization error on E for a wide range of errors.
This preliminary result provided inspiration for the current paper where we directly study the correlation between the prior P ( f ) and the generalization error.
The third main function based paper that we build upon is (Valle-Pérez & Louis, 2020) which provides a comprehensive analysis of generalization bounds.
In particular, it studies in some detail the Marginal Likelihood
PAC-Bayes bound, ﬁrst presented in Valle-Pérez et al.
(2018), which is predicts a direct link between the generalization error and the log of the marginal likelihood P ( S ) .
P ( S ) can be interpreted as the total prior probability that a function is found with zero error on the training set S , upon random sampling of parameters of the DNN.
The performance of the bound was tested for challenges such as varying amounts of data complexity, different kinds of architectures, and different amounts of training data (learning curves).
For each challenge it works remarkably well, and to our knowledge no other bound has been tested this comprehensively.
Again, the good performance of this bound, which is agnostic about optimisers, suggest that a large part of the answer to question 1) can be found in the inductive bias of type A), e.g. that found upon initialization.
The bound is not accurate enough to explain smaller effects relevant for ﬁne-tuning generalization, which can originate from other sources such as a difference in optimiser hyperparameters.
These conclusions are consistent with the different approach in this paper, where we use the prior P ( f ) (which knows nothing about SGD) and show that it also correlates with predicted test error for DNNS trained with SGD and its variants.
We do propose a simpler bound that is consistent with the observed scaling, but more work is needed to get anywhere near the rigour found in (Valle-Pérez & Louis, 2020) for the full marginal likelihood bound.
Finally, we note that in all three of these papers, GPs are used to calculate marginal likelihoods, posteriors, and priors.
Technical details of how to use GPs can be found clearly explained there.
The current paper builds on this body of work and uses some of the techniques described therein, but it is distinct.
Firstly, our measurements on ﬂatness are new, and our claim that the prior P ( f ) correlates with generalization, while indirectly present in (Mingard et al., 2021a) was not developed there at all as that paper focuses on the posterior P
B ( f | S ) , and did not use the attack set trick to vary functions that are consistent with S , and so is tackling a different question (namely how much extra inductive bias comes from using SGD over the inductive bias already present in the Bayesian posterior).
The attack set trick means that P ( S ) does not change, while clearly the generalization error (or expected test error) does change, so the marginal likelihood bound is not predictive here.
B P ARAMETER - FUNCTION MAP AND NEUTRAL SPACE
The link between the parameters of a DNN and the function it expresses is formally described by the parameter- function map:
Deﬁnition B.1 (Parameter-function map) .
Consider the model deﬁned in deﬁnition 2.2, if the model takes parameters within a set W ⊆ R n , then the parameter-function map M is deﬁned as
[Equation]
where f w denotes the function parameterized by w .
The parameter-function map, introduced in (Valle-Pérez et al., 2018), serves as a bridge between a parameter searching algorithm (e.g. SGD) and the behaviour of a DNN in function space.
In this context we can also deﬁne the:
Deﬁnition B.2 (Neutral space) .
For a model deﬁned in Deﬁnition B.1, and a given function f , the neutral space N f ⊆ W is deﬁned as
[Equation]
The nomenclature comes from genotype-phenotype maps in the evolutionary literature (Manrubia et al., 2020), where the space is typically discrete, and a neutral set refers to all genotypes that map to the same phenotype.
In this context, the Bayesian prior P ( f ) can be interpreted as the probabilistic volume of the corresponding neutral space.
C C LARIFICATION ON DEFINITION OF FUNCTIONS AND PRIOR
The discussion of "functions" represented by DNNs can be confusing without careful deﬁnition.
In ﬁg. S6 we list four different interpretations of "functions" commonly seen in literature which also are directly related to our work.
These interpretations cover both regression and classiﬁcation settings.
Let X be an arbitrary input domain and Y be the output space.
According to different interpretations of the function represented by a DNN, Y will be different, for the same choice of X and DNN.
Deﬁnition C.1 ( f DNN ) .
Consider a DNN whose input domain is X .
Then f DNN belongs to a class of functions F
DNN which deﬁne the mapping between X to the pre-activation of the last layer of DNN, which lives in R d : f DNN ∈ F DNN : X → R d d is the width of the last layer of DNN.
In standard GP terminology, f DNN is also called latent function (Rasmussen, 2003).
This is the function we care about in regression problems.
In the context of supervised learning, we have to make some assumptions about the characteristics of F DNN , as otherwise we would not know how to choose between functions which are all consistent with the training sample but might have hugely different generalization ability.
This kind of assumptions are called inductive bias .
One common approach of describing the inductive bias is to give a prior probability distribution to F DNN , where higher probabilities are given to functions that we consider to be more likely.
For DNNs, F DNN is a set of functions over an (in general) uncountably inﬁnite domain X .
There are several approaches to deﬁne probability distributions over such sets.
GP represent one approach, which generalizes Gaussian distributions to function spaces.
If we ask only for the properties of the functions at a ﬁnite number of points, i.e. restriction of F DNN to C : { c 1 , . .
, c m }
⊂ X (see deﬁnition 2.2), then inference with a GP, reduces to inference with a standard multidimensional Gaussian distribution.
This is an important property of GP called consistency , which helps in making computations with GP feasible.
As shown in appendix D, we can readily compute with this GP prior over F
DNN as long as it is restricted on a ﬁnite data set.
Later in deﬁnition C.4 we will formally deﬁne the restricted function f
RES .
In classiﬁcation tasks, we typically get a data sample from X × Y , where without loss of generality Y has the form of Y = { 1 , . . .
, k } where k is the number of classes.
For simplicity, we further assume binary classiﬁcation where Y = { 0 , 1 } Note in the scope of binary classiﬁcation we have the last layer width of d = 1 .
To grant the outputs of the function represented by a DNN a probability interpretation, we need the outputs lie in the interval (0 , 1) .
One way of doing so is to "squash" the outputs of f DNN to (0 , 1) by using a ﬁnal activation , typically a logistic or sigmoid function λ ( z ) =
(1 + exp( − z )) − 1 .
Subsequently we have the deﬁnition of f ACT in ﬁg. S6:
Deﬁnition C.2 ( f ACT ) .
Consider the setting and f DNN deﬁned in deﬁnition C.1 where d = 1 , and a logistic activation λ ( z ) =
(1 + exp( − z )) − 1 .
Then f ACT is deﬁned as :
[Equation]
where ◦ denotes function composition.
we also deﬁne the space of f ACT as
[Equation]
In real life classiﬁcation datasets, we typically do not have access to the probability of an input classiﬁed as one certain label, but the labels instead.
When we discuss functions represented by DNNs in classiﬁcation, we usually mean the coarse-grained version of f ACT ∈ F ACT , meaning we group all outputs to 1 if the probability of predicting the inputs as being label " 1 " is greater or equal than 0 . 5 , and 0 otherwise.
Mathematically, we deﬁne f LAB as:
Deﬁnition C.3 ( f LAB ) .
Consider the setting and f ACT deﬁned in deﬁnition C.2 and a threshold function
[Equation]
=
[Equation]
otherwise
[Equation]
Then we deﬁne f LAB and the space F LAB as:
[Equation]
The deﬁnition C.3 allows us to describe the function represented by a DNN in binary classiﬁcation as a binary string consisting of "0" and "1", whose length is equal to the size of input domain set |X| .
As explained earlier, in classiﬁcation we also want to put a prior over F LAB and use this prior as our belief about the task before seeing any data.
Finally, as we mentioned above, to make computations tractable, we restrict the domain to a ﬁnite set of inputs.
We use the deﬁnition of restriction in deﬁnition 2.2 to formally deﬁne the "functions" we mean and practically use in our paper:
Deﬁnition C.4 ( f RES ) .
Consider a DNN whose input domain is X with a last layer width d = 1 .
Let C
= { c 1 , . .
, c m } ⊂
X be any ﬁnite subset of X with cardinality m ∈ N .
The restriction of function space F ∈ {F DNN , F LAB } to C is denoted as F C , and is deﬁned as the space of all functions from C to Y realizable by functions in F .
We denote with f RES elements of their corresponding spaces of restricted functions.
Speciﬁcally, in regression:
[Equation]
and in binary classiﬁcation:
[Equation]
Note that in deﬁnition C.4 we only consider scalar outputs in the regression setting.
For multiple-output functions, one approach is to consider d GPs and compute the combined kernel (Alvarez et al., 2011).
In statistical learning theory, the function spaces F DNN and F LAB are also called hypotheses classes , with their elements called hypotheses (Shalev-Shwartz & Ben-David, 2014).
It is important to note that our deﬁnition of prior and its calculation is based on the restriction of the hypotheses class to the concatenation of training set and test set S + E .
Mathematically, this means the prior of a function P ( f ) we calculated in the paper is precisely P ( f RES ) , except for the Boolean system in section 5.1, where the input domain X is discrete and small enough to enumerate (this can also be thought of as the trivial restriction).
As explained above, this restriction is inevitable if we want to compute the prior over F DNN or F LAB .
A simple example on MNIST (LeCun et al., 1998) can also help to gain a intuition of the necessity of such restriction, where all inputs would include the set of 28x28 integer matrices whose entries take values from 0-255, which gives 256 784 possible inputs.
This indicates that for real-life data distributions the number of all possible inputs is hyper-astronomically large, if not inﬁnite.
Nevertheless, In some cases, such as the Boolean system described in Valle-Pérez et al.
and treated in section 5.1, there is no need for such restriction because it is feasible to enumerate all possible inputs: there are only 7 Boolean units which give 2 7 = 128 possible data sample.
However, even in such cases, the number of possible functions is still large ( 2 128 ≈ 10 38 ).
D GP APPROXIMATION OF THE PRIOR
In this section, we sketch out how we calculated the prior of a function P ( f ) (Valle-Pérez et al., 2018; Mingard et al., 2021a).
As in those papers, we use GP, which have been shown to be equivalent to DNNs in the limit of inﬁnite layer width (Neal, 1994; Lee et al., 2017; Matthews et al., 2018; Tan, 2008; Rasmussen, 2003).
These neural network GPs (NNGPs) have been shown to accurately approximate the prior over functions P ( f ) of ﬁnite-width Bayesian DNNs (Valle-Pérez et al., 2018;
Matthews et al., 2018; Mingard et al., 2021a).
For the NNGPs, a GP prior is placed on the pre-activations z of the last layer of the neural network (before a ﬁnal non-linearity, e.g. softmax, is applied), meaning that for any ﬁnite inputs set x = { x 1 , . . .
, x m } , the random output vector (pre-activations) z
=
[ z ( x 1 ) , . .
, z ( x m )]
T has a Gaussian distribution.
Note that in this paper, the the last layer has a single activation since we only focus on binary classiﬁcation.
This setting is corresponding to the deﬁnition of function restriction is deﬁnition C.4, with z ∈ R m .
Without loss of generality, we can assume such a process has a zero mean.
The prior probability of the outputs z can be calculated as:
[Equation]
Σ is the covariance matrix (often called kernel), whose entries are deﬁned as Σ( x i , x j ) ≡ E [ z ( x i ) , z ( x j )] .
Neal (1994) gave the basic form of kernel Σ in single hidden layer case, where Σ depends on the variance of weights and biases ( σ w and σ b ).
In DNNs with multiple hidden layers, the kernel for layer l can be calculated recursively by induction, assuming the layer l − 1 is a GP (Lee et al., 2017; Matthews et al., 2018).
The kernel for fully connected ReLU-activated networks has a well known arc-cosine kernel analytical form (Cho & Saul, 2009), which we used in all FCNs in our work.
For ResNet50, the analytical form of GP kernel is intractable.
Instead, we use a Monte Carlo empirical kernel (Novak et al., 2018b), and apply one step of the fully connected GP recurrence relation (Lee et al., 2017), taking advantage of the fact that the last layer of ResNet50 is fully connected.
Mathematically, the empirical kernel can be expressed as:
[Equation]
where (cid:0)
h L − 1 w m ( x ) (cid:1) c is the activation of c -th neuron in the last hidden layer ( L is the total number of layers) for the network parameterized by the m -th sampling of parameters w m , M is the number of total Monte Carlo sampling, n is the width of the ﬁnal hidden layer, and σ w , σ b are the weights and biases variance respectively.
In our experiments, M is set to be 0 .
1 × ( | S | + | E | ) .
After calculating P ( z ) with the corresponding kernel, the prior over (coarse-grained) restriction of functions P ( f ) can be calculated through likelihood P ( f | z ) , which in our case is just a Heaviside function representing a hard sign nonlinearity.
As non-Gaussian likelihood produces an intractable P ( f ) , we used Expectation Propagation (EP) algorithm for the approximation of P ( f ) (Rasmussen, 2003).
This same EP approximation was used in Mingard et al.
(2021a) where it is discussed further.
We represent the function f by the input-output pairs on the concatenation of training set and test set S + E .
E C OMPARING FLATNESS METRICS
As mentioned in section 2.2 of the main text, the sharpness metric in deﬁnition 2.1 can be directly linked to spectral norm of the Hessian by considering the second order Taylor expansion of L ( w ) around a critical point in powers of ζ (Dinh et al., 2017).
We empirically conﬁrm this relationship by showing in ﬁg. S7 the direct correlation between sharpness and spectral norm of Hessian, as well as in ﬁg. S8 the correlation between Hessian spectral norm and prior in Boolean system described in section 5.1.
In addition to the spectral norm, another widely used ﬂatness measure is the product of a subset of the positive Hessian eigenvalues, typically say the product of the top-50 largest eigenvalues (Wu et al., 2017; Zhang et al., 2018).
We measured the correlation of these Hessian-based ﬂatness metrics with sharpness as well as with generalization for the FCN/MNIST system in ﬁg. S9.
Since they correlate well with the sharpness, these ﬂatness measures show very similar correlations with generalization as sharpness does in ﬁg. 3 and ﬁg. 4.
In other words, the Hessian-based ﬂatness metrics also capture the loose correlation with generalization when the neural network is trained by SGD and the deterioration of this correlation when we change the optimizer to Adam.
[Figure]
Another detail worth noting is that Keskar et al.
(2016) used the L-BFGS-B algorithm (Byrd et al., 1995) to perform the maximization of L ( w ) in C ζ , which is the box boundary around the minimum of interest:
[Equation]
However, as a quasi-Newton method, L-BFGS-B is not scalable when there are tens of millions of parameters in modern DNNs.
To make Keskar-sharpness applicable for large DNNs (e.g. ResNet50), we use vanilla SGD for the maximization instead.
The hyperparameters for the sharpness calculation are listed in table 1.
Note that the entries batch size, learning rate and number of epochs all refer to the SGD optimizer which does the maximization in the sharpness calculation process.
The number of epochs is chosen such that the max value of
[Table]
F I MPLEMENTING PARAMETER RE - SCALING
In this section we describe in detail how we implement the alpha scaling in DNNs ﬁrst proposed by Dinh et al.
The widely used rectiﬁed linear activation (ReLU) function
[Equation]
exhibits the so-called "non-negative homogeneity" property:
[Equation]
The action of a L -layered deep feed-forward neural network can be written as:
[Equation]
L in which
• x is the input vector • W L is the weight matrix of the L -th layer
• b L is the bias vector of the L -th layer
To simplify notation, we have not included the ﬁnal activation function, which may take any form (softmax or sigmoid etc.)
without modiﬁcation of the proceeding arguments.
Generalizing the original arguments from Dinh et al.
(2017) slightly to include bias terms, we exploit the non-negative homogeneity of the ReLU function to ﬁnd that a so-called " α -scaling" of one of the layers will not change its behaviour.
Explicitly applying this to the i -th layer yields:
[Equation]
Clearly, the transformation described by ( W i , b i , W i +1 )
→ (cid:0) αW i , αb i , 1 α W i +1 (cid:1) will lead to an observationally equivalent network (that is, a network whose output is identical for any given input, even if the weight and bias terms differ).
Since the α scaling transformation does not change the function, it does not change the prior of the function.
However, for large enough α , as shown for example in ﬁg. 5, we see that SGD can be "knocked" out of the current neutral space because of the large gradients that are induced by the α scaling.
This typically leads to the prior suddenly surging up, because the random nature of the perturbation means that the system is more likely to land on large volume functions.
However, we always observe that the prior then drops back down quite quickly as SGD reaches zero training error again.
On the other hand, as shown in ﬁg.
S11, when the value of α is smaller it does not knock SGD out of the neutral space, and so the prior does not change at all.
Nevertheless, the sharpness still exhibits a strong spike due to the the alpha scaling.
Although not in the scope of this work, it is worth noting that the alpha scaling process in Convolutional Neural Networks (CNNs) with batch normalization (Ioffe & Szegedy, 2015) layer(s) is somewhat different.
Because a batch normalization layer will eliminate all afﬁne transformations applied on its inputs, one can arbitrarily alpha scale the layers before a batch normalization layer without needing to of compensate in following layer, provided the scaling is linear.
G F LATNESS AND PRIOR CORRELATION
In the main text, we showed the correlation of the Bayesian prior and of sharpness with generalization in ﬁg. 3 and ﬁg. 4.
Here, in ﬁg.
S12, we show the direct correlation of the prior and sharpness.
As expected from the ﬁgures in the main text, sharpness correlates with prior roughly as it does with generalization - i.e. reasonably for vanilla SGD but badly for entropy-SGD (Chaudhari et al., 2019) or Adam (Kingma & Ba, 2014).
We note that, as shown in ﬁg. S9, sharpness also correlates relatively well with the spectral norm of the Hessian and log product of its 50 largest eigenvalues for all the optimizers.
So the correlation of ﬂatness with prior/generalization does not depend much on which particular ﬂatness measure is used.
Overall, it is perhaps unsurprising that a local measure such as ﬂatness varies in how well it approximates the global prior.
What is unexpected (at least to us) is that Adam and Entropy-SGD break the correlation for this data set.
In appendix J.2, we show that this correlation also breaks down for other more complex optimizers, but, interestingly, not for full-batch SGD.
Further empirical and theoretical work is needed to understand this phenomenon.
For example, is the optimizer dependence of the correlation between ﬂatness and prior a general property of the optimizer, or is it speciﬁc to certain architectures and datasets?
One hint that these results may have complex dependencies on architecture and dataset comes from our observation that for ResNet50 on
Cifar10, we see less difference between SGD and Adam than we see for the FCN on MNIST.
More work is needed here.
[Figure]
H T EMPORAL BEHAVIOR OF SHARPNESS
When using sharpness in deﬁnition 2.1 as the metric of ﬂatness, there are several caveats.
First is the hyper- parameters (see table 1): the value of sharpness is only meaningful under speciﬁed hyperparameters, and in different experiments the sharpnesses are only comparable when the hyperparameters are the same.
This renders sharpness less convenient to use (but still much more efﬁcient than Hessian calculation).
Second is the time evolving behavior of sharpness:
For the classiﬁcation problems we study, and for CE loss, it can continue to change even when the function (and hence generalization) is unchanged.
Before reaching zero training error, gradients can be large, and the behavior of sharpness (deﬁnition 2.1) can be unstable under changes of box size ζ .
This effect is likely the cause of some unusual ﬂuctuations in the sharpness that can be observed in ﬁg. 5 and ﬁg.
S11 around epoch 100.
In ﬁg.
S13 we show that this artefact disappears for larger ζ .
Similarly, when the gradients are big (typical in training), sharpness may no longer link to spectral norm of Hessian very well.
In ﬁg.
S14, we ﬁrst train the FCN to zero error, then "alpha scale" after 500 epochs, and then keep post-training for another 5000 epochs, much longer than in ﬁg. 5.
The behaviour of sharpness and prior upon "alpha scaling" (not surprisingly) follows our discussion in section 5.4.
What is interesting to see here is that after enough overtraining, the effect of the alpha scaling spike appears to disappear, and the overall curve looks like a continuation of the curve prior to alpha scaling.
What this suggests is that alpha-scaling brings the system to an area of parameter space that is somehow "unnatural".
Again, this is a topic that deserves further investigation in the future.
Finally, we show the temporal behavior of a Hessian-based ﬂatness measure in ﬁg. S15.
Because of the large memory cost when calculating the Hessian, we use a smaller FCN on MNIST, with the ﬁrst hidden layer having 10 units.
We ﬁnd that the Hessian based ﬂatness exhibit similar temporal behavior to sharpness.
[Figure]
I T HE CORRELATION BETWEEN GENERALIZATION , PRIOR , AND SHARPNESS UPON OVERTRAINING
As shown in ﬁg. 5 of the main text, and further discussed in appendix H, ﬂatness measures keep decreasing upon overtraining even when the function itself does not change.
In this section, we revisit the correlation between prior, ﬂatness and generalization at different numbers of overtraining epochs, i.e. after reaching zero training error.
As can be seen in ﬁg.
S16 to ﬁg.
S21, overtraining does not meaningfully affect the correlation between sharpness, prior, and generalization we observed at the epoch where zero error is ﬁrst reached in ﬁg. 3 and ﬁg. 4.
When the optimizer is SGD, the ﬂatness, no matter if it is measured by sharpness or Hessian based metrics, correlates well with prior and (hence) generalization across difference overtraining epochs; whereas when using Adam, the poor correlation also persist in overtraining
J F URTHER EXPERIMENTS J. R ES N ET 50 TRAINED WITH A DAM
When training ResNet50 on CIFAR-10, we use training set size | S | = 5000 , attack set size | A | = 5000 , test set size | E | = 2000 .
In each experiment, we mix the whole training set with different size of subset of attack set.
The size of | A | ranges as (0 , 500 , 1000 , 1500 , ..., 5000) .
For each subset of attack set we sample 5 times.
When training ResNet50 with Adam, we empirically found it is hard to train the neural net to zero training error with attack set size
| A | > 2500 .
So we only show the results for those functions found with | A | ≤ 2500 .
In ﬁg.
S22 we show the results of correlation between sharpness and prior with generalization with limited data.
The prior, as usual, correlates tightly with generalization, while the ﬂatness-generalization correlation is much more scattered, although it is slightly better than the correlation seen for the FCN on MNIST, and closer to the behaviour we observed for SGD in the main text.
J. M ORE SGD- VARIANT OPTIMIZERS
In ﬁg. S23 we provide further empirical results for the impact of choice of optimizer on the sharpness- generalization correlation by studying three common used SGD variants: Adagrad (Duchi et al., 2011),
Momentum (Rumelhart et al., 1986) (momentum= 0 . 9 ) and RMSProp (Tieleman & Hinton, 2012), as well as full batch gradient descent.
Interestingly, full batch gradient descent (or simply gradient descent) shows behaviour that is quite similar to vanilla SGD.
By contrast, for the other three optimizers, the correlation between sharpness and generalization breaks down, whereas the correlation between prior and generalization remains intact, much as was observed in the main text for Adam and Entropy-SGD. .
J. L ARGER TRAINING SET
In order to rule out any potential training size effect on our main argument of the ﬂatness, prior and generalization relationship, we further performed the experiments on MNIST with 10k training examples.
Larger training sets are hard because of the GP-EP calculation of the prior scales badly with size.
The results are shown in ﬁg.
S24.
It is clear that the correlations between sharpness, prior and generalization follow the same pattern as we see in ﬁg. 3, in which there are only | S | = 500 , | E
[Figure] [Figure]