S ENSITIVITY AND G ENERALIZATION IN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conﬂict with classical notions of function complexity, which typically favor smaller models.
In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations.
Our experiments survey thousands of models with different fully-connected architectures, optimizers, and other hyper-parameters, as well as four different image classiﬁcation datasets.
We ﬁnd that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the input-output
Jacobian of the network, and that it correlates well with generalization.
We further establish that factors associated with poor generalization – such as full-batch training or using random labels – correspond to higher sensitivity, while factors associated with good generalization – such as data augmentation and ReLU non-linearities – give rise to more robust functions.
Finally, we demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points.
I NTRODUCTION
The empirical success of deep learning has thus far eluded interpretation through existing lenses of computational complexity (Blum & Rivest, 1988), numerical optimization (Choromanska et al., 2015; Goodfellow & Vinyals, 2014; Dauphin et al., 2014) and classical statistical learning theory (Zhang et al., 2016): neural networks are highly non-convex models with extreme capacity that train fast and generalize well.
In fact, not only do large networks demonstrate good test performance, but larger networks often generalize better , counter to what would be expected from classical measures, such as VC dimension.
This phenomenon has been observed in targeted experiments (Neyshabur et al., 2015), historical trends of Deep Learning competitions (Canziani et al., 2016), and in the course of this work (Figure 1).
This observation is at odds with Occam’s razor, the principle of parsimony, as applied to the intuitive notion of function complexity (see § A.2 for extended discussion).
One resolution of the apparent contradiction is to examine complexity of functions in conjunction with the input domain.
f ( x ) = x 3 sin( x ) may seem decisively more complex than g ( x ) = x .
But restrained to a narrow input domain of [ − 0 .
01] they appear differently: g remains a linear function of the input, while f ( x ) =
O (cid:0) x 4 (cid:1) resembles a constant 0 .
In this work we ﬁnd that such intuition applies to neural networks, that behave very differently close to the data manifold than away from it ( § 4.1).
We therefore analyze the complexity of models through their capacity to distinguish different inputs in the neighborhood of datapoints, or, in other words, their sensitivity.
We study two simple metrics presented in § 3 and ﬁnd that one of them, norm of the input-output Jacobian, correlates with generalization in a very wide variety of scenarios ( § 4.2, § 4.3), up to being predictive of the loss at individual test points ( § 4.4).
This work considers sensitivity only in the context of image classiﬁcation tasks.
We interpret the observed correlation with generalization as an expression of a universal prior on (natural) image [Figure] classiﬁcation functions that favor robustness (see § A.2 for details).
While we expect a similar prior to exist in many other perception settings, care should be taken when extrapolating our ﬁndings to other tasks, where such prior may not be justiﬁed (e g. weather forecasting).
R ELATED W ORK AND S UMMARY OF C ONTRIBUTIONS
We analyze complexity of fully-connected neural networks through (1) estimating the number of linear regions a network splits the input space into, and (2) measuring the norm of the input-output Jacobian within such regions.
The novelty of this work can be summarized as follows:
• Study of the behavior of trained neural networks on and off the data manifold through these metrics ( § 4.1); • Evaluation of these metrics on trained neural networks in a very large-scale experimental setting and ﬁnding that they correlate with generalization ( § 4.2, § 4.3, § 4.4).
A few prior works have examined measures related to the ones we consider.
In particular, Pascanu et al.
(2013); Mont´ufar et al.
; Raghu et al.
(2016) have investigated the expressive power of fully-connected neural networks built out of piecewise-linear activation functions.
Such functions are themselves piecewise-linear over their input domain, so that the number of linear regions into which input space is divided is one measure of how nonlinear the function is.
A function with many linear regions has the capacity to build complex, ﬂexible decision boundaries.
It was argued in (Pascanu et al., 2013; Mont ´ ufar et al., 2014) that an upper bound to the number of linear regions scales exponentially with depth but polynomially in width, for speciﬁc constructions.
Raghu et al.
(2016) derived a tight analytic bound and considered the number of linear regions for generic networks with random weights, as would be appropriate, for instance, at initialization.
However, the evolution of this measure after training has not been investigated before.
We examine a related measure, the number of hidden unit transitions along one-dimensional trajectories in input space, for trained networks.
Further motivation for this measure will be discussed in § 3.
Another perspective on function complexity can be gained by studying their robustness to perturbations to the input.
Indeed, Rasmussen & Ghahramani (2000) demonstrate on a toy problem how complexity as measured by the number of parameters may be of limited utility for model selection, while measuring the output variation allows to invoke Occam’s razor.
In this work we apply related ideas to a large-scale practical context of neural networks with up to a billion free parameters ( § 4.2, § 4.3) and discuss potential ways in which sensitivity permits the application of Occam’s razor to neural networks ( § A.2).
Sokolic et al.
(2017) provide theoretical support for the relevance of robustness, as measured by the input-output Jacobian, to generalization.
They derive bounds for the generalization gap in terms of the Jacobian norm within the framework of algorithmic robustness (Xu & Mannor, 2012).
Our results provide empirical support for their conclusions through an extensive number of experiments.
Several other recent papers have also focused on deriving tight generalization bounds for neural networks (Bartlett et al., 2017; Dziugaite & Roy, 2017; Anonymous, 2018a).
We do not propose theoretical bounds in this work but establish a correlation between our metrics and generalization in a substantially larger experimental setting than undertaken in prior works.
In the context of regularization, increasing robustness to perturbations is a widely-used strategy: data augmentation, noise injection (Jiang et al., 2009), weight decay (Krogh & Hertz, 1992), and max-pooling all indirectly reduce sensitivity of the model to perturbations, while Rifai et al.
Sokolic et al.
(2017) explicitly penalize the Frobenius norm of the Jacobian in the training objective.
In this work we relate several of the mentioned above regularizing techniques to sensitivity, demonstrating through extensive experiments that improved generalization is consistently coupled with better robustness as measured by a single metric, the input-output Jacobian norm ( § 4.2).
While some of these ﬁndings conﬁrm common-sense expectations (data augmentation reduces sensitivity,
Figure 6), others challenge our intuition of what makes a neural network robust (ReLU-networks, with unbounded activations, tend to be more robust than saturating HardSigmoid-networks, Figure 7).
One such ﬁnding demonstrates an inductive bias towards robustness in stochastic mini-batch optimization compared to full-batch training (Figure 8).
Interpreting this regularizing effect in terms of some measure of sensitivity, such as curvature, is not new (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016), yet we provide a novel perspective by relating it to reduced sensitivity to inputs instead of parameters.
The inductive bias of SGD (“implicit regularization”) has been previously studied in (Neyshabur et al., 2015), where it was shown through rigorous experiments how increasing the width of a singlehidden-layer network improves generalization, and an analogy with matrix factorization was drawn to motivate constraining the norm of the weights instead of their count.
Neyshabur et al.
(2017) further explored several weight-norm based measures of complexity that do not scale with the size of the model.
One of our measures, the Frobenius norm of the Jacobian is of similar nature (since the Jacobian matrix size is determined by the task and not by a particular network architecture).
However, this particular metric was not considered, and, to the best of our knowledge, we are the ﬁrst to evaluate it in a large-scale setting (e.g. our networks are up to 65 layers deep and up to 2units wide).
Sensitivity to inputs has also attracted a lot of interest in the context of adversarial examples (Szegedy et al., 2013).
Several attacks locate points of poor generalization in the directions of high sensitivity of the network (Goodfellow et al., 2014; Papernot et al., 2016; Moosavi-Dezfooli et al., 2016), while certain defences regularize the model by penalizing sensitivity (Gu & Rigazio, 2014) or employing decaying (hence more robust) non-linearities (Kurakin et al., 2016).
However, work on adversarial examples relates highly speciﬁc perturbations to a similarly speciﬁc kind of generalization (i.e. performance on a very small, adversarial subset of the data manifold), while this paper analyzes average-case sensitivity ( § 3) and typical generalization.
Finally, to the best of our knowledge, we are the ﬁrst to compare behavior of a trained network on and off the data manifold ( § 4.1).
The contrast between the two (Figure 2) further strengthens the empirical connection we draw between sensitivity and generalization; it also conﬁrms that, as mentioned in § 1, if a certain quality of a function is to be used for model comparison, input domain should always be accounted for.
S ENSITIVITY M ETRICS
We propose two simple measures of sensitivity for a fully-connected neural network f : R d → R n with respect to its input x ∈ R d (the output being unnormalized logits of the n classes).
Assume f employs a piecewise-linear activation function, like ReLU.
Then f itself, as a composition of linear and piecewise-linear functions, is a piecewise-linear map, splitting the input space R d into disjoint regions, implementing a single linear mapping on each.
Then we can measure two aspects of sensitivity by answering
How does the output of the network change as the input is perturbed within the linear region?
How likely is the linear region to change in response to change in the input?
We quantify these qualities as follows:
For a local sensitivity measure we adopt the Frobenius norm of the class probabilities Jacobian J ( x ) =
∂
f σ ( x ) /∂
x T (with J ij ( x ) =
∂
[ f σ ( x )] i /∂x j ), where f σ = σ ◦ f with σ being the softmax function 1 .
Given points of interest x test , we estimate the sensitivity of the function in those regions with the average Jacobian norm:
[Equation]
that we will further refer to as simply “Jacobian norm”.
Note that this does not require the label of x test .
Interpretation .
The Frobenius norm (cid:107) J ( x ) (cid:107) F = (cid:113)(cid:80) ij J ij ( x ) 2 estimates the averagecase sensitivity of f σ around x .
Indeed, consider an inﬁnitesimal Gaussian perturbation ∆ x ∼ N ( 0 , (cid:15) I ) : the expected magnitude of the output change is then E [Equation] [Equation] [Equation]
) (cid:107)
[Equation]
( x )
[Equation] To detect a change in linear region (further called a “transition”), we need to be able to identify it.
We do this analogously to Raghu et al.
For a network with piecewiselinear activations, we can, given an input x , assign a code to each neuron in the network f , that identiﬁes the linear region of the pre-activation of that neuron.
E.g. each ReLU unit will have 0 or 1 assigned to it if the pre-activation value is less or greater thanrespectively.
Similarly, a ReLU6 unit (see deﬁnition in § A.4) will have a code of 0 , 1 , or 2 assigned, since it has 3 linear regions 2 .
Then, a concatenation of codes of all neurons in the network (denoted by c ( x ) ) uniquely identiﬁes the linear region of the input x 3 .
Given this encoding scheme, we can detect a transition by detecting a change in the code.
We then sample k points on a closed one-dimensional trajectory T ( x ) (generated from a data point x and lying close to the data manifold; see below for details) and count transitions t ( x ) along it to quantify the number of linear regions:
[Equation]
x (cid:48) ∈T ( x ) (cid:13)(cid:13)(cid:13)(cid:13)
∂ c ( x (cid:48) )
{+∂ ( d x (cid:48) ) (cid:13)(cid:13)(cid:13)(cid:13) 
 d x (cid:48) , (1)+} where [ · ] is the Iverson bracket and the norm of the directional derivative (cid:107) ∂ c ( x (cid:48) ) /∂
( d x (cid:48) )
(cid:107) 1 amounts to a Dirac delta function at each transition.
As the sampling discretization k goes to inﬁnity, the evaluation of the integral becomes exact 4 .
By sampling multiple such trajectories around different points, we estimate the sensitivity metric:
[Equation]
that we will further refer to as simply “transitions” or “number of transitions.”
To assure the sampling trajectory T ( x test ) is close to the data manifold (since this is the region of interest), we construct it through horizontal translations of the image x test in pixel space (Figure 2, right).
We similarly augment our training data with horizontal and vertical translations in the corresponding experiments (Figure 6).
As earlier, this metric does not require knowing the label of x test .
Interpretation.
We can draw a qualitative parallel between the number of transitions and curvature of the function.
t ( x ) in Equation 1 amounts to the total (cid:96) 1 -norm of the directional derivative of c ( x ) along T ( x ) , which is 0 everywhere except for the transition boundaries.
This property is shared by the derivative of f , a constant input-logits Jacobian J l within each linear region.
Then, computing the total curvature of f along T ( x ) amounts to
[Equation]
where, as in Equation 1, the equality becomes exact for a sufﬁciently large k .
This sum is similar to t ( x ) as deﬁned in Equation 1, but quantiﬁes the amount of change in between two linear regions in a non-binary way.
However, estimating this integral on a densely sampled trajectory is a computationally-intensive task, which is one reason we instead count transitions.
As such, on a qualitative level, the two metrics (Jacobian norm and number of transitions) track the ﬁrst and second order terms of the Taylor expansion of the function.
Above we have deﬁned two sensitivity metrics to describe the learned function around the data, on average.
In § 4.1 we analyze these measures on and off the data manifold by simply measuring them along circular trajectories in input space that intersect the data manifold at certain points, but generally lie away from it (Figure 2, left). [Figure]
E XPERIMENTAL R ESULTS
We relate sensitivity and generalization through a sequence of experiments of increasing level of nuance.
Each study analyzes performance of a large number (usually thousands) of fully-connected neural networks having different hyper-parameters and optimization procedures.
Unless speciﬁed otherwise, we select only models having reached 100% training accuracy on the whole training set, thus considering only conﬁgurations relevant to practical applications.
This also allows us to study generalization disentangled from properties like expressivity and trainability, which are outside the scope of this work.
In order to efﬁciently evaluate the compute-intensive metrics ( § 3) in a very wide range of hyperparameters settings (see e.g. § A.5.5) we only consider fully-connected networks in this work.
Extending the investigation to more complex architectures is left for future research.
In § 4.1 we begin by comparing the sensitivity of trained neural networks on and off the training data manifold, i.e. in the regions of best and typical (over the whole input space) generalization.
In § 4.2 we compare sensitivity of identical trained networks that differ in a single parameter which is important for generalization.
Further, § 4.3 associates sensitivity and generalization in an unrestricted manner, i.e. comparing networks of a wide variety of hyper-parameters such as width, depth, non-linearity, weight initialization, optimizer, learning rate and batch size on CIFAR10 and CIFAR100, with a smaller, yet still substantial experiment conﬁrming a similar trend on MNIST and Fashion-MNIST (Xiao et al., 2017).
Finally, § 4.4 explores how predictive sensitivity (as measured by the Jacobian norm) is for individual test points.
4. S ENSITIVITY O N AND O FF THE T RAINING D ATA M ANIFOLD
We analyze the behavior of a trained neural network near and away from training data.
We do this by comparing sensitivity of the function along 3 types of trajectories:
A random ellipse.
This trajectory is extremely unlikely to pass anywhere near the real data, and indicates how the function behaves in random locations of the input space that it never encountered during training.
An ellipse passing through 3 training points of different class ( Figure 2, left).
This trajectory does pass through the three data points, but in between it traverses images that are linear combinations of different-class images, and hence lie outside of the natural images space.
Sensitivity of the function along this trajectory allows comparison of its behavior on and off the data manifold, as it approaches and moves away from the 3 anchor points.
An ellipse through 3 training points of the same class.
This trajectory is similar to the previous one, but, given the dataset used in the experiment (MNIST), is expected to traverse overall closer to the data manifold, since linear combinations of the same digit are more likely to resemble a realistic image.
Comparing transition density along this trajectory to the one through points of different classes allows further assessment of how sensitivity changes in response to approaching the data manifold.
We ﬁnd that, according to both the Jacobian norm and transitions metrics, functions exhibit much more robust behavior around the training data (Figure 3).
We further visualize this effect in 2D in Figure 4, where we plot the transition boundaries of the last (pre-logit) layer of a neural network before and after training.
After training we observe that training points lie in regions of low transition density.
Example of a sampling trajectory
Jacobian norm ( y -axis) of a trained network along...
π /3 (point 1) 2 π / π (point 2) 4 π / 5 π /3 (point 3) 2 π 0.
{+10k a random ellipse an ellipse through 3 training points of a different class an ellipse through 3 training points of the same class θ (position along the trajectory) Transition density ( y -axis) of a trained network along... π /3 (point 1) 2 π / π (point 2) 4 π / 5 π /3 (point 3) 
 π       a random ellipse an ellipse through 3 training points of a different class an ellipse through 3 training points of the same class θ (position along the trajectory)+} 4. S ENSITIVITY AND G ENERALIZATION F ACTORS
In § 4.1 we established that neural networks implement more robust functions in the vicinity of the training data manifold than away from it.
We now consider the more practical context of model selection.
Given two perfectly trained neural networks, does the model with better generalization implement a less sensitive function?
We study approaches in the machine learning community that are commonly believed to inﬂuence generalization: random labels (Figure 5); 6 , data augmentation (Figure 6), ReLUs (Figure 7), fullbatch training (Figure 8).
We ﬁnd that in each case, the change in generalization is coupled with the respective change in sensitivity (i.e. lower sensitivity corresponds to smaller generalization gap) as measured by the
Jacobian norm (and almost always for the transitions metric).
[-Transition density ( y -axis) of a trained network along... π /3 (point 1) 2 π / π (point 2) 4 π / 5 π /3 (point 3) 
-]
4. S ENSITIVITY AND G ENERALIZATION G AP
We now perform a large-scale experiment to establish direct relationships between sensitivity and generalization in a realistic setting.
In contrast to § 4.1, where we selected locations in the input space, and § 4.2, where we varied a single binary parameter impacting generalization, we now sweep over many different architectural and optimization choices ( § A.5.5).
Our main result is presented in Figure 9, indicating a strong relationship between the Jacobian norm and generalization.
In contrast, Figure 11 demonstrates that the number of transitions is not alone sufﬁcient to compare networks of different sizes, as the number of neurons in the networks has a strong inﬂuence on transition count.
4. S ENSITIVITY AND P ER -P OINT G ENERALIZATION
In § 4.3 we established a correlation between sensitivity (as measured by the Jacobian norm) and generalization averaged over a large test set ( 10 4 points).
We now investigate whether the Jacobian norm can be predictive of generalization at individual points.
As demonstrated in Figure 10, Jacobian norm at a point is predictive of the cross-entropy loss, but the relationship is not a linear one, and not even bijective (see § A.3 for analytic expressions explaining it).
In particular, certain misclassiﬁed points (right sides of the plots) have a Jacobian norm many orders of magnitude smaller than that of the correctly classiﬁed points (left sides).
However, we do remark a consistent tendency for points having the highest values of the Jacobian norm to be mostly misclassiﬁed.
This observation makes the jacobian norm a promising quantity to consider in the contexts of active learning and conﬁdence estimation in future research.
C ONCLUSION
We have investigated sensitivity of trained neural networks through the input-output Jacobian norm and linear regions counting in the context of image classiﬁcation tasks.
We have presented extensive experimental evidence indicating that the local geometry of the trained function as captured by the input-output Jacobian can be predictive of generalization in many different contexts, and that it varies drastically depending on how close to the training data manifold the function is evaluated.
We further established a connection between the cross-entropy loss and the Jacobian norm, indicating that it can remain informative of generalization even at the level of individual test points.
Interesting directions for future work include extending our investigation to more complex architectures and other machine learning tasks.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] CIFAR
G e n e r a li z a t i o n ga p  10k
[Figure]
[Figure]
CIFAR  0.
MNIST
G e n e r a li z a t i o n ga p
[Figure]
[Figure]
CIFAR
[Figure]
a
[Figure]
i a n n o r m
[Figure]
CIFAR
[Figure]
m
[Figure]
A A PPENDIX A. A DDITIONAL F IGURES
[Figure] [Figure] ’ S RAZOR ?
Here we brieﬂy discuss the motivation of this work in the context of Occam’s razor.
Occam’s razor is an heuristic for comparing hypotheses based on their complexity, giving preference to simple models.
It is commonly justiﬁed through the Bayesian “automatic Occam’s razor” reasoning (Jefferys & Berger, 1992; MacKay, 1991) stating that given a dataset D and uniform prior on two competing hypotheses classes H s (simple models, few parameters) and H c , (complex models, many parameters), the posterior P [ H|D ] ∝ P
[ D|H ] is higher for a simple model H s due to its likelihood
P [ D|H s ] having a smaller support in the dataset space (see Figure 12, left).
Occam’s razor for neural networks.
As seen in Figure 1, the above reasoning does not apply to neural networks: the best achieved generalization is obtained by a model that has around 10 4 more parameters than the simplest model capable of ﬁtting the dataset (within the evaluated search space).
On one hand, (Murray & Ghahramani, 2005; Telgarsky, 2015) demonstrate on concrete examples that a high number of free parameters in the model doesn’t necessarily entail high complexity.
On the other hand, a large body of work on the expressivity of neural networks (Pascanu et al., 2013; Mont´ufar et al., 2014; Raghu et al., 2016; Poole et al., 2016) shows that their ability to compute complex functions increases rapidly with size, while (Zhang et al., 2016) validates that they also easily ﬁt complex (even random) functions with stochastic optimization.
Classical metrics like VC dimension or Rademacher complexity increase with size of the network as well.
This indicates that weights of a neural network may actually correspond to its usable capacity, and hence lead to “smearing” the likelihood P [ D|H ] along a very large space of datasets D (cid:48) , making the dataset of interest
D less likely.
Potential issues.
We conjecture two reasons for why Occam’s razor may not apply: 1.
The approximate computation involved in estimating the “Occam’s factor” (MacKay, 1991; 1992) used to assess the shape of the likelihood curves in Figure 12 ( left) involves assumptions that may not hold for neural network in the context of stochastic optimization, and, in particular, do not account for the combinatorial growth of the acceptable volume of parameter space as width increases.
We conjecture that a realistic plot might look like the one shown in Figure 12 (right), which renders model comparison by their classes H impossible without highly precise knowledge of the likelihood curves.
For the proposed hypotheses classes (i
e. hypotheses binned by number of parameters) P [ H ] is far from uniform.
In fact, given the complexity of tasks we tackle with neural networks in practical applications, the prior might favor larger models to small ones (Neal et al., 1995).
We interpret this work as tackling the second issue: we suggest to deﬁne hypotheses classes based on sensitivity of the hypothesis (which yielded promising results in (Rasmussen & Ghahramani, 2000) on a toy task).
Indeed, at least in the context of natural images classiﬁcation, putting a prior on the number of parameters or Kolmogorov complexity of the hypothesis is extremely difﬁcult.
However, a statement that the true classiﬁcation function is robust to small perturbations in the input is much easier to justify.
As such, a prior P [ H ] in favor of robustness over sensitivity might fare better than a prior on speciﬁc network hyper-parameters. [Figure]
A. B OUNDING THE J ACOBIAN N ORM
Here we analyze the relationship between the Jacobian norm and the cross-entropy loss at individual test points.
Target class Jacobian .
We begin by relating the derivative of the target class probability J y ( x ) to per-point cross-entropy loss l ( x ) = − log
[ f σ ( x )]
y ( x ) (where y ( x ) is the correct integer class).
We will denote f σ ( x ) by σ and drop the x argument to de-clutter notation (e.g. write f instead of f ( x ) ).
Then the Jacobian can be expressed as
[Equation]
where (cid:12) is the Hadamard element-wise product.
Then indexing both sides of the equation at the correct class y yields
[Equation]
where e y is a vector of zeros everywhere except for e y = 1 .
Taking the norm of both sides results in (cid:107)
[Equation]
We now assume that magnitudes of the individual logit derivatives vary little in between logits and over the input space 8 :
[Equation]
which simpliﬁes Equation 2 to
[Equation]
where M = E x test (cid:13) (cid:13) ∂ f /∂ x T test (cid:13)(cid:13) 2 F /n .
And since σ lies on the ( n − 1) -simplex ∆ n − 1 , we can bound:
[Equation]
and ﬁnally
[Equation]
or, in terms of the cross-entropy loss l = − log σ y :
[Equation] [Equation]
We validate these approximate bounds in Figure 13 (top).
Full Jacobian .
Equation 3 establishes a close relationship between J y and loss l = − log σ y , but of course, at test time we do not know the target class y .
This allows us to only bound the full Jacobian norm from below:
[Equation]
For the upper bound
[Equation]
we assume the maximum-entropy case of σ y : σ i ≈ (1 − σ y ) / ( n − 1) , for i (cid:54) =
y .
The Jacobian norm is
[Equation]
where the ﬁrst summand becomes:
[Equation]
and each of the others
[Equation]
Adding n − 1 of such summands to (cid:107) J y (cid:107) 22 results in (cid:107) J (cid:107) F ≈
[Equation]
compared against the lower bound (Equation 4) and experimental data in ﬁgure 13.
A. N ON - LINEARITIES D EFINITIONS
Following activation functions are used in this work:
ReLU (Nair & Hinton, 2010): max( x, 0) ; 2.
ReLU6 (Krizhevsky, 2010): min (max( x, 0) , 6) ; 3.
Tanh: hyperbolic tangent, ( e x − e − x ) /
( e x + e − x ) ; 4.
HardTanh (Gulcehre et al., 2016): min (max( x, − 1) , 1) ; 5.
HardSigmoid (Gulcehre et al., 2016): min (max( x + 0 . 5 , 0) , 1) ;
A. E XPERIMENTAL S ETUP
All experiments were implemented in Tensorﬂow (Abadi et al., 2016) and executed with the help of
Vizier (Golovin et al., 2017).
All networks were trained with cross-entropy loss.
All networks were trained without biases.
All computations were done with 32-bit precision.
Learning rate decayed by a factor of 0 .
1 every 500 epochs.
Unless speciﬁed otherwise, initial weights were drawn from a normal distribution with zero mean and variance 2 /n for ReLU, ReLU6 and HardSigmoid; 1 /n for Tanh and HardTanh, where n is the number of inputs to the current layer.
All inputs were normalized to have zero mean and unit variance, or, in other terms, lie on the ddimensional sphere of radius √ d , where d is the dimensionality of the input.
All reported values, when applicable, were evaluated on the whole training and test sets of sizes 50000 and 10000 respectively.
E.g. “generalization gap” is deﬁned as the difference between train and test accuracies evaluated on the whole train and test sets.
When applicable, all trajectories/surfaces in input space were sampled with 2 20 points. 100n 1 μ 10 μ 100 μ 0. 100 1e −1e −1f 1p 1n 1 μ 0. CIFAR
J a c o b i a n n o r m ( c o rr ec t c l a ss )
100n 1 μ 10 μ 100 μ 0.
100 1e −1e −1f 1p 1n 1 μ 0. J a c o b i a n n o r m [Figure] A.5. P LOTS AND E RROR B ARS
All ﬁgures except for 10 and 13 are plotted with (pale) error bars (when applicable).
The reported quantity was usually evaluated 8 times with random seeds from 1 to 8 9 , unless speciﬁed otherwise.
E.g. if a network is said to be 100%-accurate on the training set, it means that each of the 8 randomlyinitialized networks is 100%-accurate after training.
The error bar is centered at the mean value of the quantity and spans the standard error of the mean in each direction.
If the bar appears to not be visible, it may be smaller than the mean value marker.
Weight initialization, training set shufﬂing, data augmentation, picking anchor points of data-ﬁtted trajectories, selecting axes of a zero-centered elliptic trajectory depend on the random seed.
A.5. S ENSITIVITY ALONG A T RAJECTORY
Relevant ﬁgure 3.
A 20-layer ReLU-network of width 200 was trained on MNIST 128 times, with plots displaying the averaged values.
A random zero-centered ellipse was obtained by generating two axis vectors with normallydistributed entries of zero mean and unit variance (as such making points on the trajectory have an expected norm equal to that of training data) and sampling points on the ellipse with given axes.
A random data-ﬁtted ellipse was generated by projecting three arbitrary input points onto a plane where they fall into vertices of an equilateral triangle, and then projecting their circumcircle back into the input space.
A.5. L INEAR R EGION B OUNDARIES
Relevant ﬁgure
A 15-layer ReLU6-network of width 300 was trained on MNIST for 2 18 steps using SGD with momentum (Rumelhart et al., 1988); images were randomly translated with wrapping by up topixels in each direction, horizontally and vertically, as well as randomly ﬂipped along each axis, and randomly rotated by 90 degrees clockwise and counter-clockwise.
The sampling grid in input space was obtain by projecting three arbitrary input points into a plane as described in § A.5.2 such that the resulting triangle was centered at
and it’s vertices were at a distance 0 .
8 form the origin.
Then, a sampling grid of points in the [ − 1; 1] × 2 square was projected back into the input space.
A.5. S MALL E XPERIMENT
Relevant ﬁgures: 6 and 9 ( bottom
All networks were trained for 2 18 steps of batch size of 256 using SGD with momentum.
Learning rate was set to 0 .
005 and momentum term coefﬁcient to 0 .
Data augmentation consisted of random translation of the input by up to 4 pixels in each direction with wrapping, horizontally and vertically.
The input was also ﬂipped horizontally with probability 0 .
For each dataset, networks of width { 100 , 200 , 500 , 1000 , 2000 , 3000 } , depth { 2 , 3 , 5 , 10 , 15 , 20 } and activation function { ReLU, ReLU6, HardTanh, HardSigmoid } were evaluated on 8 random seeds from 1 to 8 .
A.5. L ARGE E XPERIMENT
Relevant ﬁgures
335671 networks were trained for 2 19 steps with random hyper-parameters; if training did not complete, a checkpoint at step 2 18 was used instead, if available.
When using L-BFGS, the maximum number of iterations was set to 2684 .
The space of available hyper-parameters included 10 :
CIFAR10 and CIFAR100 datasets cropped to a 24 × 24 center region; 2.
all 5 non-linearities from § A.4; 3.
SGD, Momentum, ADAM (Kingma & Ba, 2014), RMSProp (Hinton et al., 2012) and LBFGS optimizers; 4.
learning rates from { 0 . 01 , 0 . 005 , 0 . 0005 } , when applicable.
Secondary coefﬁcients were ﬁxed at default values of Tensorﬂow implementations of respective optimizers; 5.
batch sizes of { 128 , 512 } (unless using L-BFGS with the full batch of 50000 ); 6.
standard deviations of initial weights from { 0 . 5 , 1 , 4 , 8 } multiplied by the default value described in § A.5; 7. widths from (cid:8) 1 , 2 , 4 , · · · , 2 16 (cid:9) ;
depths from (cid:8) 2 , 3 , 5 , · · · , 2 6 + 1 (cid:9) ; 9. true and random training labels; 10.
random seeds from 1 to 8 .
A.5. P ER - POINT G ENERALIZATION
Relevant ﬁgures: 10 and 13.
Here we trained for 2 16 steps and evaluated networks on random train and test subsets of sizes 5000and 1000 respectively of CIFAR100, CIFAR10, Fashion-MNIST and MNIST.
The hyper-parameters consisted of non-linearity (all functions from § A.4), width ( 50 , 100 or 200 ) and depth ( 2 , 5 , 10 , 20 ).
Only one random seed ( 1 ) was considered.
For each dataset, a random subset of 5 conﬁgurations among all the 100%-accurate (on training) networks was plotted.
Figure 13 was generated from a similar experiment, with the exception that the whole training set was used and 2 18 training steps were done.
A single random 100%-accurate (on training data) network was drawn to compare experimental measurements with analytic bounds on the Jacobian norm.