Original: Why neural networks ﬁnd simple solutions: the many regularizers of geometric complexity
Revised: Why neural networks ﬁnd simple solutions: the many regularizers of geometric complexity

Original: Abstract
Revised: Abstract

Original: In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design.
Revised: In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design.

Original: In deep learning, it has been difﬁcult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks.
Revised: In deep learning, it has been difﬁcult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks.

Original: Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete
Revised: Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete

Original: Dirichlet energy.
Revised: Dirichlet energy.

Original: Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, ﬂatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.
Revised: Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, ﬂatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.

Original:  Introduction
Revised:  Introduction

Original: Regularization is an essential ingredient in the deep learning recipe and understanding its impact on the properties of the learned solution is a very active area of research [32, 42, 73, 84].
Revised: Regularization is an essential ingredient in the deep learning recipe and understanding its impact on the properties of the learned solution is a very active area of research [36, 46, 79, 90].

Original: Regularization can assume a multitude of forms, either added explicitly as a penalty term in a loss function [32] or implicitly through our choice of hyperparameters [6, 76, 79, 81, 83], model architecture [57, 59, 60] or initialization [30, 36, 37, 58, 70, 93, 94].
Revised: Regularization can assume a multitude of forms, either added explicitly as a penalty term in a loss function [36] or implicitly through our choice of hyperparameters [7, 82, 85, 87, 89], model architecture [61, 63, 64] or initialization [34, 40, 41, 62, 74, 99, 100].

Original: These forms are generally not designed to be analytically tractable but in practice, regularization is often invoked in the control of model complexity, putting a pressure on a model to discover simple solutions more so than complex solutions.
Revised: These forms are generally not designed to be analytically tractable but in practice, regularization is often invoked in the control of model complexity, putting a pressure on a model to discover simple solutions more so than complex solutions.

Original: To understand regularization in deep learning we need to precisely deﬁne model ‘complexity’ for deep neural networks.
Revised: To understand regularization in deep learning we need to precisely deﬁne model ‘complexity’ for deep neural networks.

Original: Complexity theory provides many techniques for measuring the complexity of a model, such as a simple parameter count or a parameter norm measurement [4, 22, 63, 72] but many of these measures can be problematic for neural networks [44, 91].
Revised: Complexity theory provides many techniques for measuring the complexity of a model, such as a simple parameter count or a parameter norm measurement [4, 25, 67, 76] but many of these measures can be problematic for neural networks [48, 97, 77].

Original: The recently observed phenomena of ‘double-descent’ [10, 11, 71] illustrates this clearly: neural networks with high model complexity, as measured by a parameter count, can ﬁt training data closely (sometimes interpolating the data exactly), while simultaneously having low test error [71, 91].
Revised: The recently observed phenomena of ‘double-descent’ [12, 13, 75] illustrates this clearly: neural networks with high model complexity, as measured by a parameter count, can ﬁt training data closely (sometimes interpolating the data exactly), while simultaneously having low test error [75, 97].

Original: Classically, we expect that interpolation of training data is evidence of overﬁtting, but yet, neural networks seem to be capable of interpolation while also having low test error.
Revised: Classically, we expect that interpolation of training data is evidence of overﬁtting, but yet, neural networks seem to be capable of interpolation while also having low test error.

Original: It is often suggested that some form of implicit regularization or explicit regularization is responsible for this, but how should we account for this in theory, and what complexity measure is most appropriate?
Revised: It is often suggested that some form of implicit regularization or explicit regularization is responsible for this, but how should we account for this in theory, and what complexity measure is most appropriate?

Original: In this work, we develop a measure of model complexity, called Geometric Complexity (GC), that has properties that are suitable for the analysis of deep neural networks.
Revised: In this work, we develop a measure of model complexity, called Geometric Complexity (GC), that has properties that are suitable for the analysis of deep neural networks.

Original: We use theoretical and empirical techniques to demonstrate that many different forms of regularization and other training heuristics can act to control geometric complexity through different mechanisms.
Revised: We use theoretical and empirical techniques to demonstrate that many different forms of regularization and other training heuristics can act to control geometric complexity through different mechanisms.

Original: We argue that the geometric complexity provides a convenient proxy for neural network performance.
Revised: We argue that the geometric complexity provides a convenient proxy for neural network performance.

Original: Our primary contributions are:
Revised: Our primary contributions are:

Original: • We develop a computationally tractable notion of complexity (Section 2), which we call Geometric Complexity (GC), that has many close relationships with many areas in deep learning and mathematics including harmonic function theory, Lipschitz smoothness (Section 2), and regularization theory (Section 4).
Revised: • We develop a computationally tractable notion of complexity (Section 2), which we call Geometric Complexity (GC), that has many close relationships with many areas in deep learning and mathematics including harmonic function theory, Lipschitz smoothness (Section 2), and regularization theory (Section 4).

Original: • We provide evidence that common training heuristics keep the geometric complexity low, including: (i) common initialization schemes (Section 3) (ii) the use of overparametrized models with a large number of layers (Fig. 2) (iii) large learning rates, small batch sizes, and implicit gradient regularization (Section 5 and Fig. 4) (iv) explicit parameter norm regularization, spectral norm regularization, ﬂatness regularization, and label noise regularization (Section 4 and Fig. 3) • We show that the geometric complexity captures the double-descent behaviour observed in the test loss as model parameter count increases (Section 6 and Fig. 5).
Revised: • We provide evidence that common training heuristics keep the geometric complexity low, including: (i) common initialization schemes (Section 3) (ii) the use of overparametrized models with a large number of layers (Fig. 2) (iii) large learning rates, small batch sizes, and implicit gradient regularization (Section 5 and Fig. 4) (iv) explicit parameter norm regularization, spectral norm regularization, ﬂatness regularization, and label noise regularization (Section 4 and Fig. 3) • We show that the geometric complexity captures the double-descent behaviour observed in the test loss as model parameter count increases (Section 6 and Fig. 5).

Original: The aim of this paper is to introduce geometric complexity, explore its properties and highlight its connections with existing implicit and explicit regularizers.
Revised: The aim of this paper is to introduce geometric complexity, explore its properties and highlight its connections with existing implicit and explicit regularizers.

Original: To disentangle the effects studied here from optimisation choices, we use stochastic gradient descent without momentum to train all models.
Revised: To disentangle the effects studied here from optimization choices, we use stochastic gradient descent without momentum to train all models.

Original: We also study the impact of a given training heuristic on geometric complexity in isolation of other techniques to avoid masking effects.
Revised: We also study the impact of a given training heuristic on geometric complexity in isolation of other techniques to avoid masking effects.

Original: For this reason we do not use data augmentation or learning rate schedules in the main part of the paper.
Revised: For this reason we do not use data augmentation or learning rate schedules in the main part of the paper.

Original: In the Supplementary Material (SM) we redo most experiments using SGD with momentum (Section C.7) and Adam (Section C.8) with very similar conclusion.
Revised: In the Supplementary Material (SM) we redo most experiments using SGD with momentum (Section C.7) and Adam (Section C.8) with very similar conclusion.

Original: We also observe the same behavior of the geometric complexity in a setting using learning rate schedule, data augmentation, and explicit regularization in conjunction to improve model performance (Section C.6).
Revised: We also observe the same behavior of the geometric complexity in a setting using learning rate schedule, data augmentation, and explicit regularization in conjunction to improve model performance (Section C.6).

Original: The exact details of all experiments in the main paper are listed in
Revised: The exact details of all experiments in the main paper are listed in

Original: SM Section B.
Revised: SM Section B.

Original: All additional experiment results and details can be found in SM Section C.
Revised: All additional experiment results and details can be found in SM Section C.

Original:  Geometric complexity and Dirichlet energy
Revised:  Geometric complexity and Dirichlet energy

Original: Although many different forms of complexity measures have been proposed and investigated (e.g., [21, 44, 72]), it is not altogether clear what properties they should have, especially for deep learning.
Revised: Although many different forms of complexity measures have been proposed and investigated (e.g., [24, 48, 76]), it is not altogether clear what properties they should have, especially for deep learning.

Original: For instance, a number of them like the Rademacher complexity [50], the VC dimension [87], or the simple model parameter count focus on measuring the entire hypothesis space, rather than a speciﬁc function, which can be problematic in deep learning [44, 91].
Revised: For instance, a number of them like the Rademacher complexity [54], the VC dimension [93], or the simple model parameter count focus on measuring the entire hypothesis space, rather than a speciﬁc function, which can be problematic in deep learning [48, 97].

Original: Other measures like the number of linear pieces for ReLU networks [3, 80] or various versions of the weight matrix norms [72] measure the complexity of the model function independently from the task at hand, which is not desirable [75].
Revised: Other measures like the number of linear pieces for ReLU networks [3, 86] or various versions of the weight matrix norms [76] measure the complexity of the model function independently from the task at hand, which is not desirable [81].

Original: An alternative approach is to learn a complexity measure directly from data [56] or to take the whole training procedure over a dataset into account [71].
Revised: An alternative approach is to learn a complexity measure directly from data [60] or to take the whole training procedure over a dataset into account [75].

Original: Recently, other measures focusing on the model function complexity over a dataset have been proposed in [26] and [63] to help explain the surprising generalization power of deep neural networks.
Revised: Recently, other measures focusing on the model function complexity over a dataset have been proposed in [30] and [67] to help explain the surprising generalization power of deep neural networks.

Original: Following that last approach and motivated by frameworks well established in the ﬁeld of geometric analysis [46], we propose a deﬁnition of complexity related to the theory of harmonic functions and minimal surfaces.
Revised: Following that last approach and motivated by frameworks well established in the ﬁeld of geometric analysis [50], we propose a deﬁnition of complexity related to the theory of harmonic functions and minimal surfaces.

Original: Our deﬁnition has the advantage of being computationally tractable and implicitly regularized by many training heuristics in the case of neural networks.
Revised: Our deﬁnition has the advantage of being computationally tractable and implicitly regularized by many training heuristics in the case of neural networks.

Original: It focuses on measuring the complexity of individual functions rather than that of the whole function space, which makes it different from the Radamacher or VC complexity.
Revised: It focuses on measuring the complexity of individual functions rather than that of the whole function space, which makes it different from the Radamacher or VC complexity.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Let g θ : R d → R k be a neural network parameterized by θ .
Revised: Let g ✓ : R d ! R k be a neural network parameterized by ✓ .

Original: We can write g θ ( x ) = a ( f θ ( x )) where a denotes the last layer activation, and f θ its logit network.
Revised: We can write g ✓ ( x ) = a ( f ✓ ( x )) where a denotes the last layer activation, and f ✓ its logit network.

Original: The GC of the network over a dataset D is deﬁned to be the discrete Dirichlet energy of its logit network:
Revised: The GC of the network over a dataset D is deﬁned to be the discrete Dirichlet energy of its logit network:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)∇ x f θ ( x ) (cid:107) F is the Frobenius norm of the network Jacobian.
Revised: where kr x f ✓ ( x ) k F is the Frobenius norm of the network Jacobian.

Original: Note that this deﬁnition is well-deﬁned for any differentiable model, not only a neural network, and incorporates both the model function and the dataset over which the task is determined.
Revised: Note that this deﬁnition is well-deﬁned for any differentiable model, not only a neural network, and incorporates both the model function and the dataset over which the task is determined.

Original: Next, we discuss how GC relates to familiar concepts in deep learning.
Revised: Next, we discuss how GC relates to familiar concepts in deep learning.

Original: Geometric complexity and linear models: Consider a linear transformation f ( x ) =
Revised: Geometric complexity and linear models: Consider a linear transformation f ( x ) =

Original: Ax + b from R d to R k and a dataset D = { x i } Ni =1 where x i ∈ R d .
Revised: Ax + b from R d to R k and a dataset D = { x i } Ni =1 where x i 2 R d .

Original: At each point x ∈ D , we have that (cid:107)∇ x f ( x ) (cid:107) 2 F = (cid:107) A (cid:107) 2 F , hence the GC for a linear transformation is (cid:104) f θ , D (cid:105) G 
Revised: At each point x 2 D , we have that kr x f ( x ) k 2 F = k A k 2 F , hence the GC for a linear transformation is h f ✓ , D i G =

Original: =
Revised: k

Original: (cid:107) A (cid:107) 2 F .
Revised: A k 2 F .

Original: Note, this implies that GC for linear transformations (and more generally, afﬁne maps), is independent of the dataset D and zero for constant functions.
Revised: Note, this implies that GC for linear transformations (and more generally, afﬁne maps), is independent of the dataset D and zero for constant functions.

Original: Furthermore, note that the GC in this setting coincides precisely with the L2 norm of the model weight matrix.
Revised: Furthermore, note that the GC in this setting coincides precisely with the L2 norm of the model weight matrix.

Original: Thus, enforcing an L2 norm penalty is equivalent to regularizing the GC for linear models (see Section 4 for more on that point).
Revised: Thus, enforcing an L2 norm penalty is equivalent to regularizing the GC for linear models (see Section 4 for more on that point).

Original: Geometric complexity and ReLU networks: For a ReLU network g θ : R d → R k as deﬁned in
Revised: Geometric complexity and ReLU networks: For a ReLU network g ✓ : R d ! R k as deﬁned in

Original: Deﬁnition 2.1, the GC over a dataset D has a very intuitive form.
Revised: Deﬁnition 2.1, the GC over a dataset D has a very intuitive form.

Original: Since a ReLU network parameterizes piece-wise linear functions [3], the domain can be broken into a partition of subsets X i ⊂ R d where f θ is an afﬁne map
Revised: Since a ReLU network parameterizes piece-wise linear functions [3], the domain can be broken into a partition of subsets X i ⇢ R d where f ✓ is an afﬁne map

Original: A i x + b i .
Revised: A i x + b i .

Original: Now denote by D i the points in the dataset D that fall in the linear piece deﬁned on X i .
Revised: Now denote by D i the points in the dataset D that fall in the linear piece deﬁned on X i .

Original: For every point x in X i , we have that (cid:107)∇ x f θ ( x ) (cid:107) 2 F =
Revised: For every point x in X i , we have that kr x f ✓ ( x ) k 2 F =

Original: [Equation]
Revised: k

Original: A i (cid:107) 2 F .
Revised: A i k 2 F .

Original: Since the D i ’s partition the dataset D , we obtain
Revised: Since the D i ’s partition the dataset D , we obtain

Original: [Equation]
Revised: [Equation]

Original: where n i is the number of points in the dataset D falling in X i .
Revised: where n i is the number of points in the dataset D falling in X i .

Original: We see from Eqn.
Revised: We see from Eqn.

Original: (2) that for ReLU networks the GC over the whole dataset coincides exactly with the GC on a batch B ⊂ D , provided that the proportion of points in the batch falling into each of the the linear pieces are preserved.
Revised: (2) that for ReLU networks the GC over the whole dataset coincides exactly with the GC on a batch B ⇢ D , provided that the proportion of points in the batch falling into each of the the linear pieces are preserved.

Original: This makes the evaluation of the GC on large enough batches a very good proxy to the overall GC over the dataset, and computationally tractable during training.
Revised: This makes the evaluation of the GC on large enough batches a very good proxy to the overall GC over the dataset, and computationally tractable during training.

Original: Geometric complexity and Lipschitz smoothness: One way to measure the smoothness of a function f : R d → R k on a subset X ⊂ R d is by its Lipschitz constant; i.e., the smallest f L ≥ 0 such that (cid:107) f{+ 
+}
Revised: Geometric complexity and Lipschitz smoothness: One way to measure the smoothness of a function f : R d ! R k on a subset X ⇢ R d is by its Lipschitz constant; i.e., the smallest f L {+ 
+}

Original:  ( x 1 ) − f ( x 2 )(cid:107) ≤ f L (cid:107) x
Revised: such that k f ( x 1 )  f ( x 2 )

Original:  1 − x 2 (cid:107) , for all x 1 , x 2 ∈ X .
Revised: k  f L k x 1  x 2 k , for all x 1 , x 2 2 X .

Original: Intuitively, the constant f L measures the maximal amount of variation allowed by f when the inputs change by a given amount.
Revised: Intuitively, the constant f L measures the maximal amount of variation allowed by f when the inputs change by a given amount.

Original: Using the
Revised: Using the

Original: Lipschitz constant, one can deﬁne a complexity measure of a function f as the Lipschitz constant f L of the function over the input domain R d .
Revised: Lipschitz constant, one can deﬁne a complexity measure of a function f as the Lipschitz constant f L of the function over the input domain R d .

Original: Since (cid:107)∇ x f ( x ) (cid:107) 2 F ≤ min( k, d ) (cid:107)∇ x f ( x ) (cid:107) 2 op ≤ min( k, d ) f 2 L where (cid:107) · (cid:107) op is the operator norm, we obtain a general bound on the GC by the Lipschitz complexity:(cid:104)
Revised: Since kr x f ( x ) k 2 F  min( k, d ) kr x f ( x ) k 2 op  min( k, d ) f 2 L where k · k op is the operator norm, we obtain a general bound on the GC by the Lipschitz complexity:

Original: [Equation]
Revised: [Equation]

Original: While the Lipschitz smoothness of a function provides an upper bound on GC, there are a few fundamental differences between the two quantities.
Revised: While the Lipschitz smoothness of a function provides an upper bound on GC, there are a few fundamental differences between the two quantities.

Original: Firstly, GC is data dependent: a model can have low GC while having high Lipschitz constant due to the model not being smooth in parts of the space where there is no training data.
Revised: Firstly, GC is data dependent: a model can have low GC while having high Lipschitz constant due to the model not being smooth in parts of the space where there is no training data.

Original: Secondly, the GC can be computed exactly given a model and dataset, while for neural networks only loose upper bounds are available to estimate the Lipschitz constant.
Revised: Secondly, the GC can be computed exactly given a model and dataset, while for neural networks only loose upper bounds are available to estimate the Lipschitz constant.

Original: Geometric complexity, arc length and harmonic maps: Let us start with a motivating example:
Revised: Geometric complexity, arc length and harmonic maps: Let us start with a motivating example:

Original: Consider a dataset consisting of 10 points lying on a parabola in the plane and a large ReLU deep neural network trained via gradient descent to learn a function f : R → R that ﬁts this dataset (Fig.
Revised: Consider a dataset consisting of 10 points lying on a parabola in the plane and a large ReLU deep neural network trained via gradient descent to learn a function f : R ! R that ﬁts this dataset (Fig.

Original: Throughout training the model function seems to attain minimal arc length for a given level of training error.
Revised: Throughout training the model function seems to attain minimal arc length for a given level of training error.

Original: Recalling the formula for arc length of f , which is the integral of over [ − 1 , 1] of (cid:112) 1 + f (cid:48) ( x ) 2 , and using the Taylor approximation √ 1 + x 2 ≈ 1 + x 2 2 , it follows that minimizing the arc length is equivalent to minimizing the classic Dirichlet energy:
Revised: Recalling the formula for arc length of f , which is the integral of over [  1 , 1] of p 1 + f 0 ( x ) 2 , and using the Taylor approximation p 1 + x 2 ⇡ 1 + x 2 2 , it follows that minimizing the arc length is equivalent to minimizing the classic Dirichlet energy:

Original: [Equation]
Revised: [Equation]

Original: 0 epochs, GC=0. datamodel prediction 1. 10 epochs, GC=0. 1000 epochs, GC=1.
Revised: where ⌦ =

Original: 10000 epochs, GC=1.

Original: where Ω = [ − 1 , 1] .
Revised: [  1 , 1] .

Original: The Dirichlet energy can be thought intuitively of as a measure of the variability of the function f on Ω .
Revised: The Dirichlet energy can be thought intuitively of as a measure of the variability of the function f on ⌦ .

Original: Its minimizers, subject to a boundary condition f | ∂ Ω = h , are called harmonic maps , which are maps causing the “least intrinsic stretching” of the domain Ω [85].
Revised: Its minimizers, subject to a boundary condition f | @ ⌦ = h , are called harmonic maps , which are maps causing the “least intrinsic stretching” of the domain ⌦ [91].

Original: The geometric complexity in Deﬁnition 2.1 is an unbiased estimator of a very related quantity
Revised: The geometric complexity in Deﬁnition 2.1 is an unbiased estimator of a very related quantity

Original: [Equation]
Revised: [Equation]

Original: (5) where the domain Ω is replaced by the probability distribution of the features p X and the boundary condition is replaced by the dataset D .
Revised: where the domain ⌦ is replaced by the probability distribution of the features p X and the boundary condition is replaced by the dataset D .

Original: We could call the quantity deﬁned in Eqn.
Revised: We could call the quantity deﬁned in Eqn.

Original: (5) the theoretical geometric complexity as opposed to the empirical geometric complexity in Deﬁnition 2.1.
Revised: (5) the theoretical geometric complexity as opposed to the empirical geometric complexity in Deﬁnition 2.1.

Revised: The theoretical geometric complexity is very close to a complexity measure investigated in [78], where the Jacobian of the full network is considered rather than just the logit network as we do here.

Revised: In their work, the expectation is also evaluated on the test distribution.

Revised: They observe a correlation between this complexity measure and generalization empirically in a set of extensive experiments.

Revised: In our work, we use the logit network (rather than the full network) and we evaluate the empirical geometric complexity on the train set (rather than on the test set) in order to derive theoretically that the implicit gradient regularization mechanism from [7] creates a regularizing pressure on GC (see Section 5).

Original: In the remaining sections we provide evidence that common training heuristics do indeed keep the
Revised: In the remaining sections we provide evidence that common training heuristics do indeed keep the

Original: GC low, encouraging neural networks to ﬁnd intrinsically simple solutions.
Revised: GC low, encouraging neural networks to ﬁnd intrinsically simple solutions.

Original:  Impact of initialization on geometric complexity
Revised:  Impact of initialization on geometric complexity

Original: Parameter initialization choice is an important factor in deep learning.
Revised: Parameter initialization choice is an important factor in deep learning.

Original: Although we are free to specify exact parameter initialization values, in practice, a small number of default initialization schemes have emerged to work well across a wide range of tasks [30, 37].
Revised: Although we are free to specify exact parameter initialization values, in practice, a small number of default initialization schemes have emerged to work well across a wide range of tasks [34, 41].

Original: Here, we explore the relationship between some of these initialization schemes and GC.
Revised: Here, we explore the relationship between some of these initialization schemes and GC.

Original: To begin, consider the one dimensional regression example that we introduced in Figure 1.
Revised: To begin, consider the one dimensional regression example that we introduced in Figure 1.

Original: In this experiment, we employed a standard initialization scheme to initialise the parameters: we sample them from a truncated normal distribution with variance inversely proportional to the number of input units and the bias terms were set to zero.
Revised: In this experiment, we employed a standard initialization scheme to initialise the parameters: we sample them from a truncated normal distribution with variance inversely proportional to the number of input units and the bias terms were set to zero.

Original: We observe that the initialised function on the interval [ − 1 , 1] is very close to the zero function (Fig. 1), and the zero function has zero GC.
Revised: We observe that the initialised function on the interval [  1 , 1] is very close to the zero function (Fig. 1), and the zero function has zero GC.

Original: This observation suggests that initialization schemes that have low initial GC are useful for deep learning.
Revised: This observation suggests that initialization schemes that have low initial GC are useful for deep learning.

Original: To explore this further, we consider deep ReLU networks with larger input and output spaces initialized using the same scheme as above, and measure the GC of the resulting model.
Revised: To explore this further, we consider deep ReLU networks with larger input and output spaces initialized using the same scheme as above, and measure the GC of the resulting model.

Original: Speciﬁcally, consider the initialized ReLU network given by f θ 0 : R d → R k with d = 150528 and k = 1000 , with parameter initialization θ 0 , and varying network depth.
Revised: Speciﬁcally, consider the initialized ReLU network given by f ✓ 0 : R d ! R k with d = 150528 and k = 1000 , with parameter initialization ✓ 0 , and varying network depth.

Original: We measure the ReLU network output size by recording the mean and maximum output values, evaluated using input values along the line P 1 + ( P 2 − P 1 ) x with x ∈ [0 , 1] between two diagonal points P 1 and P 2 of the hyper-cube [ − 1 , 1] d .
Revised: We measure the ReLU network output size by recording the mean and maximum output values, evaluated using input values along the line P 1 + ( P 2  P 1 ) x with x 2 [0 , 1] between two diagonal points P 1 and P 2 of the hyper-cube [  1 , 1] d .

Original: We observe that these ReLU networks initialize to functions close to the zero function, and become progressively closer to a zero valued function as the number of layer increases (Fig. 2).
Revised: We observe that these ReLU networks initialize to functions close to the zero function, and become progressively closer to a zero valued function as the number of layer increases (Fig. 2).

Original: For ReLU networks, this is not entirely surprising.
Revised: For ReLU networks, this is not entirely surprising.

Original: With biases initialised to zero, we can express a
Revised: With biases initialised to zero, we can express a

Original: ReLU network in a small neighborhood of a given point x ∈ R d as a product of matrices f θ 0 ( x ) =
Revised: ReLU network in a small neighborhood of a given point x 2 R d as a product of matrices f ✓ 0 ( x ) =

Original: W 1 P 2 W 2 P 3 W 3 · · · P l W l x, where the W i ’s are the weight matrices and the P
Revised: W 1 P 2 W 2 P 3 W 3 · · · P l W l x, where the W i ’s are the weight matrices and the P

Original: i ’s are diagonal matrices with 0 and 1 on their diagonals.
Revised: i ’s are diagonal matrices with 0 and 1 on their diagonals.

Original: This representation makes it clear that at initialization the ReLU network passes through the origin; 
Revised: This representation makes it clear that at initialization the ReLU network passes through the origin; i.e., f ✓ 0 (0)

Original: i.e., f θ 0 (0) = 0 .
Revised: = 0 .

Original: Furthermore, with weight matrices initialised around zero, using a scaling that can reduce the spread of the distribution as the matrices grow, we can expect that deeper ReLU networks generated by multiplying a large number of small-valued weights, can produce output values close to zero (for input values taken from a hyper-cube [ − 1 , 1] d ).
Revised: Furthermore, with weight matrices initialised around zero, using a scaling that can reduce the spread of the distribution as the matrices grow, we can expect that deeper ReLU networks generated by multiplying a large number of small-valued weights, can produce output values close to zero (for input values taken from a hyper-cube [  1 , 1] d ).

Original: We extend these results further to include additional initialization schemes and experimentally conﬁrm that the GC can be brought close to zero with a sufﬁcient number of layers.In fact, this is true not
Revised: We extend these results further to include additional initialization schemes and experimentally conﬁrm that the GC can be brought close to zero with a sufﬁcient number of layers.

Original: [Figure] only for ReLU networks with the standard initialization scheme, but for a number of other common activation functions and initialization setups [30, 37] [Figure] and even on domains much larger than the normalized hyper-cube (Fig. 2 and SM Section C.1).
Revised: In fact, this is true not only for ReLU networks with the standard initialization scheme, but for a number of other common activation functions and initialization setups [34, 41], and even on domains much larger than the normalized hyper-cube (Fig. 2 and SM Section C.1).

Revised: Theoretically, it has been shown very recently in [5] (their Theorem 5), that under certain technical conditions, a neural network at random initialization will converge to a constant function (which has GC equal to zero) as the number of layers increases.

Original:  Impact of explicit regularization on geometric complexity
Revised:  Impact of explicit regularization on geometric complexity

Original: Next, we explore the relationship between GC and various forms of explicit regularization using a combination of theoretical and empirical results.[Figure]
Revised: Next, we explore the relationship between GC and various forms of explicit regularization using a combination of theoretical and empirical results.

Original: L2 regularization:
Revised: L2 regularization:

Original: In L2 regularization, the Euclidean norm of the parameter vector is explicitly added to a loss function, so as to penalize solutions that have excessively large parameter values.
Revised: In L2 regularization, the Euclidean norm of the parameter vector is explicitly added to a loss function, so as to penalize solutions that have excessively large parameter values.

Original: This is one of the simplest, and most widely used forms of regularization.
Revised: This is one of the simplest, and most widely used forms of regularization.

Original: For linear models( f θ ( x ) =
Revised: For linear models

Original:  Ax + b ), we saw in Section 2 that the GC coincides with the Frobenius norm of the matrix
Revised: ( f ✓ ( x ) = Ax + b ), we saw in Section 2 that the GC coincides with the Frobenius norm of the matrix

Original: A .
Revised: A .

Original: This means that standard L2 norm penalties on the weight matrix coincide in this case with a direct explicit regularization of the GC of the linear model.
Revised: This means that standard L2 norm penalties on the weight matrix coincide in this case with a direct explicit regularization of the GC of the linear model.

Original: For non-linear deep neural networks, we cannot directly identify the L2 norm penalty with the model GC.
Revised: For non-linear deep neural networks, we cannot directly identify the L2 norm penalty with the model GC.

Original: However, in the case of ReLU networks, for each input point x , the network output y can be written in a neighborhood of x as y =
Revised: However, in the case of ReLU networks, for each input point x , the network output y can be written in a neighborhood of x as y =

Original: P l W l . . .
Revised: P l W l . . .

Original: P 1 W 1 x + c , where c is a constant, the P i ’s are diagonal matrices with 0 and 1 on the diagonal, and the W i ’s are the network weight matrices.
Revised: P 1 W 1 x + c , where c is a constant, the P i ’s are diagonal matrices with 0 and 1 on the diagonal, and the W i ’s are the network weight matrices.

Original: This means that the derivative at x of the network coincides with the matrix P l W l . . .
Revised: This means that the derivative at x of the network coincides with the matrix P l W l . . .

Original: P 1 W 1 .
Revised: P 1 W 1 .

Original: Therefore, the GC is just the Frobenius norm of the product of matrices.
Revised: Therefore, the GC is just the Frobenius norm of the product of matrices.

Original: Now, an L2 penalty (cid:107) W l (cid:107) 2 F + · · · + (cid:107) W 1 (cid:107) 2 F encourages small numbers in the values of the weight matrices, which in turn is likely to encourage small numbers for the values in the product P l W l . . .
Revised: Now, an L2 penalty k W l k 2 F + · · · + k W 1 k 2 F encourages small numbers in the values of the weight matrices, which in turn is likely to encourage small numbers for the values in the product P l W l . . .

Original: P 1 W 1 , resulting in a lower GC.
Revised: P 1 W 1 , resulting in a lower GC.

Original: We can demonstrate this relationship empirically, by training a selection of neural networks with L regularization, each with a different regularization strength.
Revised: We can demonstrate this relationship empirically, by training a selection of neural networks with L regularization, each with a different regularization strength.

Original: We measure the GC for each network at the time of maximum test accuracy.
Revised: We measure the GC for each network at the time of maximum test accuracy.

Original: We observe empirically that strengthening L2 regularization coincides with a decrease in GC values (Fig. 3).
Revised: We observe empirically that strengthening L2 regularization coincides with a decrease in GC values (Fig. 3).

Original: Lipschitz regularization via spectral norm regularization: A number of explicit regularization schemes have been used to tame the Lipschitz smoothness of the model function and produce smoother models with reduced test error [2, 3, 23, 25, 35, 49, 67, 75, 90].
Revised: Lipschitz regularization via spectral norm regularization: A number of explicit regularization schemes have been used to tame the Lipschitz smoothness of the model function and produce smoother models with reduced test error [2, 3, 26, 28, 39, 53, 71, 81, 96].

Original: Smoothness regularization has also been shown to be beneﬁcial outside the supervised learning context, in GANs [15, 67, 92] and reinforcement learning [12, 31].
Revised: Smoothness regularization has also been shown to be beneﬁcial outside the supervised learning context, in GANs [17, 71, 98] and reinforcement learning [14, 35].

Original: One successful approach to regularising the Lipschitz constant of a neural network with 1-Lipschitz activation functions (e.g. ReLU, ELU) is to constrain the spectral norm of each layer of the network (i.e., the maximal singular values σ max ( W i ) of the weight matrices
Revised: One successful approach to regularising the Lipschitz constant of a neural network with 1-Lipschitz activation functions (e.g. ReLU, ELU) is to constrain the spectral norm of each layer of the network (i.e., the maximal singular values  max ( W i ) of the weight matrices

Original: W i ), since the product of the spectral norms of the networks weight matrices is an upper bound to the Lipschitz constant of the model: f L ≤ σ max ( W 1 ) · · · σ max ( W l ) .
Revised: W i ), since the product of the spectral norms of the networks weight matrices is an upper bound to the Lipschitz constant of the model: f L   max ( W 1 ) · · ·  max ( W l ) .

Original: Using inequality (3), we see that any approach that constrains the Lipschitz constant of the model constrains GC.
Revised: Using inequality (3), we see that any approach that constrains the Lipschitz constant of the model constrains GC.

Original: To conﬁrm this theoretical prediction, we train a ResNet18 model on CIFAR10 [54] and regularize using spectral regularization [90], an approach which adds the regularizer λ 2 (cid:80) l ( σ max ( W l )) 2 to the model loss function.
Revised: To conﬁrm this theoretical prediction, we train a ResNet18 model on CIFAR10 [58] and regularize using spectral regularization [96], an approach which adds the regularizer  P l (  max ( W l )) 2 to the model loss function.

Original: We observe that GC decreases as the strength of spectral regularization λ increases (Fig. 3).
Revised: We observe that GC decreases as the strength of spectral regularization  increases (Fig. 3).

Original: Noise regularization:
Revised: Noise regularization:

Original: The addition of noise during training is known to be an effective form of regularization.
Revised: The addition of noise during training is known to be an effective form of regularization.

Original: For instance, it has been demonstrated [13] that the addition of noise to training labels during the optimisation of a least-square loss using SGD exerts a regularising pressure on
Revised: For instance, it has been demonstrated [15] that the addition of noise to training labels during the optimization of a least-square loss using SGD exerts a regularising pressure on

Original: [Equation] 2 / | D | .
Revised: P ( x,y ) 2 D kr ✓ f ✓ ( x ) k 2 / | D | .

Original: Here, we demonstrate empirically that the GC of a ResNet18 trained on CIFAR10 reduces as the proportion of label noise increases (Fig. 3, middle).
Revised: Here, we demonstrate empirically that the GC of a ResNet18 trained on CIFAR10 reduces as the proportion of label noise increases (Fig. 3, middle).

Original: The same is true of an MLP trained on MNIST [19] (Fig. 3, right).
Revised: The same is true of an MLP trained on MNIST [22] (Fig. 3, right).

Original: In Section 5, we provide a theoretical argument which justiﬁes these experiments, showing that a regularizing pressure on (cid:107)∇ θ f θ ( x ) (cid:107) 2 transfers to a regularizing pressure on (cid:107)∇ x f θ ( x ) (cid:107) 2 .
Revised: In Section 5, we provide a theoretical argument which justiﬁes these experiments, showing that a regularizing pressure on kr ✓ f ✓ ( x ) k 2 transfers to a regularizing pressure on kr x f ✓ ( x ) k 2 .

Original: Thus, label noise in SGD in turn translates into a regularizing pressure on the GC in the case of neural networks.
Revised: Thus, label noise in SGD in turn translates into a regularizing pressure on the GC in the case of neural networks.

Original: Flatness regularization:
Revised: Flatness regularization:

Original: In ﬂatness regularization, an explicit gradient penalty term, (cid:107)∇ θ L B (cid:107) 2 is added to the loss (where L B is the loss evaluated across a batch B ).
Revised: In ﬂatness regularization, an explicit gradient penalty term, kr ✓ L B k 2 is added to the loss (where L B is the loss evaluated across a batch B ).

Original: It has been observed in practice that ﬂatness regularization can be effective in many deep learning settings, from supervised learning [27, 81] to GAN training [5, 66, 69, 74, 76].
Revised: It has been observed in practice that ﬂatness regularization can be effective in many deep learning settings, from supervised learning [31, 87] to GAN training [6, 70, 73, 80, 82].

Original: Flatness regularization penalizes learning trajectories that follow steep slopes across the loss surface, thereby encouraging learning to follow shallower slopes toward ﬂatter regions of the loss surface.
Revised: Flatness regularization penalizes learning trajectories that follow steep slopes across the loss surface, thereby encouraging learning to follow shallower slopes toward ﬂatter regions of the loss surface.

Original: We demonstrate empirically that GC decreases as the strength of ﬂatness regularization increases (Fig. 3).
Revised: We demonstrate empirically that GC decreases as the strength of ﬂatness regularization increases (Fig. 3).

Original: In the next section, we will provide a theoretical argument that ﬂatness regularization can control GC.
Revised: In the next section, we will provide a theoretical argument that ﬂatness regularization can control GC.

Original: Explicit GC regularization and Jacobian regularization: All the forms of regularization above have known beneﬁts for improving the test accuracy in deep learning.
Revised: Explicit GC regularization and Jacobian regularization: All the forms of regularization above have known beneﬁts for improving the test accuracy in deep learning.

Original: As we saw, they all also implicitly regularize GC.
Revised: As we saw, they all also implicitly regularize GC.

Original: This raises the question as whether regularizing for GC directly and independently of any other mechanism is sufﬁcient to improve model performance.
Revised: This raises the question as whether regularizing for GC directly and independently of any other mechanism is sufﬁcient to improve model performance.

Original: Namely, we can add GC computed on the batch to the loss L reg ( θ ) =
Revised: Namely, we can add GC computed on the batch to the loss L reg ( ✓ ) =

Original: L B ( θ ) + α/B (cid:80) x ∈ B (cid:107)∇ x f θ ( x ) (cid:107) 2 F .
Revised: L B ( ✓ ) + ↵ /B [Equation] F .

Original: This is actually a known form of explicit regularization, called Jacobian regularization , which has been correlated with increased generalization but also robustness to input shift [42, 84, 89, 90].
Revised: This is actually a known form of explicit regularization, called Jacobian regularization , which has been correlated with increased generalization but also robustness to input shift [46, 90, 95, 96].

Original: [84] speciﬁcally shows that adding Jacobian regularization to the loss function can lead to an increase in test set accuracy (their Tables III, IV, and V).
Revised: [90] speciﬁcally shows that adding Jacobian regularization to the loss function can lead to an increase in test set accuracy (their Tables III, IV, and V).

Original: In SM Section C.4 we train a MLP on MNIST and a
Revised: In SM Section C.4 we train a MLP on MNIST and a

Original: ResNet18 on CIFAR10 regularized explicitly with the geometric complexity.
Revised: ResNet18 on CIFAR10 regularized explicitly with the geometric complexity.

Original: We observe an increase of test accuracy and a decrease of GC with more regularization.
Revised: We observe an increase of test accuracy and a decrease of GC with more regularization.

Original: Related regularizers include gradient penalties of the form(cid:80) x
Revised: Related regularizers include gradient penalties of the form

Original:  ( ||∇ x f θ ( x ) || − K ) 2 which have been used for GAN training [25, 35, 48].
Revised: P x ( || r x f ✓ ( x ) ||  K ) 2 which have been used for GAN training [28, 39, 52].

Original: We leave the full investigation of the importance of GC outside supervised learning for future work.
Revised: We leave the full investigation of the importance of GC outside supervised learning for future work.

Original:  Impact of implicit regularization on geometric complexity
Revised:  Impact of implicit regularization on geometric complexity

Original: Implicit regularization is a hidden form of regularization that emerges as a bi-product of model training.
Revised: Implicit regularization is a hidden form of regularization that emerges as a bi-product of model training.

Original: Unlike explicit regularization, it is not explicitly added to a loss function.
Revised: Unlike explicit regularization, it is not explicitly added to a loss function.

Original: In deep learning settings where no explicit regularization is used, it is the only form of regularization.
Revised: In deep learning settings where no explicit regularization is used, it is the only form of regularization.

Original: Here, we use a combination of theoretical and empirical results to argue that some recently identiﬁed implicit regularization mechanisms in gradient descent [6, 61, 81] exert a regularization pressure on geometric complexity.
Revised: Here, we use a combination of theoretical and empirical results to argue that some recently identiﬁed implicit regularization mechanisms in gradient descent [7, 65, 87] exert a regularization pressure on geometric complexity.

Original: Our argument proceeds as follows: 1) we identify a mathematical term (the implicit gradient regularization term) that characterizes implicit regularization in gradient descent, 2) we demonstrate that this term depends on model gradients, 3) we identify the conditions where model gradient terms apply a regularization pressure on geometric complexity.
Revised: Our argument proceeds as follows: 1) we identify a mathematical term (the implicit gradient regularization term) that characterizes implicit regularization in gradient descent, 2) we demonstrate that this term depends on model gradients, 3) we identify the conditions where model gradient terms apply a regularization pressure on geometric complexity.

Original: Step 1: The implicit regularization that we consider emerges as a bi-product of the discrete nature of gradient descent updates.
Revised: Step 1: The implicit regularization that we consider emerges as a bi-product of the discrete nature of gradient descent updates.

Original: In particular it has been shown that a discrete gradient update θ (cid:48) = [Equation] − h ∇ θ L B ( θ ) over a batch of data implicitly minimizes a modiﬁed loss
Revised: In particular it has been shown that a discrete gradient update ✓ 0 = ✓  h r ✓ L B ( ✓ ) over a batch of data implicitly minimizes a modiﬁed loss, e

Original: [Equation]
Revised: [Equation]

Original:  where the second term is called the Implicit Gradient Regularizer (IGR) [6].
Revised: , where the second term is called the Implicit Gradient Regularizer (IGR) [7].

Original: Gradient descent optimisation is better characterized as a continuous ﬂow along the gradient of the modiﬁed loss, rather than the original unmodiﬁed loss.
Revised: Gradient descent optimization is better characterized as a continuous ﬂow along the gradient of the modiﬁed loss, rather than the original unmodiﬁed loss.

Original: By inspection, the IGR term implicitly regularizes training toward trajectories with smaller loss gradients toward ﬂatter region on the loss surface [6] .
Revised: By inspection, the IGR term implicitly regularizes training toward trajectories with smaller loss gradients toward ﬂatter region on the loss surface [7] .

Original: Step 2: Next, we develop this implicit regularizer term for a multi-class classiﬁcation cross-entropy loss term.
Revised: Step 2: Next, we develop this implicit regularizer term for a multi-class classiﬁcation cross-entropy loss term.

Original: We can write (see SM Section A.2 for details):
Revised: We can write (see SM Section A.2 for details):

Original: [Equation]
Revised: [Equation]

Original: where C B ( θ ) measures the batch gradient alignment
Revised: where C B ( ✓ ) measures the batch gradient alignment

Original: [Equation]
Revised: [Equation]

Original: (7) and A B ( θ ) measures the label gradient alignment :
Revised: and A B ( ✓ ) measures the label gradient alignment :

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) ix ( x ) = a ( f θ ( x ))
Revised: where ✏ ix ( x ) = a ( f ✓ ( x ))

Original: i − y
Revised: i  y

Original: i is the signed residual and a denotes the activation function of the last layer.
Revised: i is the signed residual and a denotes the activation function of the last layer.

Original: Note that this residual term arises in our calculation using ∇ θ L ( x, y, θ ) =
Revised: Note that this residual term arises in our calculation using r ✓ L ( x, y, ✓ ) =

Original: (cid:15) 1 x ( θ ) ∇ θ f 1 θ ( x ) + · · · + (cid:15) kx ( θ ) ∇ θ f kθ ( x ) (see SM Section A.1, Eqn.
Revised: ✏ 1 x ( ✓ ) r ✓ f 1 ✓ ( x ) + · · · + ✏ kx ( ✓ ) r ✓ f k ✓ ( x ) (see SM Section A.1, Eqn.

Original: This development also extends to other widely used loss functions such as the cross entropy and least-square loss.
Revised: This development also extends to other widely used loss functions such as the cross entropy and least-square loss.

Original: Step 3:
Revised: Step 3:

Original: Now, from the modiﬁed loss in Eqn.
Revised: Now, from the modiﬁed loss in Eqn.

Original: (6), we can observe the conditions under which SGD puts an implicit pressure on the gradient norms (cid:107)∇ θ f iθ ( x ) (cid:107) 2 to be small: the batch gradient alignment and label gradient alignment terms must be small relative to the gradient norms, or positive valued.
Revised: (6), we can observe the conditions under which SGD puts an implicit pressure on the gradient norms kr ✓ f i ✓ ( x ) k 2 to be small: the batch gradient alignment and label gradient alignment terms must be small relative to the gradient norms, or positive valued.

Original: We also derive conditions under which this implicit pressure on (cid:107)∇ θ f θ ( x ) (cid:107) 2 F transfers to a regularizing pressure on (cid:107)∇ x f θ ( x ) (cid:107) 2 F (see SM Section A.3 for the proof):.
Revised: We also derive conditions under which this implicit pressure on kr ✓ f ✓ ( x ) k 2 F transfers to a regularizing pressure on kr x f ✓ ( x ) k 2 F (see SM Section A.3 for the proof):.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Consider a logit network f θ : R d → R k with l layers, then we have the following inequality
Revised: Consider a logit network f ✓ : R d ! R k with l layers, then we have the following inequality

Original: [Equation]
Revised: [Equation]

Original: where T i is the transfer function for layer i given by
Revised: where T i is the transfer function for layer i given by

Original: [Equation]
Revised: [Equation]

Original: where h i is the subnetwork to layer i and σ max ( A ) is the maximal singular value of matrix A (i.e., its spectral norm).
Revised: where h i is the subnetwork to layer i and  max ( A ) is the maximal singular value of matrix A (i.e., its spectral norm).

Original: Here, we can see that for any settings where the sum T 21 ( x ) + · · · + T 2 l ( x ) of squared transfer functions diminishes slower than the gradient norm (cid:107)∇ θ f θ ( x ) (cid:107) 2 F during training, we expect that implicit gradient regularization will apply a regularization pressure on GC.
Revised: Here, we can see that for any settings where the sum T 21 ( x ) + · · · + T 2 l ( x ) of squared transfer functions diminishes slower than the gradient norm kr ✓ f ✓ ( x ) k 2 F during training, we expect that implicit gradient regularization will apply a regularization pressure on GC.

Original: An immediate prediction arising under these conditions is that the size of the regularization pressure on geometric complexity will depend on the implicit regularization rate h/B in Eqn.
Revised: An immediate prediction arising under these conditions is that the size of the regularization pressure on geometric complexity will depend on the implicit regularization rate h/B in Eqn.

Original: (6) (Note that the[Figure] ratio h/B has been linked to implicit regularization strength in many instances [33, 82, 65, 55, 61]).
Revised: (6) (Note that the ratio h/B has been linked to implicit regularization strength in many instances [37, 88, 69, 59, 65]).

Original: Speciﬁcally, under these conditions, networks trained with larger learning rates or smaller batch sizes, or both, will apply a stronger regularization pressure on geometric complexity,
Revised: Speciﬁcally, under these conditions, networks trained with larger learning rates or smaller batch sizes, or both, will apply a stronger regularization pressure on geometric complexity,

Original: We test this prediction by performing experiments on ResNet18 trained on CIFAR10 and show results in Figure 4.
Revised: We test this prediction by performing experiments on ResNet18 trained on CIFAR10 and show results in Figure 4.

Original: The results show that while all models achieve a zero train loss, consistently the higher the learning rate, the higher the test accuracy and the lower the GC.
Revised: The results show that while all models achieve a zero train loss, consistently the higher the learning rate, the higher the test accuracy and the lower the GC.

Original: Similarly, we observe that the lower the batch size, the lower the GC and the higher the test accuracy.
Revised: Similarly, we observe that the lower the batch size, the lower the GC and the higher the test accuracy.

Original: The increased performance of lower batch sizes has been long studied in deep learning, under the name ‘the generalisation gap’ [41, 47].
Revised: The increased performance of lower batch sizes has been long studied in deep learning, under the name ‘the generalisation gap’ [45, 51].

Original: Crucially however, this gap was recently bridged [28], showing that full batch training can achieve the same test set performance as mini-batch gradient descent.
Revised: Crucially however, this gap was recently bridged [32], showing that full batch training can achieve the same test set performance as mini-batch gradient descent.

Original: To obtain these results, they use an explicit regularizer similar to the implicit regularizer in
Revised: To obtain these results, they use an explicit regularizer similar to the implicit regularizer in

Original: Eqn.
Revised: Eqn.

Original: (6), introduced to compensate for the diminished implicit regularization in the full batch case.
Revised: (6), introduced to compensate for the diminished implicit regularization in the full batch case.

Original: Their results further strengthen our hypothesis that implicit regularization via GC results in improved generalisation.Geometric complexity and double descent
Revised: Their results further strengthen our hypothesis that implicit regularization via GC results in improved generalisation.

Original: When complexity is measured using a simple parameter count, a double descent phenomena has been consistently observed: as the number of parameters increases, the test accuracy decreases at ﬁrst, before increasing again, followed by a second descent toward a low test error value [10, 11, 71].
Revised: When complexity is measured using a simple parameter count, a double descent phenomena has been consistently observed: as the number of parameters increases, the test accuracy decreases at ﬁrst, before increasing again, followed by a second descent toward a low test error value [12, 13, 75].

Original: An excellent overview of the double descent phenomena in deep learning can be found in [10], together with connections to smoothness as an inductive bias of deep networks and the role of optimisation.
Revised: An excellent overview of the double descent phenomena in deep learning can be found in [12], together with connections to smoothness as an inductive bias of deep networks and the role of optimization.

Original: To explore the double descent phenomena using GC we follow the set up introduced in [71]: we train multiple ResNet18 networks on CIFAR10 with increasing layer width, and show results in Fig.
Revised: To explore the double descent phenomena using GC we follow the set up introduced in [75]: we train multiple ResNet18 networks on CIFAR10 with increasing layer width, and show results in Fig.

Original: We make two observations: ﬁrst, like the test loss, GC follows a double descent curve as width increases; second, when plotting GC against the test loss, we observe a U-shape curve, recovering the traditional expected behaviour of complexity measures.
Revised: We make two observations: ﬁrst, like the test loss, GC follows a double descent curve as width increases; second, when plotting GC against the test loss, we observe a U-shape curve, recovering the traditional expected behaviour of complexity measures.

Original: Importantly, we observe the connection between the generalisation properties of overparametrized models and GC: after the critical region, increase in model size leads to a decrease in GC.
Revised: Importantly, we observe the connection between the generalisation properties of overparametrized models and GC: after the critical region, increase in model size leads to a decrease in GC.

Original: This provides further evidence that GC is able to capture model capacity in a meaningful way, suggestive of a reconciliation of traditional complexity theory and deep learning.[Figure]
Revised: This provides further evidence that GC is able to capture model capacity in a meaningful way, suggestive of a reconciliation of traditional complexity theory and deep learning.

Original:  Related and Future Work
Revised:  Related and Future Work

Original: The aim of this work is to introduce a measure which captures the complexity of neural networks.
Revised: The aim of this work is to introduce a measure which captures the complexity of neural networks.

Original: There are many other approaches aiming at doing so, ranging from naive parameter count and more data driven approaches [43, 56, 63] to more traditional measures such as the VC dimension and
Revised: There are many other approaches aiming at doing so, ranging from naive parameter count and more data driven approaches [47, 60, 67] to more traditional measures such as the VC dimension and

Original: Rademacher complexity [9, 44, 53, 86] which focus on the entire hypothesis space.
Revised: Rademacher complexity [11, 48, 57, 92] which focus on the entire hypothesis space.

Original: As we saw in Fig.
Revised: [77] analyzes existing measures of complexity and shows that they increase as the size of the hidden units increases, and thus cannot explain behaviours observed in over-parameterized neural networks (their Figure 5). [48] performs an extensive study of existing complexity measures and their correlation with generalization. They ﬁnd that ﬂatness has the strongest positive connection with generalization. [9] provides a generalization bound depending on classiﬁcation margins and the product of spectral norms of the model’s weights and shows how empirically the product of spectral norms correlates with empirical risk.

Revised: [20] connects Lipschitz smoothness with overparametrization, by providing a probabilistic lower bound on the model’s Lipschitz constant based on its training performance and the inverse of its number of parameters.

Revised: In concurrent work [29] discusses the connection between the Jacobian and the Hessian of the loss with respect to inputs, namely the empirical average of kr x L k 2 and kr 2 x L k 2 over the training set and shows that they follow a double-descent curve.

Revised: [78] investigates empirically a complexity measure similar to GC using the Jacobian norm of the full network (rather than the logit network) and evaluating it on test distribution (rather than the train distribution).

Revised: They show a correlation between this measure and generalization in an extensive set of experiments.

Revised: As we saw in Fig.

Original: 1, the interpolating ReLU network with minimal GC is also the piecewise linear function with minimal volume or length [20].
Revised: 1, the interpolating ReLU network with minimal GC is also the piecewise linear function with minimal volume or length [23].

Original: In 1D this minimal function can be described only with the information given by the data points.
Revised: In 1D this minimal function can be described only with the information given by the data points.

Original: This description with minimal information is reminiscent of the Kolmogorov complexity [78] and the minimum description length [39], and we believe that the exact relationship between these notions, GC, and minimal volume is worth investigating.
Revised: This description with minimal information is reminiscent of the Kolmogorov complexity [84] and the minimum description length [43], and we believe that the exact relationship between these notions, GC, and minimal volume is worth investigating.

Original: Similarly, [1] argues that ﬂat solutions have low information content, and for neural networks these ﬂat regions are also the regions of low loss gradient and thus of low GC, as explained by the Transfer Theorem in Section 5.
Revised: Similarly, [1] argues that ﬂat solutions have low information content, and for neural networks these ﬂat regions are also the regions of low loss gradient and thus of low GC, as explained by the Transfer Theorem in Section 5.

Original: Another recent measure of complexity is the effective dimension which is deﬁned in relation to the training data, but is computed using the spectral decomposition of the loss Hessian [63], making ﬂat regions in the loss surface also regions of low complexity w.r.t.
Revised: Another recent measure of complexity is the effective dimension which is deﬁned in relation to the training data, but is computed using the spectral decomposition of the loss Hessian [67], making ﬂat regions in the loss surface also regions of low complexity w.r.t.

Original: this measure.
Revised: this measure.

Original: This hints toward the effective dimension being related to GC.
Revised: This hints toward the effective dimension being related to GC.

Original: Note that similarly to GC, the effective dimension can also capture the double descent phenomena.
Revised: Note that similarly to GC, the effective dimension can also capture the double descent phenomena.

Original: The effective dimension is an efﬁcient mechanism for model selection [63], which our experiments seem also to indicate may be the case for GC.
Revised: The effective dimension is an efﬁcient mechanism for model selection [67], which our experiments seem also to indicate may be the case for GC.

Original: Note that the generalized degrees of freedom (which considers the sensitivity of a classiﬁer to the labels, rather than to the features as GC does) explored in [34] in the context of deep learning also captures the double-descent phenomena.
Revised: Note that the generalized degrees of freedom (which considers the sensitivity of a classiﬁer to the labels, rather than to the features as GC does) explored in [38] in the context of deep learning also captures the double-descent phenomena.

Original: GC is close to considerations about smoothness [75] and the Sobolev norm implicit regularization [61].
Revised: GC is close to considerations about smoothness [81] and the Sobolev norm implicit regularization [65].

Original: While GC and Lipchitz smoothness are connected, here we focused on a tractable quantity for neural networks and its connections with existing regularizers.
Revised: While GC and Lipchitz smoothness are connected, here we focused on a tractable quantity for neural networks and its connections with existing regularizers.

Original: Smoothness regularization has been particularly successful in the GAN literature [16, 35, 48, 67, 92], and the connection between
Revised: Smoothness regularization has been particularly successful in the GAN literature [18, 39, 52, 71, 98], and the connection between

Original: Lipschitz smoothness and GC begs the question of whether their success is due to their implicit regularization of GC, but we leave the application of GC outside supervised learning for future work.
Revised: Lipschitz smoothness and GC begs the question of whether their success is due to their implicit regularization of GC, but we leave the application of GC outside supervised learning for future work.

Original: The Dirichlet energy in Eqn.
Revised: The Dirichlet energy in Eqn.

Original: (4) is a well-known quantity in harmonic function theory [24] and a fundamental concept throughout mathematics, physics and, more recently, 3D modeling [85], manifold learning [17], and image processing [29, 77].
Revised: (4) is a well-known quantity in harmonic function theory [27] and a fundamental concept throughout mathematics, physics and, more recently, 3D modeling [91], manifold learning [19], and image processing [33, 83].

Original: Minimizers of the Dirichlet energy are harmonic and thus enjoy certain guarantees in regularity; i.e., any such solution is a smooth function.
Revised: Minimizers of the Dirichlet energy are harmonic and thus enjoy certain guarantees in regularity; i.e., any such solution is a smooth function.

Original: It has been demonstrated that for mean squared error regression on wide-shallow ReLU networks, gradient descent is biased towards smooth functions at interpolation [45].
Revised: It has been demonstrated that for mean squared error regression on wide-shallow ReLU networks, gradient descent is biased towards smooth functions at interpolation [49].

Original: Similarly, our work suggests that neural networks, through a mechanism of implicit regularization of GC, are biased towards minimal Dirichlet energy and thus encourage smooth interpolation.
Revised: Similarly, our work suggests that neural networks, through a mechanism of implicit regularization of GC, are biased towards minimal Dirichlet energy and thus encourage smooth interpolation.

Original: It may be interesting to understand how the relationship between GC, Dirichlet energy, and harmonic theory can help futher improve the learning process, in particular in transfer learning, or OOD generalization.
Revised: It may be interesting to understand how the relationship between GC, Dirichlet energy, and harmonic theory can help futher improve the learning process, in particular in transfer learning, or in out-of-distribution generalization.

Original:  Discussion
Revised:  Discussion

Original: In terms of limitations , the theoretical arguments presented in this work have a focus on ReLU activations and DNN architectures, with log-likelihood losses coming from the exponential family, such as multi-dimensional regression with least-square losses, or multi-class classiﬁcation with cross-entropy loss.
Revised: In terms of limitations , the theoretical arguments presented in this work have a focus on ReLU activations and DNN architectures, with log-likelihood losses coming from the exponential family, such as multi-dimensional regression with least-square losses, or multi-class classiﬁcation with cross-entropy loss.

Original: The experimental results are obtained on the image datasets MNIST and CIFAR using DNN and ResNet architectures.
Revised: The experimental results are obtained on the image datasets MNIST and CIFAR using DNN and ResNet architectures.

Original: In terms of societal impact , we are not introducing new training methods, but focus on providing a better understanding of the impact of common existing training methods.
Revised: In terms of societal impact , we are not introducing new training methods, but focus on providing a better understanding of the impact of common existing training methods.

Original: We hope that this understanding will ultimately lead to more efﬁcient training techniques.
Revised: We hope that this understanding will ultimately lead to more efﬁcient training techniques.

Original: While existing training methods may have their own risk, we do not foresee any potential negative societal impact of this work.
Revised: While existing training methods may have their own risk, we do not foresee any potential negative societal impact of this work.

Original: In conclusion, altogether, geometric complexity provides a useful lens for understanding deep learning and sheds light into why neural networks are able to achieve low test error with highly expressive models.
Revised: In conclusion, altogether, geometric complexity provides a useful lens for understanding deep learning and sheds light into why neural networks are able to achieve low test error with highly expressive models.

Original: We hope this work will encourage further research around this new connection, and help to better understand current best practices in model training as well as discover new ones.
Revised: We hope this work will encourage further research around this new connection, and help to better understand current best practices in model training as well as discover new ones.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank Chongli Qin, Samuel Smith, Soham De, Yan Wu, and the reviewers for helpful discussions and feedback as well as Patrick Cole, Xavi Gonzalvo, and Shakir Mohamed for their support.
Revised: We would like to thank Chongli Qin, Samuel Smith, Soham De, Yan Wu, and the reviewers for helpful discussions and feedback as well as Patrick Cole, Xavi Gonzalvo, and Shakir Mohamed for their support.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The data is widely available (MNIST and CIFAR10).
Revised: [No] The data is widely available (MNIST and CIFAR10).

Original: The code we use to train the model is standard training code using standard neural network architectures (MLP and ResNet18) available out-of-the-box in most ml libraries.
Revised: The code we use to train the model is standard training code using standard neural network architectures (MLP and ResNet18) available out-of-the-box in most ml libraries.

Original: We included detailed instructions with all the hyper-parameter used in the supplemental material to reproduce them.
Revised: We included detailed instructions with all the hyper-parameter used in the supplemental material to reproduce them.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?[Yes]
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original:  (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We use the CIFAR10 and MNIST datasets for some of our experimental results.
Revised: [Yes] We use the CIFAR10 and MNIST datasets for some of our experimental results.

Original: We cite both in the paper, for CIFAR following the guidance on Krizhevsky’s homepage: https://www.cs.toronto.edu/ kriz/cifar.html.
Revised: We cite both in the paper, for CIFAR following the guidance on Krizhevsky’s homepage: https://www.cs.toronto.edu/ kriz/cifar.html.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: Supplementary Material

Original: A Proofs for Section

Original: A.1 Gradient structure in the exponential family . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Computing the expanded modiﬁed loss from Section 5 . . . . . . . . . . . . . . .

Original: A.3 The Transfer Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Experiment details

Original: C Additional Experiments

Original: C

Original: C

Original: C

Original: complexity

Original: D Comparison of the Geometric Complexity to other complexity measures

Original: D.1 Rademacher Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 VC dimension .

Original: D.3 Sharpness and Hessian related measures . .

Original: A Proofs for Section

Original: In Section A.1 we derive that for a large class of models comprising regression models with the least-square loss and classiﬁcation models with the cross-entropy loss, the norm square of the loss gradients has a very particular form (Eqn. 17) that we need to develop the modiﬁed loss in Eqn. 6 in

Original: Step 2 of Section 5.

Original: The detail of this development is given in Section A.2.

Original: In Section A.3, we give the proof of Thm.

Original: 5.1 also needed in Step 2 of Section 5, which bounds the Frobenius norm of the network Jacobian w.r.t.

Original: the input with the network Jacobian w.r.t.

Original: the parameters.

Original: A.1 Gradient structure in the exponential family

Original: In order to be able to treat both regression and classiﬁcation on the same footing, we need to introduce the notion of a transformed target y =

Original: φ ( z ) .

Original: In regression φ is typically the identity, while for classiﬁcation φ maps the labels z ∈ { 1 , . . .

Original: , k } onto their one-hot-encoded version φ ( z ) ∈ R k .

Original: We will also need to distinguish between the logit neural network f θ : R d → R k from the response function g θ ( x ) =

Original: a ( f θ ( x )) that models the transformed target.

Original: Typically, a is the activation function of the last layer.

Original: For instance, for regression both the neural network and its response function coincide, a being the identity, while for classiﬁcation f θ ( x ) are the logits and a is typically a sigmoid or a softmax function.

Original: Now both regression and classiﬁcation losses at a data point ( x, y ) can be obtained as the negative log-likelihood of a conditional probability model

Original: [Equation]

Original: The conditional probability model for both regression and classiﬁcation has the same structure.

Original: It is obtained by using the neural network to estimate the natural parameter vector η = f θ ( x ) of an exponential family distribution:

Original: [Equation]

Original: where S ( η ) is the log-partition function.

Original: For models in the exponential family, the distribution mean coincides with the gradient of the log-partition function: E ( Y | η ) = ∇ η S ( η ) .

Original: The response function (which is the mean of the conditional distribution) is of the form g θ ( x ) = ∇ η S ( f θ ( x )) , and the last layer is thus given by a ( η ) = ∇ η S ( η ) .

Original: For these models, the log-likelihood loss at a data point has the simple form

Original: [Equation]

Original: We then obtain immediately that the loss derivative w.r.t.

Original: to the parameters and w.r.t.

Original:  to the input can be written as a sum of the the corresponding network derivatives weighted by the signed residuals: [Equation] where the (cid:15) i ’s are the signed residual, that is, the i th components of the signed error between the response function and the transformed target: [Equation]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: This means that the square norm of these gradients can be written as

Original: [Equation]

Original: where A θ and A x are the gradient alignment terms:

Original: [Equation]

Original: A.1.1 Multi-class classiﬁcation

Original: Consider the multinoulli distribution where a random variable Z can take values in k classes, say z ∈ { 1 , . .

Original: . , k } with probabilities p 1 , . .

Original: . , p k for each class respectively.

Original: Let the transformed target map y = φ ( z ) associate to a class i its one-hot-encoded vector y with the i th component equal toand all other components zero.

Original: The multinoulli density can then be written as P ( y ) =

Original: p y 1 1 · · · p y k k which can be re-parameterized, showing that the multinoulli is a member of the exponential family distribution:

Original: [Equation]

Original: We can now express the natural parameter vector η in terms of the class probabilities:

Original: [Equation]

Original: Note, η k = 0 .

Original: Taking the exponential of that last equation, and summing up, we obtain that 1 /p k = (cid:80) ki

Original: =1 e η i , which we use to express the class probabilities in terms of the canonical parameters:

Original: [Equation]

Original: Since 1 /p k = (cid:80) ki =1 e η i , we obtain that the log-partition function is

Original: [Equation]

Original: whose derivative is the softmax function:

Original: [Equation] Now, if we estimate the natural parameter η with a neural network η = f θ ( x ) , we obtain the response [Equation] A.2 Computing the expanded modiﬁed loss from Section
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: In this section we verify that expanding the modiﬁed loss in Step 1.

Original: of Section 5; i.e., (cid:101) L B =

Original: L B + h 4 (cid:107)∇ L B (cid:107) 2 , yields Eqn.(6).

Original: Namely, we show that

Original: [Equation]

Original: [Equation]

Original: where

Original: A

Original: [Equation]

Original: [Equation]

Original: It sufﬁces to show

Original: [Equation]

Original: Indeed this follows directly by computation.

Original: Firstly, note that

Original: [Equation]

Original: Now, by (17) and (19)

Original: [Equation]

Original: the last equality can be simpliﬁed so that

Original: [Equation]

Original: This completes the computation.

Original: A.3 The Transfer Theorem

Original: To frame the statement and proof of the Transfer Theorem, let’s begin by setting up and deﬁning some notation.

Original: Consider a deep neural network f θ : R d → R k parameterized by θ and consisting of l layers stacked consecutively.

Original: We can express f θ as

Original: [Equation]

Original: where f i ( z ) =

Original: a i ( w i z + b i ) denotes the i -th layer of the network deﬁned by the weight matrix w i , the bias b i and the layer activation function a i which acts on the output z of the previous layer.

Original: In this way, we can write θ = ( w 1 , b 1 , w 2 , b 2 , . . .

Original: , w l , b l ) to represent all the learnable parameters of the network.

Original: Next, let h i ( x ) denote the sub-network from the input x ∈ R d up to and including the output of layer i ; that is,

Original: [Equation]

Original: (41) where we understand h 0 ( x ) to be x .

Original: Note that, by this notation, h l ( x ) represents the full network; i.e., h l ( x ) =

Original: f θ ( x ) .

Original: At times it will be convenient to consider f θ ( x ) as dependent only on a particular layer’s parameters, for example w i and b i , and independent of all other parameter values in θ .

Original: In this case, we will use the notation f w i or f b i to represent f θ as dependent only on w i or b i (resp.).

Original: Using the notation above, for each weight matrix w 1 , w 2 , . .

Original: . , w l , we can rewrite f w i ( x ) as (similarly, for f b i ( x ) )

Original: [Equation]

Original: where each g i ( z ) denotes the remainder of the full network function f θ ( x ) following the i -th layer.

Original: That is to say, g i represents the part of the network deeper than the i -th layer (i.e., from layer i to the output f θ ( x ) ) and h i represents the part of the network shallower than the i -th layer (i.e., from the input x up to layer i ).

Original: We are now ready to state our main Theorem:

Original: Theorem A.1 (Transfer Theorem) .

Original: Consider a network f θ : R d → R k with l layers parameterized by θ =

Original: ( w 1 , b 1 , . . .

Original: , w l , b l ) , then we have the following inequality

Original: [Equation]

Original: where T i ( x, θ ) is the transfer function for layer i given by

Original: [Equation]

Original: where h i is the subnetwork to layer i and σ max ( A ) is the maximal singular value of matrix A (i.e., its spectral norm).

Original: The proof of Theorem A.1, inspired by the perturbation argument in [79], follows from the two following Lemmas.

Original: The idea is to examine the layer-wise structure of the neural network to compare the gradients ∇ w i f θ with the gradients ∇ x f θ .

Original: Due to the nature of f θ ( x ) and its dependence on the inputs x and parameters θ , we show that at each layer i , a small perturbation of the inputs x of the model function f θ ( x ) transfers to a small perturbation of the weights w i .

Original: Lemma A.2.

Original: Let f θ : R d → R k represent a deep neural network consisting of l consecutive dense layers.

Original: Using the notation above, for i = 1 , 2 , . .

Original: , l , we have

Original: [Equation]

Original: [Equation]

Original: where (cid:107) · (cid:107) 2 denotes the L 2 operator norm when applied to matrices and the L 2 norm when applied to vectors.

Original: Proof.

Original: For i = 1 , 2 , . . .

Original: , l and following the notation above, the model function as it depends on the weight matrix w i of layer i can be written as f w i ( x ) =

Original: g i ( w i

Original: h

Original: i − 1 ( x ) + b i ) .

Original: Now consider a small perturbation x + δx of the input x .

Original: There exists a corresponding perturbation w i + u ( δx ) of the weight matrix in layer i such that

Original: [Equation]

Original: To see this, note that for sufﬁciently small δx we can identify h i ( x + δx ) with its linear approximation around x so that h i ( x + δx )

Original: =

Original: h i ( x ) + ∇ x h i ( x ) δx , where ∇ x h i : R d → R |

Original: h

Original: i | is the total derivative of h

Original: i : R d → R | h i | .

Original: Thus, by representing f w i as in (42), we have

Original: [Equation]

Original: Similarly for f w i + u ( δx ) (

Original: x ) , we have

Original: [Equation]

Original: Thus Eqn.

Original: (46) is satisﬁed provided u ( δx ) h

Original: i − 1 ( x ) =

Original: w i ∇ x h

Original: i − 1 ( δx ) .

Original: Using the fact that

Original: [Equation]

Original: ( x ) T = (cid:107) h

Original: i

Original: [Equation]

Original: x )

Original: [Equation]

Original: 22 and rearranging terms, we get

Original: [Equation]

Original: Deﬁning u ( δx ) as in Eqn.

Original: and taking the derivative of both sides of Eqn.

Original: (46) with respect to δx at δx = 0 via the chain rule, we get (since by Eqn.

Original: (47) we have u ( δx ) is linear in δx )

Original: [Equation]

Original: Finally, taking the square of the L 2 operator norm (cid:107) · (cid:107) 2 on both sides, we have

Original: [Equation]

Original: Rearranging the terms and since ∇ x f w i = ∇ x f θ we get (45) which completes the proof.

Original: Following the same argument, we can also prove the corresponding lemma with respect to the derivative of the biases b i at each layer:

Original: Lemma A.3.

Original: Let f θ : R d → R k represent a deep neural network consisting of l consecutive dense layers.

Original: Using the notation above, for i = 1 , 2 , . .

Original: , l , we have

Original: [Equation]

Original: where (cid:107) · (cid:107) 2 denotes the L 2 operator norm when applied to matrices and the L 2 norm when applied to vectors.

Original: Proof.

Original: The proof is similar in spirit to the proof of Lemma A.2.

Original: Namely, for small perturbations x + δx of the input x , we verify that we can ﬁnd a corresponding small perturbation

Original: b i + u ( δx ) of the bias b i so that f b i ( x + δx ) =

Original: f

Original: b i + u ( δx ) ( x ) .

Original: From Eqn.

Original: (42), this time in relation to the bias term, we can write f b i ( x ) =

Original: g i ( w i

Original: h

Original: i − 1 ( x ) + b i ) .

Original: Then, taking a small perturbation x + δx of the input x and simplifying as before, we get

Original: [Equation]

Original: As before, taking this as our deﬁnition of u ( δx ) , and taking the derivative w.r.t.

Original: δ ( x ) as before via the chain rule, we obtain

Original: [Equation]

Original: [Equation]

Original: Again, after taking the square of the L 2 operator norm on both sides, and moving the terms around we get

Original: [Equation]

Original: It remains to prove Theorem A.1:

Original: Proof of Theorem A.1.

Original: Recall, that f θ ( x ) represents a deep neural network consisting of l layers and parameterized by θ = ( w 1 , b 1 , w 2 , b 2 , . . .

Original: , w l , b l ) .

Original: Furthermore, we use the notation f w i (resp.

Original: f b i ) to denote f θ as dependent only on w i (resp.

Original: b i ); i.e., all other parameter values are considered constant.

Original: Let (cid:107) · (cid:107) F denote the Frobenius norm.

Original: For this model structure, by Pythagoras’s theorem, it follows that

Original: [Equation]
Revised: [N/A]

Original: The remainder of the proof relies on the following general property of matrix norms:

Original: Given a matrix

Original: A ∈ R m × n , let σ max ( A ) represent the largest singular value of A .

Original: The Frobenius norm (cid:107) · (cid:107) F and the L 2 operator norm (cid:107) · (cid:107) 2 are related by the following inequalities

Original: [Equation]

Original: where σ i ( A ) are the singular values of the matrix A .

Original: Considering ∇ x f θ as a map from R d to R k , by (54) it follows that

Original: [Equation]

Original: By Lemma A.2 and Lemma A.

Original: [Equation]

Original: we have that for i = 1 , 2 , . . .

Original: , l

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: we can re-write Eqn.

Original: (53) using (54), (55) and (56) to get

Original: [Equation]

Original: For i = 1 , 2 , . . .

Original: , l , deﬁne

Original: [Equation]

Original: We call T i ( x, θ ) the transfer function for layer i in the network.

Original: Then, rearranging terms in the ﬁnal inequality above, we get

Original: [Equation]

Original: [Equation]

Original: This completes the proof.

Original: B Experiment details B.1 Figure 1: Motivating 1D example

Original: We trained a ReLU MLP with 3 layers of 300 neurons each to regress 10 data points on an exact parabola ( x, x 2 ) ∈ R 2 where x ranges over 10 equidistant points in the interval [ − 1 , 1] .

Original: We use full batch gradient descent with learning rate 0 .

Original: 02 for 100000 steps/epochs.

Original: We plotted the model function across the [ − 1 , 1] range and computed its geometric complexity over the dataset at step(initialization), step 10, step 1000, and step 10000 (close to interpolation).

Original: The network was randomly initialized using the standard initialization (i.e. the weights were sampled from a truncated normal distribution with variance inversely proportional to the number of input units and the bias terms were set to zero).

Original: This model was trained ﬁve separate times using ﬁve different random seeds.

Original: Each line marks the mean of the ﬁve runs and the shaded region is 95% conﬁdence interval over these ﬁve seeds.

Original: B.

Original: Left and Middle: We initialized several ReLU MLP’s f

Original: θ 0 : R d → R k with large input and output: d = 224 × 224 × 3 and k = 1000 , with 500 neurons per layer, and with a varying number of layers l ∈ [1 , 2 , 4 , 8 , 16 , 32 , 64] .

Original: We used the standard initialization scheme: we sample them from a truncated normal distribution with variance inversely proportional to the number of input units and the bias terms were set to zero.

Original: We measured and plotted the following quantities

Original: [Equation]

Original: with x ranging over 50 equidistant points in the interval [0 , 1] .

Original: The points P 1 and P 2 where chosen to be the two diagonal points ( − 1 , . . .

Original: , − 1) and (1 , . .

Original: . , 1) , respectively, of the normalized data hyper-cube [ − 1 , 1] d .

Original: Each line marks the mean of the 5 runs and the shaded region is 95% conﬁdence interval over these ﬁve seeds.

Original: What we observe with the standard initialization scheme also persists with the Glorot initialization scheme, where the biases are set to zero, and the weight matrices parameters are sampled from the uniform distribution on [ − 1 , 1] and scaled at each layer by (cid:112) 6 /d l + d l − 1 , where d l is the number of units in layer l .

Original: We report this in Fig.

Original: 6 below.

Original: [Figure]

Original: Glorot initialization scheme is the one described in the paragraph above.

Original: For the standard initialization we set the biases to zero, and initialized the weight matrices parameters from a normal distribution truncated to the range [ − 2 , 2] and rescaled using 1 / (cid:112) d l − 1 at each layer.

Original: We plotted the mean geometric complexity with error bars representing the 95% conﬁdence interval over therandom seeds.

Original: The error bars are tiny in comparison of the plotted quantities and are therefore not visible on the plot.

Original: Left: We trained a ResNet18 on CIFAR10 three times with different random seeds with a learning rate of 0.02, batch size of 512, for 10000 steps for each combination of regularization rate and regularization type in { L2 , Spectral , Flatness } × [0 , 0 . 01 , 0 .

Original: We measured the geometric complexity at time of maximum test accuracy for each of these runs and plotted the mean with 95% conﬁdence interval over the 3 random seeds.

Original: For the L2 regularization we added the sum of the parameter squares to the loss multiplied with the regularization rate.

Original: For the spectral regularization, we followed the procedure described in [90] by adding to the loss the penalty ( α/ 2) (cid:80) i σ max ( W i )where W i is either the layer weight matrix for a dense layer or, for a convolution layer, the matrix of shape b × ak w k h obtained by reshaping the convolution layer with a input channels, b output channels, and a kernel of size k w × k h .

Original: For the ﬂatness regularization, we added to the batch loss L B at each step the norm square of the batch loss gradient (cid:107)∇ θ L B ( θ ) (cid:107) 2 multiplied by the regularization rate.

Original: Middle: We trained a ResNet18 on CIFAR10 with learning rate 0.005 and batch size 16 for 100000 steps with varying proportion of label noise.

Original: For each label noise proportion α ∈ [0 , 0 . 05 , 0 . 1 , 0 . 15 , 0 . 25] , we trained three times with different random seeds.

Original: We plotted the mean geometric complexity at time of maximum test accuracy as well as the 95% conﬁdence interval over the three runs.

Original: The label noise was created by mislabelling α % of the true labels before training.

Original: B.4 Figure

Original: complexity

Original: Top row (varying learning rates):

Original: We trained a ResNet18 on CIFAR10 with batch size 512, for 30000 steps with varying learning rates.

Original: For each learning rate in [0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2] , we trained the neural network three times with a different random seed.

Original: We plotted the learning curves for the test accuracy, the geometric complexity, and the training loss, where the solid lines represent the mean of these quantities over the three runs and the shaded area represents the 95% conﬁdence interval over the three runs.

Original: We applied a smoothing over 50 steps for each run, before computing the mean and the conﬁdence interval of the three runs.

Original: Bottom row (varying batch sizes):

Original: We trained a ResNet18 on CIFAR10 with learning rate 0.2, for 100000 steps with varying batch sizes.

Original: For each batch size in [8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024] , we trained the neural network three times with a different random seed.

Original: We plotted the learning curves for the test accuracy, the geometric complexity, and the training loss, where the solid lines represent the mean of these quantities over the three runs and the shaded area represents the 95% conﬁdence interval over the three runs.

Original: We applied a smoothing over 50 steps for each run, before computing the mean and the conﬁdence interval of the three run.

Original: We train a ResNet18 on CIFAR10 with learning rate 0.8, batch size 124, for 100000 steps, with 18 different network widths.

Original: For each network width in [1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 20 , 30 , 64 , 100 , 128] we train three times with a different random seed.

Original: Deﬁnition of width: We follow the description of ResNet width discussed in [71].

Original: The ResNet architecture we follow is that of [38].

Original: The original ResNet18 architecture has four successive ResNet blocks each formed by two identical stacked sequences of a BatchNorm and a convolution layer.

Original: The number of ﬁlters for the convolution layers in each of the successive ResNet block is ( k, 2 k, 4 k, 8 k ) with k = 64 for the original RestNet18.

Original: Following [71] we take k to be the width of the network, which we vary during our experiments.

Original: C Additional Experiments C.1 Geometric complexity at initialization decreases with added layers on large domains

Original: We reproduce the experiments from Fig.

Original: 2 on a larger domain with the same conclusion: For both the standard and the Glorot initialization schemes ReLU networks initialize closer to the zero function and with lower geometric complexity as the number of layer increases.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: C.2 Geometric complexity decreases with implicit regularization for MNIST

Original: We replicate the implicit regularization experiments we conducted for CIFAR10 with ResNet18 in

Original: Fig.

Original: 4 for MLP’s trained on MNIST.

Original: The conclusion is the same: higher learning rates and smaller batch sizes decrease geometric complexity through implicit gradient regularization

Original: [Figure]

Original: and are correlated with higher test accuracy.

Original: [Figure]

Original: varying batch sizes.

Original: For each learning rate in [0 . 01 , 0 . 025 , 0 . 05 , 0 . 075 , 0 . 1] , we trained 5 different times with a different random seed.

Original: The MLP were initialized using the standard initialization scheme.

Original: [Figure]

Original: For each batch size in [8 , 16 , 32 , 64 ,

Original: , we trained 5 different times with a different random

Original: C.3 Geometric complexity decreases with explicit regularization for MNIST

Original: We replicate the explicit regularization experiments we conducted for CIFAR10 with ResNet18 in

Original: Fig.

Original: 3 for MLP’s trained on MNIST with similar conclusions: higher regularization rates for L2, spectral, and ﬂatness regularization decrease geometric complexity and are correlated with higher test accuracy.

Original: 005 , 0 . 0075] , we trained 5 different times with a different random seed.

Original: The

Original: MLP were initialized using the standard initialization scheme.

Original: C.4 Explicit geometric complexity regularization for MNIST and CIFAR

Original: In this section

Original: [Figure]

Original: we explicitly regularize for the geometric complexity.

Original: This is a known form of regularization also called Jacobian regularization [42, 84, 89, 90].

Original: We ﬁrst perform this regularization for a MLP trained on MNIST (Fig. 18) and then for a ResNet18 trained on CIFAR10 (Fig. 19) with the following conclusion: test accuracy increases with higher regularization strength while the geometric complexity decreases.

Original: [Figure]

Original: [Figure]

Original: C.5 Separate L2, ﬂatness, and spectral regularization experiments for CIFAR

Original: For the sake of space in Fig.

Original: 3 (right) in the main paper, we used the same regularization rate range for all types of explicit regularization we tried.

Original: In this section, we perform the experiments on a targeted range for each regularization type, leading to clearer plots (Fig. 20, Fig. 21, and Fig. 22)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: C.6 Geometric complexity in the presence of multiple tuning mechanisms

Original: For most of this paper, we studied the impact of tuning strategies, like the choice of initialization, hyper-parameters, or explicit regularization in isolation from other very common heuristics like learning rate schedules and data-augmentation.

Original: In this section, we reproduce the implicit regularization effect of the batch size and the learning rate on GC (c.f. Fig.

Original: 4 in the main paper) while using these standard tricks to achieve better performance.

Original: The resulting models achieve performance closer to

Original: SOTA for the ResNet18 architecture.

Original: Although the learning curves are messier and harder to interpret (Fig. 23) because of the multiple mechanisms interacting in complex ways, we still observe that the general effect of the learning rate (Fig. 24) and the batch size (Fig. 25) on geometric complexity is preserved in this context.

Original: More importantly, we also note that the sweeps with higher test accuracy solutions tend also to come with a lower geometric complexity, even in this more complex setting (Fig.

Original: 24 right and Fig.

Original: 25 right).

Original: Namely, models with higher test accuracy have correspondingly lower GC.

Original: Speciﬁcally in terms of implicit regularization, as the learning rate increases, the geometric complexity decreases and the maximum test accuracy increases.

Original: Similarly, smaller batch size leads to lower geometric complexity as well as higher test accuracy.

Original: [Figure]

Original: [Figure]

Original: t r i c

Original: C o m

Original: [Figure]

Original: l e x i t

Original: [Figure]

Original: [Figure]

Original: C.7 Geometric complexity in the presence of momentum

Original: We replicate the implicit and explicit regularization experiments using SGD with momentum , which is widely used in practise.

Original: The conclusion remains the same as for vanilla SGD: More regularization (implicit through batch size or learning rate or explicit through ﬂatness, spectral, and L2 penalties) produces solutions with higher test accuracy and lower geometric complexity.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: trained with momentum :

Original: We trained a selection of ResNet18 with learning rate of 0.0005 with batch size of 512 for 10000 steps using SGD with a momentum of 0.9.

Original: For each regularization rate in α ∈ [0 , 0 . 01 , 0 . 025 , 0 . 05 , 0 .

Original: 1] , we trained 3 different times with a different random seed

Original: [Figure]

Original: [Figure]

Original: with learning rate of 0.0005 with batch size of 512 for 10000 steps using SGD with a momentum of 0.9.

Original: We regularized the loss by adding to it the spectral norm penalty α/ 2 (cid:80) i σ max ( W i ) 2 where W i are the layer weight matrices as described in [67].

Original: For each regularization rate in α ∈ [0 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 075 , 0 . 1] , we traineddifferent times with a different random seed.

Original: C.8 Geometric complexity in the presence of Adam

Original: We replicate the implicit and explicit regularization experiment using Adam, which is widely used in practice.

Original: In this case the conclusions are less clear than with vanilla SGD or SGD with momentum.

Original: While higher learning rates, and higher explicit ﬂatness, L2, and spectral regularization still put a regularizing pressure on the geometric complexity for most of the training, the effect of batch size on geometric complexity is not clear.

Original: This may be that the local built-in re-scaling of the gradient sizes in Adam affects the pressure on the geometric complexity in complex ways when the batch size changes.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: trained with Adam : We trained a selection of ResNet18 with learning rate of 0.0001 with batch size of 512 for 10000 steps using using Adam with b 1 = 0 .

Original: 9 , b 2 = 0 .

Original: For each regularization rate in α ∈ [0 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 075 , 0 . 1] , we trained 3 different times with a different random seed.

Original: [Figure]

Original: [Figure]

Original: trained with Adam : We trained a selection of ResNet18 with learning rate of 0.0001 with batch size of 512 for 10000 steps using Adam with b 1 = 0 .

Original: 9 , b 2 = 0 .

Original: We regularized the loss by adding to it the spectral norm penalty α/ 2 (cid:80) i σ max ( W i ) 2 where W i are the layer weight matrices as described in [67].

Original: For each regularization rate in α ∈ [0 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 075 , 0 . 1] , we trained 3 different times with a different random seed.

Original: D Comparison of the Geometric Complexity to other complexity measures

Original: One of the primary challenges in deep learning is to better understand mechanisms or techniques that correlate well with (or can imply a bound on) the generalization error for large classes of models.

Original: The standard approach of splitting the data into a train, validation and test set has become the de facto way to achieve such a bound.

Original: With this goal in mind, a number of complexity measures have been proposed in the literature with varying degrees of theoretical justiﬁcation and/or empirical success.

Original: In this section we compare our geometric complexity measure with other, more familiar complexity measures such as the Rademacher complexity, VC dimension and sharpness-based measures.

Original: D.1 Rademacher Complexity

Original: Perhaps the most historically popular and widely known complexity measure is the Rademacher complexity [7, 9, 51, 52].

Original: Loosely speaking the Rademacher complexity measures the degree to which a class of functions H can ﬁt random noise.

Original: The idea behind this complexity measure is that a more complex function space is able to generate more complex representation vectors and thus, on average, produce learned functions that are better able to correlate with random noise than a less complex function space.

Original: To make this deﬁnition more precise and frame it in the context of machine learning (see also [68]), given an input feature space X and a target space Y , let G denote a family of loss functions

Original: L : Z = X

Original: × Y → R associated with a function class H .

Original: Notationally,

Original: [Equation]

Original: We deﬁne the empirical Rademacher complexity as follows:

Original: Deﬁnition D.1 (Empirical Rademacher Complexity) .

Original: With G as above, let S = { z 1 , z 2 , . .

Original: , z m } be a ﬁxed sample of size m of elements of Z = X

Original: × Y .

Original: The empirical Rademacher complexity of G with respect to the sample S is deﬁned as:

Original: [Equation]

Original: where σ = ( σ 1 , σ 2 , . .

Original: , σ m ) (cid:124) , with the σ i ’s being independent uniform random variables which take values in {− 1 , +1 } .

Original: These random variables σ i are called Rademacher variables .

Original: If we let g S denote the vector of values taken by function g over the sample S , then the Rademacher complexity, in essence, measures the expected value of the supremum of the correlation of g S with a vector of random noise σ ; i.e., the empirical Rademacher complexity measures on average how well the function class G correlates with random noise on the set S .

Original: More complex families G can generate more vectors g S and thus better correlate with random noise on average, see [68] for more details.

Original: Note that the empirical Rademacher complexity depends on the sample S .

Original: The Rademacher complexity is then an average of this empirical measure over the distribution from which all samples are drawn:

Original: [-Deﬁnition D.
-]
Revised: [-Deﬁnition D.
-]

Original: Rademacher Complexity) .

Original: Let D denote the distribution from which all samples

Original: S are drawn.

Original: For any integer m ≥ 1 , the Rademacher complexity of G is the expectation of the empirical Rademacher complexity over all samples of size m drawn according to D :

Original: [Equation]

Original: The Rademacher complexity is distribution dependent and deﬁned for any class of real-valued functions.

Original: However, computing it can be intractable for modern day machine learning models.

Original: Similar to the empirical Rademacher complexity, the geometric complexity is also computed over a sample of points, in this case the training dataset, and is well-deﬁned for any class of differentiable functions.

Original: In contrast, the Rademacher complexity (and the VC dimension which we discuss below) measures the complexity for an entire hypothesis space, while the geometric complexity focuses only on single functions.

Original: Furthermore, since the Geometric Complexity relies only on ﬁrst derivatives of the learned model function making it much easier to compute.

Original: D.2 VC dimension

Original: The Vapnik–Chervonenkis (VC) dimension [14, 18, 88] is another common approach to measuring the complexity of a class of functions H and is often easier to compute than the Rademacher Complexity, see [68] for further discussion on explicit bounds which compare the Rademacher complexity with the VC dimension.

Original: The VC dimension is a purely combinatorial notion and deﬁned using the concept of a shattering of a set of points.

Original: A set of points is said to be shattered by H if, no matter how we assign a binary label to each point, there exists a member of H that can perfectly separate the points; i.e., the growth function for H is 2 m .

Original: The VC dimension of a class H is the size of the largest set that can be shattered by H .

Original: More formally, we have

Original: Deﬁnition D.3 (VC dimension) .

Original: Let H denote a class of functions on X taking values in {− 1 , +1 } .

Original: Deﬁne the growth function Π H : N → N as

Original: [Equation]

Original: If Π H = 2 m we say H shatters the set { x 1 , . . .

Original: , x m } .

Original: The VC dimension of H is the size of the largest shattered set, i.e.

Original: [Equation]

Original: If there is no largest m , we deﬁne VCdim ( H ) = ∞ .

Original: The VC dimension is appealing partly because it can be upper bounded for many classes of functions (see for example, [8]).

Original: Similar to the Rademacher complexity, the VC dimension is measured on the entire hypothesis space.

Original: The Geometric Complexity, in contrast, is instead measured for given function within the hypothesis space allowing for more direct comparison between elements within the class H .

Original: Computing the VC dimension for a given function set H may not be always convenient since, by deﬁnition, it requires computing the growth function Π H ( m ) for all subsets of order m ≥ 1 ; whereas, the Geometric Complexity relies only on ﬁrst derivatives and is much easier to precisely compute.

Original: D.3 Sharpness and Hessian related measures

Original: Another broad category of generalization measures concerns the concept of “sharpness” of the local minima; for example, see [40, 47, 64].

Original: Such complexity measures aim to quantify the sensitivity of the loss to perturbations in model parameters.

Original: Here a ﬂat minimizer is a point in parameter space where the loss varies only slightly in a relatively large neighborhood of the point.

Original: Conversely, the variation of the loss function is less controlled in a neighborhood around a sharp minimizer.

Original: The idea is that for sharp minimizers the training function is more sensitive to perturbations in the model parameters and thus negatively impacts the model’s ability to generalize; see also [1] which argues that ﬂat solutions have low information content.

Original: The sharpness of a minimizer can be characterized by the magnitude of the eigenvalues of the Hessian of the loss function.

Original: However, since the Hessian requires two derivatives this can be computationally costly in most deep learning use cases.

Original: To overcome this drawback, [47] suggest a metric that explores the change in values of the loss function f

Original: within small neighborhoods of points.

Original: More precisely, let C (cid:15) denote a box around an optimal point in the domain of f and let A ∈ R n × p be a matrix whose columns are randomly generated.

Original: The constraint C (cid:15) is then deﬁned as:

Original: [Equation]

Original: where A + denotes the pseudo-inverse of A and (cid:15) controls the size of the box.

Original: Keskar et al.

Original: [47] then deﬁne “sharpness” by

Original: Deﬁnition D.4 (Sharpness) .

Original: Given x ∈ R n , (cid:15) > 0 and A ∈ R n × p , the ( C (cid:15) ) -sharpness of f at x is deﬁned as

Original: [Equation]

Original: Another related complexity measure is the effective dimension which is computed using the spectral decomposition of the loss Hessian [63].

Original: Since the effective dimension relies on the Hessian, it causes ﬂat regions in the loss surface to also be regions of low complexity w.r.t.

Original: this measure.

Original: Effective dimensionality [62] was originally proposed to measure the dimensionality of the parameter space determined by the data and is computed using the eigenspectrum of the Hessian of the training loss.

Original: Deﬁnition D.5 (Effective dimensionality of a symmetric matrix) .

Original: The effective dimensionality of a symmeteric matrix A ∈ R k × k is deﬁned as

Original: [Equation]

Original: where λ i are the eigenvalues of A and z > 0 is a regularization constant.

Original: When used in the context of measuring the effective dimension of a neural network f ( x ; θ ) with inputs x and parameters θ ∈ R k , we take A to be the Hessian of the loss function; i.e., the k × k matrix of second derivatives of the loss L over the data distribution D deﬁned as Hess θ =

Original: −∇ 2 L ( θ, D ) .

Original: Furthermore, the computation of the effective dimension involves both double derivatives (to compute the Hessian) but also evaluation of the eigenvalues of the resulting matrix.

Original: This can introduce a prohibitive cost in computation for many deep learning models.

Original: The Geometric Complexity differs from these complexity measures in a meaningful way.

Original: Sharpness and effective dimension are ultimately concerned with the behavior and derivatives of the loss function with respect to the parameter space.

Original: The Geometric Complexity, however, is measured using derivatives of the learned model function with respect to the model inputs.

Original: Furthermore, the

Original: Geometric Complexity is computed using only a single derivative, making it computationally tractable to measure and track.

Original: That being said, these sharpness measures and the Geometric Complexity are also quite related.

Original: For example, as explained by the Transfer Theorem in Section 5, for neural networks these ﬂat regions are also the regions of low loss gradient and thus of low GC.

Original: Furthermore, similar to the GC, the effective dimension can also capture the double descent phenomena and, in [63], the authors argue the effective dimension provides an efﬁcient mechanism for model selection.