Original: Why neural networks ﬁnd simple solutions: the many regularizers of geometric complexity
Revised: Why neural networks ﬁnd simple solutions: the many regularizers of geometric complexity

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In many contexts, simpler models are preferable to more complex models andthe control of this model complexity is the goal for many methods in machinelearning such as regularization, hyperparameter tuning and architecture design.
Revised: In many contexts, simpler models are preferable to more complex models andthe control of this model complexity is the goal for many methods in machinelearning such as regularization, hyperparameter tuning and architecture design.

Original: Indeep learning, it has been difﬁcult to understand the underlying mechanisms ofcomplexity control, since many traditional measures are not naturally suitable fordeep neural networks.
Revised: Indeep learning, it has been difﬁcult to understand the underlying mechanisms ofcomplexity control, since many traditional measures are not naturally suitable fordeep neural networks.

Original: Here we develop the notion of geometric complexity, whichis a measure of the variability of the model function, computed using a discrete
Revised: Here we develop the notion of geometric complexity, whichis a measure of the variability of the model function, computed using a discrete

Original: Dirichlet energy.
Revised: Dirichlet energy.

Original: Using a combination of theoretical arguments and empiricalresults, we show that many common training heuristics such as parameter normregularization, spectral norm regularization, ﬂatness regularization, implicit gradi- ent regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models
Revised: Using a combination of theoretical arguments and empiricalresults, we show that many common training heuristics such as parameter normregularization, spectral norm regularization, ﬂatness regularization, implicit gradi- ent regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models

Original:  Introduction
Revised:  Introduction

Original: Regularization is an essential ingredient in the deep learning recipe and understanding its impact onthe properties of the learned solution is a very active area of research [28, 34, 58, 68].
Revised: Regularization is an essential ingredient in the deep learning recipe and understanding its impact onthe properties of the learned solution is a very active area of research [28, 36, 62, 73].

Original: Regularizationcan assume a multitude of forms, either added explicitly as a penalty term in a loss function [28] orimplicitly through our choice of hyperparameters [5, 61, 64, 66, 67], model architecture [46, 48, 49]or initialization [26,30, 31, 47, 55, 76, 77].
Revised: Regularizationcan assume a multitude of forms, either added explicitly as a penalty term in a loss function [28] orimplicitly through our choice of hyperparameters [6, 65, 68, 70, 72], model architecture [49, 51, 52]or initialization [26, 31, 32, 50, 59, 81, 82].

Original: These forms are generally not designed to be analyticallytractable, but in practice, regularization is often invoked in the control of model complexity, putting apressure on a model to discover simple solutions more so than complex solutions.
Revised: These forms are generally not designed to be analyticallytractable, but in practice, regularization is often invoked in the control of model complexity, putting apressure on a model to discover simple solutions more so than complex solutions.

Original: To understand regularization in deep learning, we need to precisely deﬁne model ‘complexity’ fordeep neural networks.
Revised: To understand regularization in deep learning, we need to precisely deﬁne model ‘complexity’ fordeep neural networks.

Original: Complexity theory provides many techniques for measuring the complexityof a model, such as a simple parameter count, or a parameter norm measurement [3, 18, 51, 57] butmany of these measures can be problematic for neural networks [36, 74].
Revised: Complexity theory provides many techniques for measuring the complexityof a model, such as a simple parameter count, or a parameter norm measurement [4, 18, 54, 61] butmany of these measures can be problematic for neural networks [38, 79].

Original: The recently observedphenomena of ‘double-descent’ [7, 8, 56] illustrates this clearly: neural networks with high modelcomplexity, as measured by a parameter count, can ﬁt training data closely (sometimes interpolatingthe data exactly), while simultaneously having low test error [56, 74].
Revised: The recently observedphenomena of ‘double-descent’ [8, 9, 60] illustrates this clearly: neural networks with high modelcomplexity, as measured by a parameter count, can ﬁt training data closely (sometimes interpolatingthe data exactly), while simultaneously having low test error [60, 79].

Original: Classically, we expect thatinterpolation of training data is evidence of overﬁtting, but yet, neural networks seem to be capableof interpolation while also having low test error.
Revised: Classically, we expect thatinterpolation of training data is evidence of overﬁtting, but yet, neural networks seem to be capableof interpolation while also having low test error.

Original: It is often suggested that some form of implicitregularization or explicit regularization is responsible for this, but how should we account for this intheory, and what complexity measure is most appropriate?
Revised: It is often suggested that some form of implicitregularization or explicit regularization is responsible for this, but how should we account for this intheory, and what complexity measure is most appropriate?

Original: In this work, we develop a measure of model complexity, called Geometric Complexity (GC), that hasproperties that are suitable for the analysis of deep neural networks.
Revised: In this work, we develop a measure of model complexity, called Geometric Complexity (GC), that hasproperties that are suitable for the analysis of deep neural networks.

Original: We use theoretical and empiricaltechniques to demonstrate that many different forms of regularization and other training heuristics can act to control geometric complexity through different mechanisms.
Revised: We use theoretical and empiricaltechniques to demonstrate that many different forms of regularization and other training heuristics can act to control geometric complexity through different mechanisms.

Original: We argue that the geometriccomplexity provides a convenient proxy for neural network performance.
Revised: We argue that the geometriccomplexity provides a convenient proxy for neural network performance.

Original: Our primary contributions are:
Revised: Our primary contributions are:

Original: • We develop a computationally tractable notion of complexity (Section 2), which we callGeometric Complexity (GC), that has many close relationships with many areas in deeplearning and mathematics including harmonic function theory, Lipschitz smoothness (Section2), and regularization theory (Section 4).
Revised: • We develop a computationally tractable notion of complexity (Section 2), which we callGeometric Complexity (GC), that has many close relationships with many areas in deeplearning and mathematics including harmonic function theory, Lipschitz smoothness (Section2), and regularization theory (Section 4).

Original: • We provide evidence that common training heuristics keep the geometric complexity low,including: (i) common initialization schemes (Section 3) (ii) the use of overparametrizedmodels with a large number of layers (Fig. 2) (iii) large learning rates, small batch sizes, andimplicit gradient regularization (Section 5 and Fig. 4) (iv) explicit parameter norm regularization, spectral norm regularization, ﬂatness regularization, and label noise regularization
Revised: • We provide evidence that common training heuristics keep the geometric complexity low,including: (i) common initialization schemes (Section 3) (ii) the use of overparametrizedmodels with a large number of layers (Fig. 2) (iii) large learning rates, small batch sizes, andimplicit gradient regularization (Section 5 and Fig. 4) (iv) explicit parameter norm regular- ization, spectral norm regularization, ﬂatness regularization, and label noise regularization

Original: (Section 4 and Fig. 3)
Revised: (Section 4 and Fig. 3)

Original: • We show that the geometric complexity captures the double-descent behaviour observed inthe test loss as model parameter count increases (Section 6 and Fig. 5).
Revised: • We show that the geometric complexity captures the double-descent behaviour observed inthe test loss as model parameter count increases (Section 6 and Fig. 5).

Original: The aim of this paper is to introduce geometric complexity, explore its properties and highlight itsconnections with existing implicit and explicit regularizers.
Revised: The aim of this paper is to introduce geometric complexity, explore its properties and highlightits connections with existing implicit and explicit regularizers.

Original: To disentangle the effects studied herefrom optimisation choices, we use stochastic gradient descent without momentum to train all models.
Revised: To disentangle the effects studiedhere from optimisation choices, we use stochastic gradient descent without momentum to train allmodels.

Original: We also study the impact of a given training heuristic on geometric complexity in isolation of othertechniques to avoid masking effects.
Revised: We also study the impact of a given training heuristic on geometric complexity in isolationof other techniques to avoid masking effects.

Original: For this reason we do not use data augmentation or learning rateschedules.
Revised: For this reason we do not use data augmentation orlearning rate schedules in the main part of the paper.

Revised: In the Supplementary Material (SM) we redomost experiments using SGD with momentum (Section C.7) and Adam (Section C.8) with verysimilar conclusion.

Revised: We also observe the same behavior of the geometric complexity in a settingusing learning rate schedule, data augmentation, and explicit regularization in conjunction to improvemodel performance (Section C.6).

Original: All the exact details of the experiments  are listed in the Supplementary Materials (SM) .
Revised: The exact details of all experiments in the main paper are listed in SM Section B.

Revised: All additional experiment results and details can be found in SM Section C.

Original:  Geometric complexity and Dirichlet energy
Revised:  Geometric complexity and Dirichlet energy

Original: Although many different forms of complexity measures have been proposed and investigated (e.g.,[17, 36, 57]), it is not altogether clear what properties they should have, especially for deep learning.
Revised: Although many different forms of complexity measures have been proposed and investigated (e.g.,[17, 38, 61]), it is not altogether clear what properties they should have, especially for deep learning.

Original: For instance, a number of them like the Rademacher complexity [42], the VC dimension [71], or thesimple model parameter count focus on measuring the entire hypothesis space, rather than a speciﬁcfunction, which can be problematic in deep learning [36, 74].
Revised: For instance, a number of them like the Rademacher complexity [44], the VC dimension [76], or thesimple model parameter count focus on measuring the entire hypothesis space, rather than a speciﬁcfunction, which can be problematic in deep learning [38, 79].

Original: Other measures like the number oflinear pieces for ReLU networks [2, 65] or various versions of the weight matrix norms [57] measurethe complexity of the model function independently from the task at hand, which is not desirable[60].
Revised: Other measures like the number oflinear pieces for ReLU networks [3, 69] or various versions of the weight matrix norms [61] measurethe complexity of the model function independently from the task at hand, which is not desirable[64].

Original: An alternative approach is to learn a complexity measure directly from data [45] or to take thewhole training procedure over a dataset into account [56].
Revised: An alternative approach is to learn a complexity measure directly from data [48] or to take thewhole training procedure over a dataset into account [60].

Original: Recently, other measures focusing on themodel function complexity over a dataset have been proposed in [22] and [51] to help explain thesurprising generalization power of deep neural networks.
Revised: Recently, other measures focusing on themodel function complexity over a dataset have been proposed in [22] and [54] to help explain thesurprising generalization power of deep neural networks.

Original: Following that last approach and motivatedby frameworks well established in the ﬁeld of geometric analysis [38], we propose a deﬁnition ofcomplexity related to the theory of harmonic functions and minimal surfaces.
Revised: Following that last approach and motivatedby frameworks well established in the ﬁeld of geometric analysis [40], we propose a deﬁnition ofcomplexity related to the theory of harmonic functions and minimal surfaces.

Original: Our deﬁnition has theadvantage of being computationally tractable and implicitly regularized by many training heuristicsin the case of neural networks.
Revised: Our deﬁnition has theadvantage of being computationally tractable and implicitly regularized by many training heuristicsin the case of neural networks.

Revised: It focuses on measuring the complexity of individual functionsrather than that of the whole function space, which makes it different from the Radamacher or VCcomplexity.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Let g ✓ : R d !
Revised: Let g ✓ : R d !

Original: R k be a neural network parameterized by ✓ .
Revised: R k be a neural network parameterized by ✓ .

Original: We can write g ✓ ( x ) =a ( f ✓ ( x )) where a denotes the last layer activation, and f ✓ its logit network.
Revised: We can write g ✓ ( x ) =a ( f ✓ ( x )) where a denotes the last layer activation, and f ✓ its logit network.

Original: The GC of the networkover a dataset D is deﬁned to be the discrete Dirichlet energy of its logit network:
Revised: The GC of the networkover a dataset D is deﬁned to be the discrete Dirichlet energy of its logit network:

Original: [Equation]
Revised: [Equation]

Original: where kr x f ✓ ( x ) k F is the Frobeniusnorm of the network Jacobian.
Revised: where kr x f ✓ ( x ) k F is the Frobenius

Original: Note that this deﬁnition is well-deﬁned for any differentiable model, not only a neural network, andincorporates both the model function and the dataset over which the task is determined.
Revised: Note that this deﬁnition is well-deﬁned for any differentiable model, not only a neural network, andincorporates both the model function and the dataset over which the task is determined.

Original: Next, wediscuss how GC relates to familiar concepts in deep learning.
Revised: Next, wediscuss how GC relates to familiar concepts in deep learning.

Original: Geometric complexity and linear models: Consider a linear transformation f ( x ) =
Revised: Geometric complexity and linear models: Consider a linear transformation f ( x ) =

Original: Ax + bfrom R d to R k and a dataset D = { x i } Ni =1 where x i 2 R d .
Revised: Ax + bfrom R d to R k and a dataset D = { x i } Ni =1 where x i 2 R d .

Original: At each point x 2 D , we have thatkr x f ( x ) k 2 F =
Revised: At each point x 2 D , we have thatkr x f ( x ) k 2 F =

Revised: k A k 2 F , hence the GC for a linear transformation is h f ✓ , D i G =

Original: k A  implies that GC for linear transformations (and more generally, afﬁne maps), is independent of thedataset D and zero for constant functions.
Revised: k A k 2 F . Note, this 89 implies that GC for linear transformations (and more generally, afﬁne maps), is independent of thedataset D and zero for constant functions.

Original: Furthermore, note that the GC in this setting coincidesprecisely with the L2 norm of the model weight matrix.
Revised: Furthermore, note that the GC in this setting coincidesprecisely with the L2 norm of the model weight matrix.

Original: Thus, enforcing an L2 norm penalty isequivalent to regularizing the GC for linear models (see Section 4 for more on that point).
Revised: Thus, enforcing an L2 norm penalty isequivalent to regularizing the GC for linear models (see Section 4 for more on that point).

Original: Geometric complexity and ReLU networks: For a ReLU network g ✓ : R d !
Revised: Geometric complexity and ReLU networks: For a ReLU network g ✓ : R d !

Original: R k as deﬁned in
Revised: R k as deﬁned in

Original: Deﬁnition 2.1, the GC over a dataset D has a very intuitive form.
Revised: Deﬁnition 2.1, the GC over a dataset D has a very intuitive form.

Original: Since a ReLU network parameterizespiece-wise linear functions [2], the domain can be broken into a partition of subsets X i ⇢ R d wheref ✓ is an afﬁne map
Revised: Since a ReLU network parameterizespiece-wise linear functions [3], the domain can be broken into a partition of subsets X i ⇢ R d wheref ✓ is an afﬁne map

Original: A i x + b i .
Revised: A i x + b i .

Original: Now denote by D i the points in the dataset D that fall in the linearpiece deﬁned on X i .
Revised: Now denote by D i the points in the dataset D that fall in the linearpiece deﬁned on X i .

Original: For every point x in X i , we have that kr x f ✓ ( x ) k 2 F =
Revised: For every point x in X i , we have that kr x f ✓ ( x ) k 2 F =

Original: k
Revised: k

Original: A i k 2 F .
Revised: A i k 2 F .

Original: Since the D i ’spartition the dataset D , we obtain
Revised: Since the D i ’spartition the dataset D , we obtain

Original: [Equation]
Revised: [Equation]

Original: where n i is the number of points in the dataset D falling in X i .
Revised: where n i is the number of points in the dataset D falling in X i .

Original: We see from Eqn.
Revised: We see from Eqn.

Original: (2) that for ReLUnetworks the GC over the whole dataset coincides exactly with the GC on a batch B ⇢ D , providedthat the proportion of points in the batch falling into each of the the linear pieces are preserved.
Revised: (2) that for ReLUnetworks the GC over the whole dataset coincides exactly with the GC on a batch B ⇢ D , providedthat the proportion of points in the batch falling into each of the the linear pieces are preserved.

Original: Thismakes the evaluation of the GC on large enough batches a very good proxy to the overall GC over thedataset, and computationally tractable during training.
Revised: Thismakes the evaluation of the GC on large enough batches a very good proxy to the overall GC over thedataset, and computationally tractable during training.

Original: Geometric complexity and Lipschitz smoothness: One way to measure the smoothness of afunction f : R d !
Revised: Geometric complexity and Lipschitz smoothness: One way to measure the smoothness of afunction f : R d !

Original: R k on a subset X ⇢ R d is by its Lipschitz constant; i.e., the smallest f L  0 suchthat k f ( x 1 )  f ( x 2 )
Revised: R k on a subset X ⇢ R d is by its Lipschitz constant; i.e., the smallest f L  0 suchthat k f ( x 1 )  f ( x 2 )

Original: k  f L k x 1  x 2 k , for all x 1 , x 2 2 X .
Revised: k  f L k x 1  x 2 k , for all x 1 , x 2 2 X .

Original: Intuitively, the constant f L measures the 100 maximal amount of variation allowed by f when the inputs change by a given amount.
Revised: Intuitively, the constant f L measures the 107 maximal amount of variation allowed by f when the inputs change by a given amount.

Original: Using the
Revised: Using the

Original: Lipschitz constant, one can deﬁne a complexity measure of a function f as the Lipschitz constant f L ofthe function over the input domain R d .
Revised: Lipschitz constant, one can deﬁne a complexity measure of a function f as the Lipschitz constant f L ofthe function over the input domain R d .

Original: Since kr x f ( x ) k 2 F  min( k, d ) kr x f ( x ) k 2 op  min( k, d ) f 2 Lwhere k · k op is the operator norm, we obtain a general bound on the GC by the Lipschitz complexity:
Revised: Since kr x f ( x ) k 2 F  min( k, d ) kr x f ( x ) k 2 op  min( k, d ) f 2 Lwhere k · k op is the operator norm, we obtain a general bound on the GC by the Lipschitz complexity:

Original: [Equation]
Revised: [Equation]

Original: While the Lipschitz smoothness of a function provides an upper bound on GC, there are a fewfundamental differences between the two quantities.
Revised: While the Lipschitz smoothness of a function provides an upper bound on GC, there are a fewfundamental differences between the two quantities.

Original: Firstly, GC is data dependent: a model can have [Figure] low GC while having high Lipschitz constant due to the model not being smooth in parts of the spacewhere there is no training data.
Revised: Firstly, GC is data dependent: a model can havelow GC while having high Lipschitz constant due to the model not being smooth in parts of the spacewhere there is no training data.

Original: Secondly, the GC can be computed exactly given a model and dataset,while for neural networks only loose upper bounds are available to estimate the Lipschitz constant.
Revised: Secondly, the GC can be computed exactly given a model and dataset,while for neural networks only loose upper bounds are available to estimate the Lipschitz constant.

Original: Geometric complexity, arc length and harmonic maps: Let us start with a motivating example:
Revised: Geometric complexity, arc length and harmonic maps: Let us start with a motivating example:

Original: Consider a dataset consisting of 10 points lying on a parabola in the plane and a large ReLU deepneural network trained via gradient descent to learn a function f : R !
Revised: Consider a dataset consisting of 10 points lying on a parabola in the plane and a large ReLU deepneural network trained via gradient descent to learn a function f : R !

Original: R that ﬁts this dataset (Fig.1).
Revised: R that ﬁts this dataset (Fig.1).

Original: Throughout training the model function seems to attain minimal arc length for a given level oftraining error.
Revised: Throughout training the model function seems to attain minimal arc length for a given level oftraining error.

Original: Recalling the formula for arc length of f , which is the integral of over [  1 , 1] ofp 1 + f 0 ( x ) 2 , and using the Taylor approximation p 1 + x 2 ⇡ 1 + x 2 2 , it follows that minimizingthe arc length is equivalent to minimizing the classic Dirichlet energy:
Revised: Recalling the formula for arc length of f , which is the integral of over [  1 , 1] ofp 1 + f 0 ( x ) 2 , and using the Taylor approximation p 1 + x 2 ⇡ 1 + x 2 2 , it follows that minimizingthe arc length is equivalent to minimizing the classic Dirichlet energy:

Original: [Equation]
Revised: [Equation]

Original: points, the complexity of the function being learned gradually grows in training, while avoiding unnecessary complexity by keeping the function arc length minimal.

Original:  where ⌦ =
Revised: 4) where ⌦ =

Original: [  1 , 1] .
Revised: [  1 , 1] .

Original: The Dirichlet energy can be thought intuitively of as a measure of the variabilityof the function f on ⌦ .
Revised: The Dirichlet energy can be thought intuitively of as a measure of the variabilityof the function f on ⌦ .

Original: Its minimizers, subject to a boundary condition f | @ ⌦
Revised: Its minimizers, subject to a boundary condition f | @ ⌦

Original: = h , are called harmonicmaps , which are maps causing the “least intrinsic stretching” of the domain ⌦ [69].
Revised: = h , are called harmonicmaps , which are maps causing the “least intrinsic stretching” of the domain ⌦ [74].

Original: The geometriccomplexity in Deﬁnition 2.1 is an unbiased estimator of a very related quantity
Revised: The geometric

Original: [Equation]
Revised: [Equation]

Original: ) where the domain ⌦ is replaced by the probability distribution of the features p X and the boundarycondition is replaced by the dataset D .
Revised: where the domain ⌦ is replaced by the probability distribution of the features p X and the boundarycondition is replaced by the dataset D .

Original: We could call the quantity deﬁned in Eqn.
Revised: We could call the quantity deﬁned in Eqn.

Original: (5) the theoreticalgeometric complexity as opposed to the empirical geometric complexity in Deﬁnition 2.1.
Revised: (5) the theoreticalgeometric complexity as opposed to the empirical geometric complexity in Deﬁnition 2.1.

Original: In the remaining sections we provide evidence that common training heuristics do indeed keep the
Revised: In the remaining sections we provide evidence that common training heuristics do indeed keep the

Original: GC low, encouraging neural networks to ﬁnd intrinsically simple solutions.
Revised: GC low, encouraging neural networks to ﬁnd intrinsically simple solutions.

Original:  Impact of initialization on geometric complexity
Revised:  Impact of initialization on geometric complexity

Original: Parameter initialization choice is an important factor in deep learning.
Revised: Parameter initialization choice is an important factor in deep learning.

Original: Although we are free to specifyexact parameter initialization values, in practice, a small number of default initialization schemeshave emerged to work well across a wide range of tasks [26, 31].
Revised: Although we are free to specifyexact parameter initialization values, in practice, a small number of default initialization schemeshave emerged to work well across a wide range of tasks [26, 32].

Original: Here, we explore the relationshipbetween some of these initialization schemes and GC.
Revised: Here, we explore the relationshipbetween some of these initialization schemes and GC.

Original: To begin, consider the one dimensional regression example that we introduced in Figure 1.
Revised: To begin, consider the one dimensional regression example that we introduced in Figure 1.

Original: In thisexperiment, we employed a standard initialization scheme to initialise the parameters: we samplethem from a truncated normal distribution with variance inversely proportional to the number of inputunits and the bias terms were set to zero.
Revised: In thisexperiment, we employed a standard initialization scheme to initialise the parameters: we samplethem from a truncated normal distribution with variance inversely proportional to the number of inputunits and the bias terms were set to zero.

Original: We observe that the initialised function on the interval[  1 , 1] is very close to the zero function (Fig. 1), and the zero function has zero GC.
Revised: We observe that the initialised function on the interval[  1 , 1] is very close to the zero function (Fig. 1), and the zero function has zero GC.

Original: This observationsuggests that initialization schemes that have low initial GC are useful for deep learning.
Revised: This observationsuggests that initialization schemes that have low initial GC are useful for deep learning.

Original: To explore this further, we consider deep ReLU networks with larger input and output spacesinitialized using the same scheme as above, and measure the GC of the resulting model.
Revised: To explore this further, we consider deep ReLU networks with larger input and output spacesinitialized using the same scheme as above, and measure the GC of the resulting model.

Original: Speciﬁcally,consider the initialized ReLU network given by f ✓ 0 : R d !
Revised: Speciﬁcally,consider the initialized ReLU network given by f ✓ 0 : R d !

Original: R k with d = 150528 and k = 1000 ,with parameter initialization ✓ 0 , and varying network depth.
Revised: R k with d = 150528 and k = 1000 ,with parameter initialization ✓ 0 , and varying network depth.

Original: We measure the ReLU network outputsize by recording the mean and maximum output values, evaluated using input values along theline P 1 + ( P 2  P 1 ) x with x 2 [0 , 1] between two diagonal points P 1 and P 2 of the hyper-cube[  1 , 1] d .
Revised: We measure the ReLU network outputsize by recording the mean and maximum output values, evaluated using input values along the line P 1 + ( P 2  P 1 ) x with x 2 [0 , 1] between two diagonal points P 1 and P 2 of the hyper-cube[  1 , 1] d .

Original: We observe that these ReLU networks initialize to functions close to the zero function,and become progressively closer to a zero valued function as the number of layer increases (Fig. 2).
Revised: We observe that these ReLU networks initialize to functions close to the zero function,and become progressively closer to a zero valued function as the number of layer increases (Fig. 2).

Original: For ReLU networks, this is not entirely surprising.
Revised: For ReLU networks, this is not entirely surprising.

Original: With biases initialised to zero, we can express a
Revised: With biases initialised to zero, we can express a

Original: ReLU network in a small neighborhood of a given point x 2 R d as a product of matrices f ✓ 0 ( x ) =
Revised: ReLU network in a small neighborhood of a given point x 2 R d as a product of matrices f ✓ 0 ( x ) =

Original: W 1 P 2 W 2 P 3 W 3 · · · P l W l x, where the W i ’s are the weight matrices and the P
Revised: W 1 P 2 W 2 P 3 W 3 · · · P l W l x, where the W i ’s are the weight matrices and the P

Original: i ’s are diagonal matrices 148 with 0 and 1 on their diagonals.
Revised: i ’s are diagonal matrices 155 with 0 and 1 on their diagonals.

Original: This representation makes it clear that at initialization the ReLUnetwork passes through the origin; i.e., f ✓ 0 (0)
Revised: This representation makes it clear that at initialization the ReLUnetwork passes through the origin; i.e., f ✓ 0 (0)

Original: = 0 .
Revised: = 0 .

Original: Furthermore, with weight matrices initialisedaround zero, using a scaling that can reduce the spread of the distribution as the matrices grow, we can expect that deeper ReLU networks generated by multiplying a large number of small-valued weights,can produce output values close to zero (for input values taken from a hyper-cube [  1 , 1] d ).
Revised: Furthermore, with weight matrices initialisedaround zero, using a scaling that can reduce the spread of the distribution as the matrices grow, we canexpect that deeper ReLU networks generated by multiplying a large number of small-valued weights,can produce output values close to zero (for input values taken from a hyper-cube [  1 , 1] d ).

Original: Weextend these results further to include additional initialization schemes and experimentally conﬁrmthat the GC can be brought close to zero with a sufﬁcient number of layers.
Revised: Weextend these results further to include additional initialization schemes and experimentally conﬁrmthat the GC can be brought close to zero with a sufﬁcient number of layers.

Original: In fact, this is true notonly for ReLU networks with the standard initialization scheme, but for a number of other commonactivation functions and initialization setups [26, 31], and even on domains much larger than thenormalized hyper-cube (Fig. 2 and SM).
Revised: In fact, this is true notonly for ReLU networks with the standard initialization scheme, but for a number of other commonactivation functions and initialization setups [26, 32], and even on domains much larger than thenormalized hyper-cube (Fig. 2 and SM Section C.1).

Original:  Impact of explicit regularization on geometric complexity
Revised:  Impact of explicit regularization on geometric complexity

Original: Next, we explore the relationship between GC and various forms of explicit regularization using acombination of theoretical and empirical results.
Revised: Next, we explore the relationship between GC and various forms of explicit regularization using acombination of theoretical and empirical results.

Original: L2 regularization:
Revised: L2 regularization:

Original: In L2 regularization, the Euclidean norm of the parameter vector is explicitlyadded to a loss function, so as to penalize solutions that have excessively large parameter values.
Revised: In L2 regularization, the Euclidean norm of the parameter vector is explicitlyadded to a loss function, so as to penalize solutions that have excessively large parameter values.

Original: This is one of the simplest, and most widely used forms of regularization.
Revised: This is one of the simplest, and most widely used forms of regularization.

Original: For linear models
Revised: For linear models

Original: ( f ✓ ( x ) =
Revised: ( f ✓ ( x ) =

Original: Ax + b ), we saw in Section 2 that the GC coincides with the Frobenius norm of the matrix
Revised: Ax + b ), we saw in Section 2 that the GC coincides with the Frobenius norm of the matrix

Original: A .
Revised: A .

Original: This means that standard L2 norm penalties on the weight matrix coincide in this case with adirect explicit regularization of the GC of the linear model.
Revised: This means that standard L2 norm penalties on the weight matrix coincide in this case with adirect explicit regularization of the GC of the linear model.

Original: For non-linear deep neural networks, wecannot directly identify the L2 norm penalty with the model GC.
Revised: For non-linear deep neural networks, wecannot directly identify the L2 norm penalty with the model GC.

Original: However, we can demonstrate thisrelationship empirically, by training a selection of neural networks with L2 regularization, each witha different regularization strength.
Revised: However, we can demonstrate thisrelationship empirically, by training a selection of neural networks with L2 regularization, each witha different regularization strength.

Original: We measure the GC for each network at the time of maximum testaccuracy.
Revised: We measure the GC for each network at the time of maximum testaccuracy.

Original: We observe empirically that strengthening L2 regularization coincides with a decrease in
Revised: We observe empirically that strengthening L2 regularization coincides with a decrease in

Original: GC values (Fig. 3).
Revised: GC values (Fig. 3).

Original: Lipschitz regularization via spectral norm regularization: A number of explicit regularizationschemes have been used to tame the Lipschitz smoothness of the model function and producesmoother models with reduced test error [1, 2, 19, 21, 29, 41, 53, 60, 73].
Revised: Lipschitz regularization via spectral norm regularization: A number of explicit regularizationschemes have been used to tame the Lipschitz smoothness of the model function and producesmoother models with reduced test error [2, 3, 19, 21, 30, 43, 57, 64, 78].

Original: Smoothness regularizationhas also been shown to be beneﬁcial outside the supervised learning context, in GANs [12, 53, 75]and reinforcement learning [10, 27].
Revised: Smoothness regularizationhas also been shown to be beneﬁcial outside the supervised learning context, in GANs [12, 57, 80]and reinforcement learning [10, 27].

Original: One successful approach to regularising the Lipschitz constantof a neural network with 1-Lipschitz activation functions (e.g. ReLU, ELU) is to constrain the spectralnorm of each layer of the network (i.e., the maximal singular values  max ( W i ) of the weight matrices
Revised: One successful approach to regularising the Lipschitz constantof a neural network with 1-Lipschitz activation functions (e.g. ReLU, ELU) is to constrain the spectralnorm of each layer of the network (i.e., the maximal singular values  max ( W i ) of the weight matrices

Original: W i ), since the product of the spectral norms of the networks weight matrices is an upper bound tothe Lipschitz constant of the model: f L   max ( W 1 ) · · ·  max ( W l ) .
Revised: W i ), since the product of the spectral norms of the networks weight matrices is an upper bound tothe Lipschitz constant of the model: f L   max ( W 1 ) · · ·  max ( W l ) .

Original: Using inequality (3), we see181 that any approach that constrains the Lipschitz constant of the model constrains GC.
Revised: Using inequality (3), we see that any approach that constrains the Lipschitz constant of the model constrains GC.

Original: To conﬁrm thistheoretical prediction, we train a ResNet18 model on CIFAR10 [44] and regularize using spectralregularization [73], an approach which adds the regularizer 
Revised: To conﬁrm thistheoretical prediction, we train a ResNet18 model on CIFAR10 [46] and regularize using spectralregularization [78], an approach which adds the regularizer

Original: P l (  max ( W l )) 2 to the model lossfunction.
Revised: P l (  max ( W l )) 2 to the model lossfunction.

Original: We observe that GC decreases as the strength of spectral regularization  increases (Fig. 3).
Revised: We observe that GC decreases as the strength of spectral regularization  increases (Fig. 3).

Original: Jacobian regularization: Direct explicit regularization with GC itself evaluated on a batch isknown as Jacobian regularization.

Original: For deep neural networks this regularization is correlated withincreased robustness and generalization of the model [34, 68, 72, 73].

Original: Related regularizers known asgradient penalties have been used for GAN training [21, 29, 40].

Original: Noise regularization:
Revised: Noise regularization:

Original: The addition of noise during training is known to be an effective form ofregularization.
Revised: The addition of noise during training is known to be an effective form ofregularization.

Original: For instance, it has been demonstrated [11] that the addition of noise to traininglabels during the optimisation of a least-square loss using SGD exerts a regularising pressure on
Revised: For instance, it has been demonstrated [11] that the addition of noise to traininglabels during the optimisation of a least-square loss using SGD exerts a regularising pressure on

Original: P ( x,y ) 2 D kr ✓ f ✓ ( x ) k 2 / | D | .
Revised: P ( x,y ) 2 D kr ✓ f ✓ ( x ) k 2 / | D | .

Original: Here, we demonstrate empirically that the GC of a ResNet18 trainedon CIFAR10 reduces as the proportion of label noise increases (Fig. 3, middle).
Revised: Here, we demonstrate empirically that the GC of a ResNet18 trainedon CIFAR10 reduces as the proportion of label noise increases (Fig. 3, middle).

Original: The same is trueof an MLP trained on MNIST [15] (Fig. 3, right).
Revised: The same is trueof an MLP trained on MNIST [15] (Fig. 3, right).

Original: In Section 5, we provide a theoretical argumentwhich justiﬁes these experiments, showing that a regularizing pressure on kr ✓ f ✓ ( x ) k 2 transfers to aregularizing pressure on kr x f ✓ ( x ) k 2 .
Revised: In Section 5, we provide a theoretical argumentwhich justiﬁes these experiments, showing that a regularizing pressure on kr ✓ f ✓ ( x ) k 2 transfers to aregularizing pressure on kr x f ✓ ( x ) k 2 .

Original: Thus, label noise in SGD in turn translates into a regularizingpressure on the GC in the case of neural networks.
Revised: Thus, label noise in SGD in turn translates into a regularizingpressure on the GC in the case of neural networks.

Original: Flatness regularization:
Revised: Flatness regularization:

Original: In ﬂatness regularization, an explicit gradient penalty term, kr ✓ L B k 2 isadded to the loss (where L B is the loss evaluated across a batch B ).
Revised: In ﬂatness regularization, an explicit gradient penalty term, kr ✓ L B k 2 isadded to the loss (where L B is the loss evaluated across a batch B ).

Original: It has been observed in practicethat ﬂatness regularization can be effective in many deep learning settings, from supervised learning[23, 66] to GAN training [4, 52, 54, 59, 61].
Revised: It has been observed in practicethat ﬂatness regularization can be effective in many deep learning settings, from supervised learning[23, 70] to GAN training [5, 56, 58, 63, 65].

Original: Flatness regularization penalizes learning trajectoriesthat follow steep slopes across the loss surface, thereby encouraging learning to follow shallowerslopes toward ﬂatter regions of the loss surface.
Revised: Flatness regularization penalizes learning trajectoriesthat follow steep slopes across the loss surface, thereby encouraging learning to follow shallowerslopes toward ﬂatter regions of the loss surface.

Original: We demonstrate empirically that GC decreases as thestrength of ﬂatness regularization increases (Fig. 3).
Revised: We demonstrate empirically that GC decreases as thestrength of ﬂatness regularization increases (Fig. 3).

Original: In the next section, we will provide a theoreticalargument that ﬂatness regularization can control GC.
Revised: In the next section, we will provide a theoreticalargument that ﬂatness regularization can control GC.

Original: Impact ofimplicit regularization  on geometric complexity
Revised: Explicit GC regularization and Jacobian regularization: All the forms of regularization abovehave known beneﬁts for improving the test accuracy in deep learning. As we saw, they all alsoimplicitly regularize GC. This raises the question as whether regularizing for GC directly andindependently of any other mechanism is sufﬁcient to improve model performance. Namely, wecan add GC computed on the batch to the loss L reg ( ✓ ) =

Revised: L B ( ✓ ) + ↵ /B

Revised: P x 2 B kr x f ✓ ( x ) k 2 F .

Revised: This is 215 actually a known form of explicit regularization, called Jacobian regularization , which has beencorrelated with increased generalization but also robustness to input shift [36, 73, 77, 78].

Revised: [73]speciﬁcally shows that adding Jacobian regularization to the loss function can lead to an increase intest set accuracy (their Tables III, IV, and V).

Revised: In SM Section C.4 we train a MLP on MNIST and a

Revised: ResNet18 on CIFAR10 regularized explicitly with the geometric complexity.

Revised: We observe an increaseof test accuracy and a decrease of GC with more regularization.

Revised: Related regularizers include gradientpenalties of the form

Revised: [Equation]

Revised: which have been used for GAN training [21, 30, 42].

Revised: We leave the full investigation of the importance of GC outside supervised learning for future work.

Revised:  Impact of implicit regularization on geometric complexity

Original: Implicit regularization is a hidden form of regularization that emerges as a bi-product of modeltraining.
Revised: Implicit regularization is a hidden form of regularization that emerges as a bi-product of modeltraining.

Original: Unlike explicit regularization, it is not explicitly added to a loss function.
Revised: Unlike explicit regularization, it is not explicitly added to a loss function.

Original: In deep learningsettings where no explicit regularization is used, it is the only form of regularization.
Revised: In deep learningsettings where no explicit regularization is used, it is the only form of regularization.

Original: Here, we usea combination of theoretical and empirical results to argue that some recently identiﬁed implicitregularization mechanisms in gradient descent [5, 50, 66] exert a regularization pressure on geometriccomplexity. 
Revised: Here, we usea combination of theoretical and empirical results to argue that some recently identiﬁed implicitregularization mechanisms in gradient descent [6, 53, 70] exert a regularization pressure on geometriccomplexity. Our argument proceeds as follows: 1) we

Original: Our argument proceeds as follows: 1) we identify a mathematical term (the implicitgradient regularization term) that characterizes implicit regularization in gradient descent, 2) wedemonstrate that this term depends on model gradients, 3) we identify the conditions where modelgradient terms apply a regularization pressure on geometric complexity.
Revised: identify a mathematical term (the implicit gradient regularization term) that characterizes implicit regularization in gradient descent, 2) wedemonstrate that this term depends on model gradients, 3) we identify the conditions where modelgradient terms apply a regularization pressure on geometric complexity.

Original: Step 1: The implicit regularization that we consider emerges as a bi-product of the discrete natureof gradient descent updates.
Revised: Step 1: The implicit regularization that we consider emerges as a bi-product of the discrete natureof gradient descent updates.

Original: In particular it has been shown that a discrete gradient update ✓ 0 =✓  h r ✓ L B ( ✓ ) over a batch of data implicitly minimizes a modiﬁed loss,
Revised: In particular it has been shown that a discrete gradient update ✓ 0 =✓  h r ✓ L B ( ✓ ) over a batch of data implicitly minimizes a modiﬁed loss,

Original: [Equation]
Revised: [Equation]

Original: 219 where the second term is called the Implicit Gradient Regularizer (IGR) [5].
Revised: 236 where the second term is called the Implicit Gradient Regularizer (IGR) [6].

Original: Gradient descentoptimisation is better characterized as a continuous ﬂow along the gradient of the modiﬁed loss,rather than the original unmodiﬁed loss.
Revised: Gradient descentoptimisation is better characterized as a continuous ﬂow along the gradient of the modiﬁed loss, rather than the original unmodiﬁed loss.

Original: By inspection, the IGR term implicitly regularizes trainingtoward trajectories with smaller loss gradients toward ﬂatter region on the loss surface [5] .
Revised: By inspection, the IGR term implicitly regularizes trainingtoward trajectories with smaller loss gradients toward ﬂatter region on the loss surface [6] .

Original: Step 2: Next, we develop this implicit regularizer term for a multi-class classiﬁcation cross-entropyloss term.
Revised: Step 2: Next, we develop this implicit regularizer term for a multi-class classiﬁcation cross-entropyloss term.

Original: We can write:
Revised: We can write (see SM Section A.2 for details):

Original: [Equation]
Revised: [Equation]

Original: where C B ( ✓ ) measures the batch gradient alignment
Revised: where C B ( ✓ ) measures the batch gradient alignment

Original: [Equation]
Revised: [Equation]

Original: and A B ( ✓ ) measures the label gradient alignment :
Revised: and A B ( ✓ ) measures the label gradient alignment :

Original: [Equation]
Revised: [Equation]

Original: where ✏ ix ( x ) = a ( f ✓ ( x ))
Revised: where ✏ ix ( x ) = a ( f ✓ ( x ))

Original: i  y i) is the signed residual and a denotes the activation function of the lastlayer.
Revised: i  y i is the signed residual and a denotes the activation function of the lastlayer.

Original: Note that this residual term arises in our calculation using r ✓ L ( x, y, ✓ ) =
Revised: Note that this residual term arises in our calculation using r ✓ L ( x, y, ✓ ) =

Original: ✏ 1 x ( ✓ ) r ✓ f 1 ✓ ( x ) +· · · + ✏ kx ( ✓ ) r ✓ f k ✓ ( x ) (see SM).
Revised: ✏ 1 x ( ✓ ) r ✓ f 1 ✓ ( x ) +· · · + ✏ kx ( ✓ ) r ✓ f k ✓ ( x ) (see SM Section A.1, Eqn.

Original: This development also extends to other widely used loss functionssuch as the cross entropy and least-square loss.
Revised: This development also extends to other widelyused loss functions such as the cross entropy and least-square loss.

Original: Step 3:
Revised: Step 3:

Original: Now, from the modiﬁed loss in Eqn.
Revised: Now, from the modiﬁed loss in Eqn.

Original: (6), we can observe the conditions under which SGDputs an implicit pressure on the gradient norms kr ✓ f i ✓ ( x ) k 2 to be small: the batch gradient alignmentand label gradient alignment terms must be small relative to the gradient norms, or positive valued.
Revised: (6), we can observe the conditions under which SGDputs an implicit pressure on the gradient norms kr ✓ f i ✓ ( x ) k 2 to be small: the batch gradient alignmentand label gradient alignment terms must be small relative to the gradient norms, or positive valued.

Original: We also derive conditions under which this implicit pressure on kr ✓ f ✓ ( x ) k 2 F transfers to a regularizing pressure on kr x f ✓ ( x ) k 2 F (see SM  for the proof):.
Revised: We also derive conditions under which this implicit pressure on kr ✓ f ✓ ( x ) k 2 F transfers to a regularizing pressure on kr x f ✓ ( x ) k 2 F (see SM Section A.3 for the proof):.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Consider a logit network f ✓ : R d !
Revised: Consider a logit network f ✓ : R d !

Original: R k with l layers, then we have the followinginequality
Revised: R k with l layers, then we have the followinginequality

Original: [Equation]
Revised: [Equation]

Original: where T i is the transfer function for layer i given by
Revised: where T i is the transfer function for layer i given by

Original: [Equation]
Revised: [Equation]

Original: where h i is the subnetwork to layer i and  max ( A ) is the maximal singular value of matrix A (i.e.,its spectral norm).
Revised: where h i is the subnetwork to layer i and  max ( A ) is the maximal singular value of matrix A (i.e.,its spectral norm).

Original: Here, we can see that for any settings where the sum T 21 ( x ) + · · · + T 2 l ( x ) of squared transferfunctions diminishes slower than the gradient norm kr ✓ f ✓ ( x ) k 2 F during training, we expect thatimplicit gradient regularization will apply a regularization pressure on GC.
Revised: Here, we can see that for any settings where the sum T 21 ( x ) + · · · + T 2 l ( x ) of squared transferfunctions diminishes slower than the gradient norm kr ✓ f ✓ ( x ) k 2 F during training, we expect thatimplicit gradient regularization will apply a regularization pressure on GC.

Original: An immediate prediction arising under these conditions is that the size of the regularization pressureon geometric complexity will depend on the implicit regularization rate h/B in Eqn.
Revised: An immediate prediction arising under these conditions is that the size of the regularization pressureon geometric complexity will depend on the implicit regularization rate h/B in Eqn.

Revised: (6) (Note that theratio h/B has been linked to implicit regularization strength in many instances [29, 71, 55, 47, 53]).

Original: Speciﬁcally,under these conditions, networks trained with larger learning rates or smaller batch sizes, or both,will apply a stronger regularization pressure on geometric complexity,
Revised: Speciﬁcally, under these conditions, networks trained with larger learning rates or smaller batch sizes,or both, will apply a stronger regularization pressure on geometric complexity,

Original: We test this prediction by performing experiments on ResNet18 trained on CIFAR10 and show resultsin Figure 4.
Revised: We test this prediction by performing experiments on ResNet18 trained on CIFAR10 and show resultsin Figure 4.

Original: The results show that while all models achieve a zero train loss, consistently the higherthe learning rate, the higher the test accuracy and the lower the GC.
Revised: The results show that while all models achieve a zero train loss, consistently the higherthe learning rate, the higher the test accuracy and the lower the GC.

Original: Similarly, we observe that thelower the batch size, the lower the GC and the higher the test accuracy.
Revised: Similarly, we observe that thelower the batch size, the lower the GC and the higher the test accuracy.

Original: The increased performance of lower batch sizes has been long studied in deep learning, under thename ‘the generalisation gap’ [33, 39].
Revised: The increased performance of lower batch sizes has been long studied in deep learning, under thename ‘the generalisation gap’ [35, 41].

Original: Crucially however, this gap was recently bridged [24],showing that full batch training can achieve the same test set performance as mini-batch gradientdescent.
Revised: Crucially however, this gap was recently bridged [24],showing that full batch training can achieve the same test set performance as mini-batch gradientdescent.

Original: To obtain these results, they use an explicit regularizer similar to the implicit regularizer in
Revised: To obtain these results, they use an explicit regularizer similar to the implicit regularizer in

Original: Eqn.
Revised: Eqn.

Original: (6), introduced to compensate for the diminished implicit regularization in the full batch case.
Revised: (6), introduced to compensate for the diminished implicit regularization in the full batch case.

Original: Their results further strengthen our hypothesis that implicit regularization via GC results in improvedgeneralisation.
Revised: Their results further strengthen our hypothesis that implicit regularization via GC results in improvedgeneralisation.

Original: When complexity is measured using a simple parameter count, a double descent phenomena has beenconsistently observed: as the number of parameters increases, the test accuracy decreases at ﬁrst,before increasing again, followed by a second descent toward a low test error value [7, 8, 56].
Revised: When complexity is measured using a simple parameter count, a double descent phenomena has beenconsistently observed: as the number of parameters increases, the test accuracy decreases at ﬁrst,before increasing again, followed by a second descent toward a low test error value [8, 9, 60].

Original: Anexcellent overview of the double descent phenomena in deep learning can be found in [7], togetherwith connections to smoothness as an inductive bias of deep networks and the role of optimisation.
Revised: Anexcellent overview of the double descent phenomena in deep learning can be found in [8], togetherwith connections to smoothness as an inductive bias of deep networks and the role of optimisation.

Original: To explore the double descent phenomena using GC we follow the set up introduced in [56]: we trainmultiple ResNet18 networks on CIFAR10 with increasing layer width, and show results in Fig.
Revised: To explore the double descent phenomena using GC we follow the set up introduced in [60]: we trainmultiple ResNet18 networks on CIFAR10 with increasing layer width, and show results in Fig.

Original: Wemake two observations: ﬁrst, like the test loss, GC follows a double descent curve as width increases;second, when plotting GC against the test loss, we observe a U-shape curve, recovering the traditionalexpected behaviour of complexity measures.
Revised: Wemake two observations: ﬁrst, like the test loss, GC follows a double descent curve as width increases;second, when plotting GC against the test loss, we observe a U-shape curve, recovering the traditionalexpected behaviour of complexity measures.

Original: Importantly, we observe the connection between thegeneralisation properties of overparametrized models and GC: after the critical region, increase inmodel size leads to a decrease in GC.
Revised: Importantly, we observe the connection between thegeneralisation properties of overparametrized models and GC: after the critical region, increase inmodel size leads to a decrease in GC.

Original: This provides further evidence that GC is able to capture modelcapacity in a meaningful way, suggestive of a reconciliation of traditional complexity theory anddeep learning.
Revised: This provides further evidence that GC is able to capture modelcapacity in a meaningful way, suggestive of a reconciliation of traditional complexity theory anddeep learning.

Original: Related  Work
Revised: Related and Future Work

Original: The aim of this work is to introduce a measure which captures the complexity of neural networks.
Revised: The aim of this work is to introduce a measure which captures the complexity of neural networks.

Original: There are many other approaches aiming at doing so, ranging from naive parameter count and moredata driven approaches [35, 45, 51] to more traditional measures such as the VC dimension and
Revised: There are many other approaches aiming at doing so, ranging from naive parameter count and moredata driven approaches [37, 48, 54] to more traditional measures such as the VC dimension and

Original: Rademacher complexity [6, 36, 43, 70] which focus on the entire hypothesis space.
Revised: Rademacher complexity [7, 38, 45, 75] which focus on the entire hypothesis space.

Original: The deﬁnitionof GC we develop here is similar to the Kolmogorov complexity [63] and the minimum descriptionlength [32] and extends connections to the  volume of the graph of a neural network [16]. 
Revised: As we sawin Fig. 1, the interpolating ReLU network with minimal GC is also the piecewise linear functionwith minimal volume or length [16]. In 1D this minimal function can be described only with theinformation given by the data points.

Original: GC hassimilarities to the effective dimension which is deﬁned in relation to the training data, but is computedusing the eigenspecturm of the Hessian of the loss [51].
Revised: This description with minimal information is reminiscent of the Kolmogorov complexity [67] and the minimum description length [34], and we believe that the exactrelationship between these notions, GC, and minimal volume is worth investigating. Similarly, [1]argues that ﬂat solutions have low information content, and for neural networks these ﬂat regionsare also the regions of low loss gradient and thus of low GC, as explained by the Transfer Theoremin Section 5. Another recent measure of complexity is the effective dimension which is deﬁned inrelation to the training data, but is computed using the spectral decomposition of the loss Hessian[54], making ﬂat regions in the loss surface also regions of low complexity w.r.t.

Original: In that paper, theauthors also show howthe effective dimension can be used to understand the double descent phenomena and as an efﬁcientmechanism for model selection.
Revised: this measure. Thishints toward the effective dimension being related to GC. Note that similarly to GC, the effectivedimension can also capture the double descent phenomena. The effective dimension is an efﬁcientmechanism for model selection [54], which our experiments seem also to indicate may be the casefor GC.

Original: GC is close to considerations about smoothness [60] and the Sobolev norm implicit regularization[50].
Revised: GC is close to considerations about smoothness [64] and the Sobolev norm implicit regularization[53].

Original: While GC and Lipchitz smoothness are connected, here we focused on a tractable quantityfor neural networks and its connections with existing regularizers.
Revised: While GC and Lipchitz smoothness are connected, here we focused on a tractable quantityfor neural networks and its connections with existing regularizers.

Original: Smoothness regularization hasbeen particularly successful in the GAN literature [13, 29, 40, 53, 75], and the connection between
Revised: Smoothness regularization hasbeen particularly successful in the GAN literature [13, 30, 42, 57, 80], and the connection between

Original: Lipschitz smoothness and GC begs the question of whether their success is due to their implicitregularization of GC, but we leave the application of GC outside supervised learning for future work.
Revised: Lipschitz smoothness and GC begs the question of whether their success is due to their implicitregularization of GC, but we leave the application of GC outside supervised learning for future work.

Original: The Dirichlet energy in Eqn.
Revised: The Dirichlet energy in Eqn.

Original: (4) is a well-known quantity in harmonic function theory [20] anda fundamental concept throughout mathematics, physics and, more recently, 3D modeling [69],manifold learning [14], and image processing [25, 62].
Revised: (4) is a well-known quantity in harmonic function theory [20] anda fundamental concept throughout mathematics, physics and, more recently, 3D modeling [74],manifold learning [14], and image processing [25, 66].

Original: Minimizers of the Dirichlet energy are harmonic and thus enjoy certain guarantees in regularity; i.e., any such solution is a smooth function.
Revised: Minimizers of the Dirichlet energy areharmonic and thus enjoy certain guarantees in regularity; i.e., any such solution is a smooth function.

Original: It has been demonstrated that for mean squared error regression on wide-shallow ReLU networks,gradient descent is biased towards smooth functions at interpolation [37].
Revised: It has been demonstrated that for mean squared error regression on wide-shallow ReLU networks,gradient descent is biased towards smooth functions at interpolation [39].

Original: Similarly, our work suggeststhat neural networks, through a mechanism of implicit regularization of GC, are biased towardsminimal Dirichlet energy and thus encourage smooth interpolation.
Revised: Similarly, our worksuggests that neural networks, through a mechanism of implicit regularization of GC, are biasedtowards minimal Dirichlet energy and thus encourage smooth interpolation.

Original: The dataset plays an important role in the deﬁnition of GC. Just as the classic Dirichlet energy is ameasure of the variability of a function, the GC measures, in some sense, the sensitivity of the modelfunction at points in the dataset. Lower GC indicates more robustness of the model to input noise andtakes a similar form to the class of Tikhonov regularizers which has been shown to be equivalent totraining with input noise [9].
Revised: It may be interesting tounderstand how the relationship between GC, Dirichlet energy, and harmonic theory can help futherimprove the learning process, in particular in transfer learning, or OOD generalization.

Original:  Discussion
Revised:  Discussion

Original: We introduce the notion of geometric complexity as a means of measuring model complexity indeep learning.

Original: The deﬁnition is motivated by the Dirichlet energy integral, a familiar quantity inthe ﬁeld of geometric analysis and harmonic maps, which is a ﬁrst order measure of the variabilityof a function.

Original: The deﬁnition of geometric complexity that we propose takes a discretized versionof the Dirichlet energy and thus relies not only on the model function but on the training data aswell.

Original: We have shown that geometric complexity is well-suited for the analysis of neural networks andprovides a convincing framework for understanding the role of many familiar training heuristics indeep learning, including common initialization schemes and explicit and implicit regularization.

Original: Wealso show how geometric complexity captures the double-descent behaviour that has been observedin the test loss as the parameter count increases.

Original: We demonstrate these results through experimentson the CIFAR10 and MNIST datasets and provide theoretical justiﬁcation.

Original: By measuring the geometric complexity at initialization, we show that common initialization schemesproduce initial model functions with low (near zero) geometric complexity.

Original: In fact, we show thiseffect is exacerbated as the number of layers increases, pushing the geometric complexity closer andcloser to zero.

Original: We show through experiments that well known explicit regularization techniques suchas L2 parameter norm regularization, spectral norm and ﬂatness regularization and label noise alltranslate to regularization of the geometric complexity through training.

Original: We also show how implicitregularization mechanisms exert a regularization pressure on geometric complexity and verify thiseffect through experiments on the learning rate and batch size, two quantities that have been shown toinﬂuence this implicit regularization effect.

Original: Lastly, we draw a compelling connection between geometric complexity and double descent.

Original: Preciselymeasuring the complexity of deep learning models is a complicated task and active area of research.

Original: When using a naive parameter count to determine model complexity a double descent phenomenaoccurs where, after interpolation, the test error actually decreases as the model complexity increases.

Original: However, when geometric complexity is used instead of a parameter account, we observe throughexperiments that a classical U-shaped test error curve is recovered, suggesting that the geometriccomplexity is a useful proxy for measuring the generalization properties of deep neural networks,particularly for overparameterized models.

Original: Limitations: The theoretical arguments presented in this work have a focus on ReLU activationsand DNN architectures, with log-likelihood losses coming from the exponential family, such asmulti-dimensional regression with least-square losses, or multi-class classiﬁcation with cross-entropyloss.
Revised: In terms of limitations , the theoretical arguments presented in this work have a focus on ReLUactivations and DNN architectures, with log-likelihood losses coming from the exponential family,such as multi-dimensional regression with least-square losses, or multi-class classiﬁcation withcross-entropy loss.

Original: The experimental results are obtained on the image datasets MNIST and CIFAR using DNNand ResNet architectures.
Revised: The experimental results are obtained on the image datasets MNIST and CIFARusing DNN and ResNet architectures.

Original: Societal impact: We are not introducing new training methods, but focus on providing a betterunderstanding of the impact of common existing training methods.
Revised: In terms of societal impact , we are not introducing new training methods, but focus on providinga better understanding of the impact of common existing training methods.

Original: We hope that this understandingwill ultimately lead to more efﬁcient training techniques.
Revised: We hope that thisunderstanding will ultimately lead to more efﬁcient training techniques.

Original: While existing training methods may havetheir own risk, we do not foresee any potential negative societal impact of this work.
Revised: While existing trainingmethods may have their own risk, we do not foresee any potential negative societal impact of thiswork.

Original: Conclusion: Altogether, geometric complexity provides a useful lens for understanding deeplearning and sheds light into why neural networks are able to achieve low test error with highlyexpressive models.
Revised: In conclusion, altogether, geometric complexity provides a useful lens for understanding deep learningand sheds light into why neural networks are able to achieve low test error with highly expressivemodels.

Original: We hope this work will encourage further research around this new connection,and help to better understand current best practices in model training as well as discover new ones.
Revised: We hope this work will encourage further research around this new connection, and help tobetter understand current best practices in model training as well as discover new ones.

Original: Checklist
Revised: Checklist

Original: For all authors..
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The data is widely available (MNIST and CIFAR10).
Revised: [No] The data iswidely available (MNIST and CIFAR10).

Original: The code we use to train the model is stan-
Revised: The code we use to train the model is stan-

Original: dard training code using standard neural network architectures (MLP and ResNet18)
Revised: dard training code using standard neural network architectures (MLP and ResNet18)

Original: available out-of-the-box in most ml libraries.
Revised: available out-of-the-box in most ml libraries.

Original: We included detailed instructions with all
Revised: We included detailed instructions with all

Original: the hyper-parameter used in the supplemental material to reproduce them.
Revised: the hyper-parameter used in the supplemental material to reproduce them.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We use the
Revised: [Yes] We use the

Original: CIFAR10 and MNIST datasets for some of our experimental results.
Revised: CIFAR10 and MNIST datasets for some of our experimental results.

Original: We cite
Revised: We cite

Revised: both

Revised: in

Original: both in the paper, for CIFAR following the guidance on Krizhevsky’s homepage: https://www.cs.toronto.edu/ kriz/cifar.html.
Revised: the paper , for CIFAR following the guidance on Krizhevsky’s homepage:https://www.cs.toronto.edu/ kriz/cifar.html.

Original: (b) Did you mention the license of the assets?
Revised: b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: ) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: ) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: ) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: ) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: Board (IRB) approvals, if applicable? [N/A] ) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]