Original: Predicting generalization with degrees of freedom in neural networks
Revised: Predicting generalization with degrees of freedom in neural networks

Original: Abstract
Revised: Abstract

Original: Model complexity is fundamentally tied to predictive power in the sciences as well as in applications.
Revised: Model complexity is fundamentally tied to predictive power in the sciences as well as in applications.

Original: However, there is a divergence between naive measures of complexity such as parameter count and the generalization performance of overparameterized machine learning models.
Revised: However, there is a divergence between naive measures of complexity such as parameter count and the generalization performance of overparameterized machine learning models.

Original: Prior empirical approaches to capturing intrinsic complexity in a more sophisticated manner than parameter count are computationally intractable, do not capture the implicitly regularizing effects of the entire machine-learning pipeline, or do not provide a quantitative ﬁt to the double descent behavior of overparameterized models.
Revised: Prior empirical approaches to capturing intrinsic complexity in a more sophisticated manner than parameter count are computationally intractable, do not capture the implicitly regularizing effects of the entire machine-learning pipeline, or do not provide a quantitative ﬁt to the double descent behavior of overparameterized models.

Original: In this work, we introduce an empirical complexity measure inspired by the classical notion of generalized degrees of freedom in statistics.
Revised: In this work, we introduce an empirical complexity measure inspired by the classical notion of generalized degrees of freedom in statistics.

Original: This measure can be approximated efﬁciently and is a function of the entire machine learning training pipeline.
Revised: This measure can be approximated efﬁciently and is a function of the entire machine learning training pipeline.

Original: We demonstrate that this measure correlates with generalization performance in the double-descent regime.
Revised: We demonstrate that this measure correlates with generalization performance in the double-descent regime.

Original: 1. Introduction
Revised: 1. Introduction

Original: What makes a machine learning model a good scientiﬁc explanation?
Revised: What makes a machine learning model a good scientiﬁc explanation?

Original: Classical perspectives call for trading off low complexity and high predictive accuracy (Forster & Sober, 1994).
Revised: Classical perspectives call for trading off low complexity and high predictive accuracy (Forster & Sober, 1994).

Original: However, deep learning approaches to scientiﬁc modeling (Baraniuk et al., 2020; Bianchini et al., 2020; Raghu & Schmidt, 2020; Hope et al., 2022) come with new challenges in navigating this trade-off due to the difﬁculty in assessing both complexity and predictive accuracy.
Revised: However, deep learning approaches to scientiﬁc modeling (Baraniuk et al., 2020; Bianchini et al., 2020; Raghu & Schmidt, 2020; Hope et al., 2022) come with new challenges in navigating this trade-off due to the difﬁculty in assessing both complexity and predictive accuracy.

Original: First, many established measures of complexity do not apply to deep learning models because they are vacuous (Dziugaite
Revised: First, many established measures of complexity do not apply to deep learning models because they are vacuous (Dziugaite

Original: [Figure]
Revised: [Figure]

Original: g e n e r a li z a ti on g a p
Revised: g e n e r a li z a ti on g a p

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: c a p a c it y m u lti p li e r
Revised: c a p a c it y m u lti p li e r

Original: Roy, 2017),
Revised: Roy, 2017),

Original: or they do not capture the non-monotonic behavior of the evaluation error to which complexity is fundamentally tied (Belkin et al., 2019; Nakkiran et al., 2021).
Revised: or they do not capture the non-monotonic behavior of the evaluation error to which complexity is fundamentally tied (Belkin et al., 2019; Nakkiran et al., 2021).

Original: Secondly, over-parameterized deep learning models often obtain zero training error (Zhang et al., 2017), meaning that model selection by predictive accuracy on the training set alone is not possible, while maintaining a test set representative of downstream applications is challenging in practice (Nagarajan et al., 2021).
Revised: Secondly, over-parameterized deep learning models often obtain zero training error (Zhang et al., 2017), meaning that model selection by predictive accuracy on the training set alone is not possible, while maintaining a test set representative of downstream applications is challenging in practice (Nagarajan et al., 2021).

Original: These challenges mean that practitioners lack foundational methods for model selection, hindering the reliable deployment of many machine learning techniques (Amodei et al., 2016; D’Amour et al., 2020; Geirhos et al., 2020) and giving rise to poorly understood pathologies such as susceptibility to adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015).
Revised: These challenges mean that practitioners lack foundational methods for model selection, hindering the reliable deployment of many machine learning techniques (Amodei et al., 2016; D’Amour et al., 2020; Geirhos et al., 2020) and giving rise to poorly understood pathologies such as susceptibility to adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015).

Original: In this work, we explore the extent to which the generalized degrees of freedom (GDoF) framework introduced by
Revised: In this work, we explore the extent to which the generalized degrees of freedom (GDoF) framework introduced by

Original: Efron (1983) and Ye (1998) can respond to these challenges.
Revised: Efron (1983) and Ye (1998) can respond to these challenges.

Original: GDoF, as its name suggests, generalizes the basic notion of degrees of freedom for non-linear models, and is equivalent to the number of features for linear models.
Revised: GDoF, as its name suggests, generalizes the basic notion of degrees of freedom for non-linear models, and is equivalent to the number of features for linear models.

Original: This correspondence provides a natural measurement of the complexity of machine learning models, including deep neural networks.
Revised: This correspondence provides a natural measurement of the complexity of machine learning models, including deep neural networks.

Original: Given the fundamental connection between complexity and generalization, we hypothesize that the GDoF predicts a model’s generalization performance.
Revised: Given the fundamental connection between complexity and generalization, we hypothesize that the GDoF predicts a model’s generalization performance.

Original: GDoF has been ex- plored in various contexts, though Gao & Jojic (2016) was the ﬁrst to study this measurement for deep neural networks.
Revised: GDoF has been explored in various contexts, though Gao & Jojic (2016) was the ﬁrst to study this measurement for deep neural networks.

Original: In this work, we make the following additional contributions:
Revised: In this work, we make the following additional contributions:

Original: • We introduce approximations that enable scalable computation of the GDoF during training of deep neural networks (Section ( 3 )).
Revised: • We introduce approximations that enable scalable computation of the GDoF during training of deep neural networks (Section ( 3 )).

Original: • We reconcile the double-descent phenomenon with the classical trade-off between complexity and predictive accuracy by demonstrating the monotonicity of the generalization gap in GDoF (Fig.
Revised: • We reconcile the double-descent phenomenon with the classical trade-off between complexity and predictive accuracy by demonstrating the monotonicity of the generalization gap in GDoF (Fig.

Original: • We use the online nature of the approximation to reveal the evolution of GDoF over training (Fig.
Revised: • We use the online nature of the approximation to reveal the evolution of GDoF over training (Fig.

Original: ( 3 ) and Fig.
Revised: ( 3 ) and Fig.

Original: • We demonstrate that controlling GDoF via subspace training (Li et al., 2018) can improve generalization (Fig.
Revised: • We demonstrate that controlling GDoF via subspace training (Li et al., 2018) can improve generalization (Fig.

Original: ( 4 ) and Fig.
Revised: ( 4 ) and Fig.

Original: 2. Background
Revised: 2. Background

Original: When reasoning about model complexity, we contrast changes in the parameter dimension of a machine learning model—measures proportional to the parameter count of a model—with changes in the effective dimensionality , which is indirectly controlled by various components such as the architecture and training procedure.
Revised: When reasoning about model complexity, we contrast changes in the parameter dimension of a machine learning model—measures proportional to the parameter count of a model—with changes in the effective dimensionality , which is indirectly controlled by various components such as the architecture and training procedure.

Original: While trivially equivalent re-parametrizations—for instance, replacing a connection weight by the product of two weights, w = w 1 w 2 —may result in vastly different parameter dimensions, effective dimensionality captures the intrinsic complexity as a result of the complete system speciﬁcation.
Revised: While trivially equivalent re-parametrizations—for instance, replacing a connection weight by the product of two weights, w = w 1 w 2 —may result in vastly different parameter dimensions, effective dimensionality captures the intrinsic complexity as a result of the complete system speciﬁcation.

Original: According to classic notions of complexity-ﬁt trade-off, models with low effective dimensionality have low complexity, are less likely to overﬁt and thus have lower generalization error (Schmidhuber, 1997).
Revised: According to classic notions of complexity-ﬁt trade-off, models with low effective dimensionality have low complexity, are less likely to overﬁt and thus have lower generalization error (Schmidhuber, 1997).

Original: The preference for simplicity in model selection is also known as Occam’s Razor (MacKay, 1991).
Revised: The preference for simplicity in model selection is also known as Occam’s Razor (MacKay, 1991).

Original: However, the classical trade-off between complexity and predictive power, which focuses on the bias and variance tradeoff (Geman et al., 1992), has been challenged by the recently re-discovered double descent phenomenon (Belkin et al., 2019): While large models are traditionally understood to be inherently more complex and thus subject to poor generalization as a result of overﬁtting, modern over-parameterized machine learning models exhibit surprisingly good generalization performance (Zhang et al., 2017).
Revised: However, the classical trade-off between complexity and predictive power, which focuses on the bias and variance tradeoff (Geman et al., 1992), has been challenged by the recently re-discovered double descent phenomenon (Belkin et al., 2019): While large models are traditionally understood to be inherently more complex and thus subject to poor generalization as a result of overﬁtting, modern over-parameterized machine learning models exhibit surprisingly good generalization performance (Zhang et al., 2017).

Original: While the origin of the discrepancy between parameter dimension and effective dimensionality is outside of the scope of this work, previous work has shown that it may arise from implicit regularization due to various components of the machine-learning pipeline, including the model architecture (Golubeva et al., 2020), the speciﬁc initialization procedure (Kubo et al., 2019; Mehta et al., 2021), gradient-based optimization (Barrett & Dherin, 2021), and overparametrization itself (Neyshabur et al., 2018; Advani et al., 2020).
Revised: While the origin of the discrepancy between parameter dimension and effective dimensionality is outside of the scope of this work, previous work has shown that it may arise from implicit regularization due to various components of the machine-learning pipeline, including the model architecture (Golubeva et al., 2020), the speciﬁc initialization procedure (Kubo et al., 2019; Mehta et al., 2021), gradient-based optimization (Barrett & Dherin, 2021), and overparametrization itself (Neyshabur et al., 2019; Advani et al., 2020).

Original: Various attempts have been made to capture effective dimensionality via model complexity from the perspective of a model’s loss landscape (Larsen et al., 2021) and, in particular, its local curvature (Maddox et al., 2020; Loukas et al., 2021).
Revised: Various attempts have been made to capture effective dimensionality via model complexity from the perspective of a model’s loss landscape (Larsen et al., 2021) and, in particular, its local curvature (Maddox et al., 2020; Loukas et al., 2021).

Original: The statistical concept of degrees of freedom can also be related to the loss landscape.
Revised: The statistical concept of degrees of freedom can also be related to the loss landscape.

Original: For linear models, the degree of freedom is equivalent to parameter count, but
Revised: For linear models, the degree of freedom is equivalent to parameter count, but

Original: Efron (1983) and Ye (1998) generalized this concept to nonlinear models from the perspectives of expected optimism and average sensitivity as follows:
Revised: Efron (1983) and Ye (1998) generalized this concept to nonlinear models from the perspectives of expected optimism and average sensitivity as follows:

Original: For a classiﬁer f parameterized by θ , the GDoF is deﬁned from the derivatives of the model-predicted label f n,c with respect to the data label y n,
Revised: For a classiﬁer f parameterized by θ , the GDoF is deﬁned from the derivatives of the model-predicted label f n,c with respect to the data label y n,c : Φ :=

Original: [Equation]
Revised: N (cid:88) n =

Original: [Equation]
Revised: C (cid:88) c =

Original:  where n ∈ { 1 , . . .
Revised: ∂f n,c ( θ ) ∂y n,c , (1) where n ∈ { 1 , . . .

Original: , N } is the index of the sample in the training set, c ∈ { 1 , . . .
Revised: , N } is the index of the sample in the training set, c ∈ { 1 , . . .

Original: , C } is the index of the class.
Revised: , C } is the index of the class.

Original: When explicit dependence on the training procedure is required, we further expand Eq.
Revised: When explicit dependence on the training procedure is required, we further expand Eq.

Original: (1) and specify the algorithm A , the initial parameter θ 0 , and the training step t such that
Revised: (1) and specify the algorithm A , the initial parameter θ 0 , and the training step t such that

Original: the
Revised: the

Original: GDoF can reﬂect the model complexity at any step t before convergence as
Revised: GDoF can reﬂect the model complexity at any step t before convergence as

Original: [Equation]
Revised: [Equation]

Original: Naively, computing these derivatives would require backpropagating through the whole training procedure speciﬁed by A .
Revised: Naively, computing these derivatives would require backpropagating through the whole training procedure speciﬁed by A .

Original: The GDoF measurement Φ in Eq.
Revised: The GDoF measurement Φ in Eq.

Original: (2) can be interpreted as the sensitivity of the model f to perturbation in the training label y n,c .
Revised: (2) can be interpreted as the sensitivity of the model f to perturbation in the training label y n,c .

Original: Gao & Jojic (2016) introduced GDoF to neural networks by approximating the derivatives in Eq.
Revised: Gao & Jojic (2016) introduced GDoF to neural networks by approximating the derivatives in Eq.

Original: (2) by a speciﬁc random perturbation and demonstrated that
Revised: (2) by a speciﬁc random perturbation and demonstrated that

Original: the
Revised: the

Original: GDoF of such over-parameterized models are only a small fraction of the total number of parameters.
Revised: GDoF of such over-parameterized models are only a small fraction of the total number of parameters.

Original: However, (Gao &
Revised: However, (Gao &

Original: Jojic, 2016) computed GDoF by training a perturbed model, as well as the original model, on the complete training dataset until convergence, thus requiring dual training runs for each iteration of computing the GDoF.
Revised: Jojic, 2016) computed GDoF by training a perturbed model, as well as the original model, on the complete training dataset until convergence, thus requiring dual training runs for each iteration of computing the GDoF.

Original: Here we show that the approximation of Eq.
Revised: Here we show that the approximation of Eq.

Original: (2) can be broken-up in both space—with mini-batches—and in time—by incremental computation.
Revised: (2) can be broken up in both space—with mini-batches—and in time—by incremental computation.

Original: In addition, the latter incremental approximation enables us to reveal the evolution of GDoF during training.
Revised: In addition, the latter incremental approximation enables us to reveal the evolution of GDoF during training.

Original: 3. Method
Revised: 3. Method

Original: In this work, we introduce additional approximations to
Revised: In this work, we introduce additional approximations to

Original: GDoF, deﬁned as Φ in Eq.
Revised: GDoF, deﬁned as Φ in Eq.

Original: (2) for a more scalable estimate of model complexity.
Revised: (2) for a more scalable estimate of model complexity.

Original: The ﬁrst straightforward step of approximation is using a Monte-Carlo estimator based on a subset B < N of samples from the training set:
Revised: The ﬁrst straightforward step of approximation is using a Monte-Carlo estimator based on a subset B < N of samples from the training set:

Original: [Equation]
Revised: [Equation]

Original: The second is to remove the requirement of training the perturbed model to convergence by exploiting the sequential updating of model parameters θ during gradient-based training.
Revised: The second is to remove the requirement of training the perturbed model to convergence by exploiting the sequential updating of model parameters θ during gradient-based training.

Original: The following proposition shows that, given the GDoF of a model trained from initialization to time t , the GDoF at step t (cid:48) > t can be approximated by simply adding the additional sensitivity brought by the increment ∆ θ =
Revised: The following proposition shows that, given the GDoF of a model trained from initialization to time t , the GDoF at step t (cid:48) > t can be approximated by simply adding the additional sensitivity brought by the increment ∆ θ =

Original: [Equation]
Revised: [Equation]

Original: as long as ∆ θ is small.
Revised: as long as ∆ θ is small.

Original: Proposition 1.
Revised: Proposition 1.

Original: The GDoF at step t (cid:48) is related to the GDoF at t < t (cid:48) via:
Revised: The GDoF at step t (cid:48) is related to the GDoF at t < t (cid:48) via:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ∆Φ( t, t (cid:48) )
Revised: where ∆Φ( t, t (cid:48) )

Original: =
Revised: =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: [Equation]
Revised: [Equation]

Original: measures the increment of sensitivity after training f with
Revised: measures the increment of sensitivity after training f with

Original: A to step t (cid:48) from θ t at step t .
Revised: A to step t (cid:48) from θ t at step t .

Original: Proof.
Revised: Proof.

Original: We ﬁrst use the following shorthand to represent the instance- and class-wise contributions to Φ :
Revised: We ﬁrst use the following shorthand to represent the instance- and class-wise contributions to Φ :

Original: [Equation]
Revised: [Equation]

Original: With this, Eq.
Revised: With this, Eq.

Original: (2) can be written as
Revised: (2) can be written as

Original: [Equation]
Revised: [Equation]

Original: For additional brevity, we drop the example subscript i and the class subscript c when they are irrelevant to the derivation, giving
Revised: For additional brevity, we drop the example subscript i and the class subscript c when they are irrelevant to the derivation, giving

Original: [Equation]
Revised: [Equation]

Original: We then construct a Taylor expansion of f ( θ ) near θ = θ t :
Revised: We then construct a Taylor expansion of f ( θ ) near θ = θ t :

Original: [Equation]
Revised: [Equation]

Original: which gives, when ∆ θ = θ t (cid:48) − θ t is small,
Revised: which gives, when ∆ θ = θ t (cid:48) − θ t is small,

Original: [Equation]
Revised: [Equation]

Original: Plugging in the approximation Eq.
Revised: Plugging in the approximation Eq.

Original: (10) into Eq.
Revised: (10) into Eq.

Original: (8) gives
Revised: (8) gives

Original: [Equation]
Revised: [Equation]

Original: Here the approximation comes from dropping the second order term, which is negligible when the update ∆ θ or the second derivative
Revised: Here the approximation comes from dropping the second order term, which is negligible when the update ∆ θ or the second derivative

Original: [Equation]
Revised: [Equation]

Original: 2 f
Revised: 2 f

Original: [Equation]
Revised: [Equation]

Original: t is small.
Revised: t is small.

Original: Moreover, since φ ( t, t (cid:48) ) depends on y only through the update ∆ θ , we can employ the chain rule to give:
Revised: Moreover, since φ ( t, t (cid:48) ) depends on y only through the update ∆ θ , we can employ the chain rule to give

Original: [Equation]
Revised: [Equation]

Original: where the last line of approximation comes from using the derivatives of ∂f ( θ ) ∂θ at θ t for θ t (cid:48) .
Revised: where the last line of approximation comes from using the derivatives of ∂f ( θ ) ∂θ at θ t for θ t (cid:48) .

Original: In Eq.
Revised: In Eq.

Original: (12), we used the deﬁnition of the derivative, while changing the variable θ = θ t + ∆ :
Revised: (12), we used the deﬁnition of the derivative, while changing the variable θ = θ t + ∆ :

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (13) with Eq.
Revised: (13) with Eq.

Original: (11), we have
Revised: (11), we have

Original: [Equation]
Revised: [Equation]

Original: We can now obtain Proposition ( 1 ) from the deﬁnitions of φ and Φ .
Revised: We can now obtain Proposition ( 1 ) from the deﬁnitions of φ and Φ .

Original: Although several steps of the approximation require 
Revised: Although several steps of the approximation require (cid:107)

Original: [Equation]
Revised: [Equation]

Original: to be small, in our experiments we found the resulting estimator of GDoF predictive of generalization even when ∆ θ represents a change of up to 1000 update steps.
Revised: to be small, in our experiments we found the resulting estimator of GDoF predictive of generalization even when ∆ θ represents a change of up to 1000 update steps.

Original: In addition, we found performing multiple update steps while re-sampling mini-batches ( K in Algorithm ( 1 )) improves the estimate of GDoF, as in our experiments in Section ( 4 ).
Revised: In addition, we found performing multiple update steps while re-sampling mini-batches ( K in Algorithm ( 1 )) improves the estimate of GDoF, as in our experiments in Section ( 4 ).

Original: We leave detailed analysis of the approximations, such as the error accumulated through the trajectory, for future work.
Revised: We leave detailed analysis of the approximations, such as the error accumulated through the trajectory, for future work.

Original: The next section empirically demonstrates the predictive power of the measurement Eq.
Revised: The next section empirically demonstrates the predictive power of the measurement Eq.

Original: (2) and the two approximations we make in Eq.
Revised: (2) and the two approximations we make in Eq.

Original: (3) and Eq.
Revised: (3) and Eq.

Original: (4) for neural networks.
Revised: (4) for neural networks.

Original: Algorithm ( 1 ) summarizes the algorithm for computing the
Revised: Algorithm ( 1 ) summarizes the algorithm for computing the

Original: GDoF from a series of checkpoints saved during training.
Revised: GDoF from a series of checkpoints saved during training.

Original: Notice that taking T = 0 , and K = ∞ ( i.e., training the original and perturbed model from initialization until convergence) recovers the algorithm of Gao & Jojic (2016).
Revised: Notice that taking T = 0 , and K = ∞ ( i.e., training the original and perturbed model from initialization until convergence) recovers the algorithm of Gao & Jojic (2016).

Original: Algorithm
Revised: Algorithm

Original: Require: saved checkpoints C 1 , C 2 , . . .
Revised: Require: saved checkpoints C 1 , C 2 , . . .

Original: C T , the number of update steps K , mini-batch size B , perturbation step (cid:15)
Revised: C T , the number of update steps K , mini-batch size B , perturbation step (cid:15)

Original: Φ ← 0 for t = 1 , . . .
Revised: Φ ← 0 for t = 1 , . . .

Original: T do
Revised: T do

Original: Load model f ( θ t ) from checkpoint C t for k = 1 , . . .
Revised: Load model f ( θ t ) from checkpoint C t for k = 1 , . . .

Original: K do
Revised: K do

Original: Sample a mini-batch of data { x b , y b } Bb =
Revised: Sample a mini-batch of data { x b , y b } Bb =

Original: Sample perturbation { z b } Bb =1 ∼ N (0 , 1)
Revised: Sample perturbation { z b } Bb =1 ∼ N (0 , 1)

Original: Update f
Revised: Update f

Original: ( θ t +1 ) using { x b , y b } Bb =
Revised: ( θ t +1 ) using { x b , y b } Bb =

Original: Update f (cid:48)
Revised: Update f (cid:48)

Original: ( θ t +1 ) using { x b , y b + (cid:15) · z b } Bb = end for
Revised: ( θ t +1 ) using { x b , y b + (cid:15) · z b } Bb = end for

Original: Approximate ∆Φ( t, t + 1) ←
Revised: Approximate ∆Φ( t, t + 1) ←

Original: NB (cid:80)
Revised: NB (cid:80)

Original: [Table]

Original:  =1 (cid:80) Cc = [Table] z b ( c ) f (cid:48) 
Revised: Bb =1 (cid:80) Cc =1 z b ( c ) f (cid:48) b,c

Original: i, [Table]
Revised: ( θ t +1 )

Original: [Table] [Table]
Revised: − f b,c ( θ t +1 ) (cid:15)

Original: Accumulate Φ( t + 1) ← Φ( t ) + ∆Φ( t, t + 1) end for
Revised: Accumulate Φ( t + 1) ← Φ( t ) + ∆Φ( t, t + 1) end for

Original: 4. Experiments
Revised: 4. Experiments

Original: We demonstrate results measuring the GDoF of a convolutional neural network (CNN) trained on the CIFARdataset.
Revised: We demonstrate results measuring the GDoF of a convolutional neural network (CNN) trained on the CIFARdataset.

Original: The CNNs follow the architecture from Nakkiran et al.
Revised: The CNNs follow the architecture from Nakkiran et al.

Original: and have 3 convolutional layers.
Revised: (2021) with three convolutional layers.

Original: We vary their capacities via a capacity multiplier , the multiple of the number of ﬁlter channels.
Revised: We vary their capacities via a capacity multiplier , the multiple of the number of ﬁlter channels.

Original: We thus have a batch of models whose ﬁrst layer hasthe ﬁlter sizesof { 1 , 2 , 4 , 8 , 12 , 16 , 20 , 24 , 32 , 40 , 48 , 64 } .
Revised: We thus have a batch of models whose ﬁrst layer has ﬁlter sizes { 1 , 2 , 4 , 8 , 12 , 16 , 20 , 24 , 32 , 40 , 48 , 64 } .

Original: All models are trained with a batch size of 128 for 3 × 10 5 training iterations.
Revised: All models are trained with a batch size of 128 for 3 × 10 5 training iterations.

Original: In Fig.
Revised: In Fig.

Original: ( 2 ) we plot the training error, evaluation error, alongside their difference—the generalization gap—and the approximated GDoF for models with different capacities.
Revised: ( 2 ) we plot the training error and the evaluation error alongside their difference—the generalization gap—and the approximated GDoF for models with different capacity multipliers.

Original: The double descent curve (Belkin et al., 2019) can be observed in the evaluation error curve, and the measured GDoF closely tracks the generalization gap.
Revised: The double descent curve (Belkin et al., 2019) can be observed in the evaluation error curve, and the measured GDoF closely tracks the generalization gap.

Original: To further investigate the correlation between the GDoF and the generalization gap, we represent the same data in a scatter plot in Fig.
Revised: To further investigate the correlation between the GDoF and the generalization gap, we represent the same data in a scatter plot in Fig.

Original: ( 1 ), which reveals that the generalization gap increased monotonically with the GDoF, as predicted bythe classical theory of computational complexity (Schmidhuber, 1997).
Revised: ( 1 ), which reveals that the generalization gap increased monotonically with the GDoF, as predicted by classical notions of program complexity (Schmidhuber, 1997).

Original: To quantify the correlation and verify our observation, we compute Spearman’s rank correlation as r (36) = .
Revised: To quantify the correlation and verify our observation, we compute Spearman’s rank correlation, giving r (36) = .

Original: 99 , p = 2 .
Revised: 99 , p = 2 .

Original: 5 × 10 − 11  and Pearson’s correlation
Revised: 5 × 10 − 11 , and Pearson’s correlation

Original: [Figure]
Revised: [Figure]

Original: coefﬁcient as r (36) = .
Revised: coefﬁcient, giving r (36) = .

Original: 88 , p = 9 .
Revised: 88 , p = 9 .

Original: 9 × 10 − 11 .
Revised: 9 × 10 − 11 .

Original: For each capacity multiplier, we  expand the whole training curves in Fig.
Revised: For each capacity multiplier, we also expand the whole training curves into Fig.

Original: ( 3 ) to show the evolution ofall the  quantities measured.
Revised: ( 3 ) to depict the evolution of the measured quantities over training.

Original: The early plateau of the  generalization gapfor larger models (darker curves) suggests the improved generalization for model sizes past the interpolation threshold asoccurred towards the tail of the “double descent” (Belkin et al., 2019).
Revised: We again observe an increase and then decrease in the ﬁnal generalization gap as the model size increases (curve color darkens), and this trend is reﬂected in the GDoF: An intermediate capacity multiplier has highest ﬁnal generalization gap and highest

Original: This behavior is reﬂected in the low-down (though not plateau) of  the  GDoF for larger models.
Revised: GDoF. Further, the dynamics of the generalization gap and GDoF match over the course of training: For example, the full model (darkest curve) increases in generalization gap as well as GDoF most quickly, but plateaus at intermediate values.

Original: We also explore how the GDoF reacts when model complexity is  controlled.
Revised: We also explore how the GDoF reacts when model complexity is explicitly controlled.

Original: We measure GDoF of the same set of models as in Fig.
Revised: We measure GDoF of the same set of models as in Fig.

Original: ( 2 ), here trained with the subspace method of Li et al.
Revised: ( 2 ), here trained with the subspace method of Li et al.

Original: (2018), which was originally proposed as an ingredient of another method to approximate the intrinsic dimensionality of a model.
Revised: (2018), which was originally proposed as an ingredient of another method to approximate the intrinsic dimensionality of a model.

Original: In particular, following
Revised: In particular, following

Original: Li et al.
Revised: Li et al.

Original: (2018), we randomly project model parameters to a random, lower-dimensional subspace θ S =
Revised: (2018), we randomly project model parameters to a random, lower-dimensional subspace θ S =

Original: R θ , where R is a random matrix ﬁxed at the start of training, and optimize the lower-dimensional vector θ S instead of θ to train the model.
Revised: R θ , where R is a random matrix ﬁxed at the start of training, and optimize the lower-dimensional vector θ S instead of θ to train the model.

Original: Intuitively, the model complexity is constrained by the coupling introduced by the random projection.
Revised: Intuitively, the model complexity is constrained by the coupling introduced by the random projection.

Original: We choose as the base model the CNN with the highest capacity (ﬁrst layer ﬁlter size 64 , ≈ 1 .
Revised: We choose as the base model the CNN with the highest capacity (ﬁrst layer ﬁlter size 64 , ≈ 1 .

Original: 55 million parameters).
Revised: 55 million parameters).

Original: We then train this base mode using subspace dimensions of{ 1 , 2 , 3 , 5 , 7 , 10 , 20 , 30 , 50 , 70 , 90 } × 10 4 . Fig. ( 4 ) demonstrates that the GDoF still tracks the generalization gap when complexity is controlled with subspace optimization, and Fig. ( 5 ) and Fig.
Revised: We then train this base mode using subspace dimensions of

Original: ( 6 ) further show that a suitable space

Original: 0 .0 .t
Revised: 0 .0 .t

Original: r a i n i ng e rr o r 0 .0 .e
Revised: r a i n i ng e rr o r 0 .0 .e

Original: v a l u a ti on e rr o r 0 .0 .g
Revised: v a l u a ti on e rr o r 0 .0 .g

Original: e n e r a li z a ti on g a p    training iteration × 

Revised: e n e r a li z a ti on g a p    training iteration ×

Original: GD o
Revised: GD o

Revised: F Φ 10101010c a p a c it y m u lti p li e r

Revised: { 1 , 2 , 3 , 5 , 7 , 10 , 20 , 30 , 50 , 70 , 90 } × 10 4 .

Revised: Fig.

Revised: ( 4 ) demonstrates that the GDoF still tracks the generalization gap when complexity is controlled with subspace optimization, and Fig.

Original: F Φ 10101010c a p a c it y m u lti p li e r dimension can improve performance by reducing the generalization gap; that is, intermediate subspace dimensions can lower the  evaluation error below that of the full model.
Revised: ( 5 ) and Fig. ( 6 ) further shows that a suitable space dimension can improve performance by reducing the generalization gap; that is, intermediate subspace dimensions can lower the ﬁnal evaluation error below that of the full model.

Original: 5. Related Work
Revised: 5. Related Work

Original: Whence generalization?
Revised: Whence generalization?

Original: Generalization properties of deep neural networks has receivedmuch renewed interest from both theoretical and empirical perspectives (Jiang et al., 2020a).
Revised: Generalization properties of deep neural networks have received renewed interest from both theoretical and empirical perspectives (Jiang et al., 2021).

Original: However, it seems that many properties that correlate with generalization performance are neither necessary nor sufﬁcient to ensure it; for example, sharpness of the loss surface around a minimum (Dinh et al., 2017), or monotonic linear interpolation (Lucas et al., 2021).Loukas et al.
Revised: However, it seems that many properties that correlate with generalization performance are neither necessary nor sufﬁcient to ensure it; for example, sharpness of the loss surface around a minimum (Dinh et al., 2017), or monotonic linear interpolation (Lucas et al., 2021).

Original: (2021) explore the Benevolent Training Hypothesis and provides evidence that model complexity, as measured by Lipschitz constant in their work, can be deduced from training dynamics.

Original: They further provided a generalization bound depends on both data and training.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) discuss the results of a recent community competition(Jiang et al., 2020a) that targeted complexity measures that reliably capture generalization performance; however, they do not investigate the trend ofthe these complexity measures as the parameter dimension is manipulated.
Revised: (2021) discuss the results of a recent community competition that targeted complexity measures that reliably capture generalization performance; however, they do not investigate the trend of these complexity measures as the parameter dimension is manipulated.

Original: Maddox et al.
Revised: Maddox et al.

Original: (2020) resolve the double descent paradox with the introduc0 .0 .e
Revised: 0 .0 .e

Original: rr o r training evaluation 0 .0 .g
Revised: rr o r training evaluation 0 .0 .g

Original: e n e r a li z a ti on g a p        capacity multiplier × 

Revised: e n e r a li z a ti on g a p        capacity multiplier ×

Original: GD o
Revised: GD o

Original: F Φ tion of a capacity estimate inspired by posterior contraction in Bayesian learning that renders evaluation performance monotonically increasing.
Revised: F Φ (2020) resolve the double descent paradox with the introduction of a capacity estimate inspired by posterior contraction in Bayesian learning that renders evaluation performance monotonically increasing.

Original: Capturing complexity.
Revised: Capturing complexity.

Original: Proposals to measure effective dimensionality have taken many forms, from more classical parameter and representation norms (Bartlett et al., 2017) and sharpness measures (Keskar et al., 2017), to measures motivated speciﬁcally by modern deep learning systems, such as measures of gradient noise (Smith & Le, 2018) and optimization speed (Hardt et al., 2016); see Jiang et al.
Revised: Proposals to measure effective dimensionality have taken many forms, from more classical parameter and representation norms (Bartlett et al., 2017) and sharpness measures (Keskar et al., 2017), to measures motivated speciﬁcally by modern deep learning systems, such as measures of gradient noise (Smith & Le, 2018) and optimization speed (Hardt et al., 2016); see Jiang et al.

Original: (2020b) for a thorough overview and an empirical comparison.
Revised: (2020) for a thorough overview and an empirical comparison.

Original: The common idea behind all such approaches is to quantify variability and thus complexity in the function represented by a neural network.
Revised: The common idea behind all such approaches is to quantify variability and thus complexity in the function represented by a neural network.

Original: This functional complexity may result from the interaction of the particular parameterization or architecture in addition to all other aspects of the machine learning pipeline, including the optimization algorithm and other hyper-parameters.
Revised: This functional complexity may result from the interaction of the particular parameterization or architecture in addition to all other aspects of the machine learning pipeline, including the optimization algorithm and other hyper-parameters.

Original: Crucially, however, the exact relationship between neural network design decisions and functional complexity is unknown, prompting the study of implicit regularization in deep learning (Neyshabur et al., 2015; Neyshabur, 2017; Dherin et al., 2021).
Revised: Crucially, however, the exact relationship between neural network design decisions and functional complexity is unknown, prompting the study of implicit regularization in deep learning (Neyshabur et al., 2015; Neyshabur, 2017; Dherin et al., 2021).

Original: Finding neural networks with minimal complexity
Revised: Finding neural networks with minimal complexity.

Original: One class of methods directly search models as programs with low Kolmogorov complexity.
Revised: One class of methods directly searches models as programs with low Kolmogorov complexity.

Original: For example, Schmidhuber (1997) proposes a probabilistic search algorithm to
Revised: For example, Schmid-

Original: 0 .0 .t
Revised: 0 .0 .t

Original: r a i n i ng e rr o r 0 .0 .e
Revised: r a i n i ng e rr o r 0 .0 .e

Original: v a l u a ti on e rr o r 0 .0 .g
Revised: v a l u a ti on e rr o r 0 .0 .g

Original: e n e r a li z a ti on g a p    training iteration × 

Revised: e n e r a li z a ti on g a p    training iteration ×

Original: GD o
Revised: GD o

Original: F Φ 101010ca p a c it y m u lti p li e
Revised: F Φ 101010ca p a c it y m u lti p li e

Original: r full model  discover neural networks with low Kolmogorov complexity, although it has only been demonstrated with toy problems that can be solved with relatively short programs consist of limited primitives.
Revised: r full model huber (1997) proposes a probabilistic search algorithm to discover neural networks with low Kolmogorov complexity, although it has only been demonstrated with toy problems that can be solved with relatively short programs consisting of limited primitives.

Original: Another class of methods encourages optimizers to ﬁnd simpler neural networks by penalize deviation from a speciﬁed prior of the weights.
Revised: Another class of methods encourages optimizers to ﬁnd simpler neural networks by penalizing deviation from a speciﬁed prior over the weights.

Original: The simplest example is “weight-decay” (Hinton, 1987), which can be interpreted as ﬁnding the maximum a posteriori (MAP) solution given an isotropic Gaussian prior over weights.
Revised: The simplest example is “weight-decay” (Hinton, 1987), which can be interpreted as ﬁnding the maximum a posteriori (MAP) solution given an isotropic Gaussian prior over weights.

Original: Nowlan & Hinton (1992) elaborates this idea by introducing  mixture of Gaussian priors over  weights, but still assumes that all weights are quantized with the same tolerance of noise.
Revised: Nowlan & Hinton (1992) elaborate this idea by introducing a mixture of Gaussians prior over the weights, but still assume that all weights are quantized with the same noise tolerance.

Original: This assumption has been drop in (Hinton & Van Camp, 1993), which employs the variational method to approximate the posterior distribution of noisy weights with different precisions.
Revised: Hinton & Van Camp (1993) remove this assumption by employing a variational method to approximate the posterior distribution of noisy weights with different precisions.

Original: Bayesian model selection and Occam’s Razor.
Revised: Bayesian model selection and Occam’s Razor.

Original: MacKay (1992a) reviews how model selection or comparison can be achieved in a Bayesian context by comparing the evidence for each model.
Revised: MacKay (1992a) reviews how model selection or comparison can be achieved in a Bayesian context by comparing the evidence for each model.

Original: In particular, given a parameterized model family H that speciﬁes a prior distribution over parameters w , P ( w | H ) and a predictive distribution over data 0 .0 .t
Revised: In particular, given a parameterized model family H that speciﬁes a prior distribution over parameters w , P ( w | H ) and a predictive distribution over data 0 .0 .t

Original: r a i n i ng e rr o r    training iteration × 100 .0 .e
Revised: r a i n i ng e rr o r    training iteration × 100 .0 .e

Original: v a l u a ti on e rr o r 101010c a p a c it y m u lti p li e
Revised: v a l u a ti on e rr o r 101010c a p a c it y m u lti p li e

Original: r full model
Revised: r full model

Original: D , P ( D | w , H ) , the evidence is P ( D | H ) =
Revised: D , P ( D | w , H ) , the evidence is P ( D | H ) =

Original: (cid:82) P ( D | w ,
Revised: (cid:82) P ( D | w ,

Original: H ) P ( w | H )
Revised: H ) P ( w | H )

Original: d w .
Revised: d w .

Original: As a rule of thumb, greater entropy in the prior term P ( w | H ) , corresponding to a more ﬂexible hypothesis class, results in a lesser value for the evidence, a regularizing effect known as Occam’s Razor .
Revised: As a rule of thumb, greater entropy in the prior term P ( w | H ) , corresponding to a more ﬂexible hypothesis class, results in a lesser value for the evidence, a regularizing effect known as Occam’s Razor .

Original: It should be noted that the behavior of Occam’s Razor—in particular what notion of ﬂexibility is penalized—implicitly relies on the model parameterization and is therefore not an a priori effect (Wolpert, 1995); for instance, MacKay (1992a) discuss Occam’s Razor and Bayesian model comparison for models that admit Gaussian approximation and quadratic regularizers, and MacKay (1992b) study such approximations in the context of neural networks.
Revised: It should be noted that the behavior of Occam’s Razor—in particular, what notion of ﬂexibility is penalized—implicitly relies on the model parameterization and is therefore not an a priori effect (Wolpert, 1995); for instance, MacKay (1992a) discuss Occam’s Razor and Bayesian model comparison for models that admit Gaussian approximation and quadratic regularizers, and MacKay (1992b) study such approximations in the context of neural networks.

Original: Nevertheless, Bayesian model selection allows coherent comparison of appropriately related prior distributions (Jefferys & Berger, 1992), and robustness with respect to the selection of the prior distribution can be ensured in some situations (Berger et al., 1994).
Revised: Nevertheless, Bayesian model selection allows coherent comparison of appropriately related prior distributions (Jefferys & Berger, 1992), and robustness with respect to the selection of the prior distribution can be ensured in some situations (Berger et al., 1994).

Original: Implicit biases in model design.
Revised: Implicit biases in model design.

Original: Lineages of machine learning models are often the result of selection on the basis of performance on standardized benchmarks (Dehghani et al., 2021).
Revised: Lineages of machine learning models are often the result of selection on the basis of performance on standardized benchmarks (Dehghani et al., 2021).

Original: As a result, such lineages implicitly depend on the types of data to which they have been applied—in particular, such models are likely to have implicit biases that aid performance on such datasets.
Revised: As a result, such lineages implicitly depend on the types of data to which they have been applied—in particular, such models are likely to have implicit biases that aid performance on such datasets.

Original: Analogously, prior selection in Bayesian analysis often implicitly relies on data (Gelman et al., 2017).
Revised: Analogously, prior selection in Bayesian analysis often implicitly relies on data (Gelman et al., 2017).

Original: From a Bayesian perspective, the impact of prior information can be quantiﬁed using sensitivity analysis (e.g., M¨uller 2012) or with measures of prior-data and prior-likelihood conﬂict (Bousquet, 2008; Nott et al., 2020;
Revised: From a Bayesian perspective, the impact of prior information can be quantiﬁed using sensitivity analysis (e.g., M¨uller 2012) or with measures of prior-data and prior-likelihood conﬂict (Bousquet, 2008; Nott et al., 2020;

Original: Reimherr et al., 2021).
Revised: Reimherr et al., 2021).

Original: 6. Conclusion
Revised: 6. Conclusion

Original: In this work, we investigated a complexity measure inspired by the classical notion of degrees of freedom in statistics.
Revised: In this work, we investigated a complexity measure inspired by the classical notion of degrees of freedom in statistics.

Original: This measure  is scalable and is a function of the entire machine learning training pipeline.
Revised: the variant of this measure that we propose is scalable and is a function of the entire machine learning training pipeline.

Original: We demonstrated that this measure strongly correlates with generalization performance in the double-descent regime, and that directly controlling model complexity  via subspace training improves can improve generalization.
Revised: We demonstrated that this measure strongly correlates with generalization performance in the double-descent regime, and that, when model complexity is controlled via subspace training, the resulting improvement in the generalization gap is tracked by this measure.