Original: M ULTI - SCALE F EATURE L EARNING D YNAMICS : I NSIGHTS FOR D OUBLE D ESCENT
Revised: M ULTI - SCALE F EATURE L EARNING D YNAMICS : I NSIGHTS FOR D OUBLE D ESCENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the highdimensional interactions between the large number of network parameters.
Revised: A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the highdimensional interactions between the large number of network parameters.

Original: Such non-trivial interactions lead to intriguing model behaviors such as the phenomenon of “double descent” of the generalization error.
Revised: Such non-trivial dynamics lead to intriguing model behaviors such as the phenomenon of “double descent” of the generalization error.

Original: The more commonly studied aspect of this phenomenon corresponds to model-wise double descent where the test error exhibits a second descent with increasing model complexity, beyond the classical U-shaped error curve.
Revised: The more commonly studied aspect of this phenomenon corresponds to model-wise double descent where the test error exhibits a second descent with increasing model complexity, beyond the classical U-shaped error curve.

Original: In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two non-monotonous transitions, or descents as the training time increases.
Revised: In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two nonmonotonous transitions, or descents as the training time increases.

Original: By leveraging tools fromrandom matrix theory and statistical physics, we study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks.
Revised: By leveraging tools from statistical physics, we study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks.

Original: In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training.
Revised: In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training.

Original: We ﬁnd that double descent can be attributed to distinct features being learned at different scales: as fast-learning features overﬁt, slower-learning features start to ﬁt, resulting in a second descent in test error.
Revised: We ﬁnd that double descent can be attributed to distinct features being learned at different scales: as fast-learning features overﬁt, slowerlearning features start to ﬁt, resulting in a second descent in test error.

Original: We validate our ﬁndings through numerical experiments where our theory accurately predicts empirical ﬁndings and remains consistent with observations in deep neural networks.
Revised: We validate our ﬁndings through numerical experiments where our theory accurately predicts empirical ﬁndings and remains consistent with observations in deep neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Classical wisdom in statistical learning theory predicts a trade-off between the generalization ability of a machine learning model and its complexity, with highly complex models less likely to generalize well (Friedman et al., 2001).
Revised: Classical wisdom in statistical learning theory predicts a trade-off between the generalization ability of a machine learning model and its complexity, with highly complex models less likely to generalize well (Friedman et al., 2001).

Original: If the number of parameters measures complexity, deep learning models sometimes go against this prediction (Zhang et al., 2016): deep neural networks trained by stochastic gradient descent exhibit a so-called double descent behavior (Belkin et al., 2019b) with increasing model parameters.
Revised: If the number of parameters measures complexity, deep learning models sometimes go against this prediction (Zhang et al., 2016): deep neural networks trained by stochastic gradient descent exhibit a so-called double descent behavior (Belkin et al., 2019b) with increasing model parameters.

Original: Speciﬁcally, with increasing complexity, the generalization error
Revised: Speciﬁcally, with increasing complexity, the generalization error

Original: ﬁrst obeys the classical U-shaped curve consistent with statistical learning theory.
Revised: ﬁrst obeys the classical U-shaped curve consistent with statistical learning theory.

Original: However, a second regime emerges as the number of parameters is further increased past a transition threshold where generalization error drops again, hence the “double descent” or more accurately model-wise double descent (Nakkiran et al., 2019).
Revised: However, a second regime emerges as the number of parameters is further increased past a transition threshold where generalization error drops again, hence the “double descent” or more accurately model-wise double descent (Nakkiran et al., 2019).

Original: Nakkiran et al.
Revised: Nakkiran et al.

Original: (2019) showed that the phenomenon of double descent is not limited to varying model size but is also observed as a function of training time or epochs.
Revised: (2019) showed that the phenomenon of double descent is not limited to varying model size but is also observed as a function of training time or epochs.

Original: In this case as well, the so-called epoch-wise double descent is in apparent contradiction with the classical understanding of over-ﬁtting (Vapnik, 1998), where one expects that longer training of a sufﬁciently large model beyond a certain threshold should result in over-ﬁtting.
Revised: In this case as well, the so-called epoch-wise double descent is in apparent contradiction with the classical understanding of over-ﬁtting (Vapnik, 1998), where one expects that longer training of a sufﬁciently large model beyond a certain threshold should result in over-ﬁtting.

Original: This has important implications for practitioners and raises questions about one of the most widely used regularization method in deep learning (Goodfellow et al., 2016): early stopping.
Revised: This has important implications for practitioners and raises questions about one of the most widely used regularization method in deep learning (Goodfellow et al., 2016): early stopping.

Original: Indeed, while one might expect early stopping to prevent over-ﬁtting, it might in fact prevent models from being trained at their fullest potential.
Revised: Indeed, while one might expect early stopping to prevent over-ﬁtting, it might in fact prevent models from being trained at their fullest potential.

Original: Since the 1990s, there has been much interest in understanding the origins of non-trivial generalization behaviors of neural networks (Opper, 1995; Opper & Kinzel, 1996).
Revised: Since the 1990s, there has been much interest in understanding the origins of non-trivial generalization behaviors of neural networks (Opper, 1995; Opper & Kinzel, 1996).

Original: The authors of Krogh & Hertz (1992b) were among the ﬁrst to provide theoretical explanations for (model-wise) double descent in linear models.
Revised: The authors of Krogh & Hertz (1992b) were among the ﬁrst to provide theoretical explanations for (model-wise) double descent in linear models.

Original: Summarily, at intermediate levels of complexity, where the model size is equal to the number of training examples, the model is very sensitive to noise in training data and 
Revised: Summarily, at intermediate levels of complexity, where the model size is equal to the number of training examples, the model is very sensitive to noise in training data and hence, generalizes poorly. This sensitivity to noise reduces if the model complexity is either de-

Original: [Table] 
Revised: [Table] [Table]

Revised: /uDiT9w==</latexit

Original: [Figure]
Revised: [Figure]

Original: hence, generalizes poorly. This sensitivity to noise reduces if the model complexity is either decreased or increased.
Revised: creased or increased.

Original: More recently, the double descent phenomena has been also studied for more complex models such as two-layer neural networks and random feature models (Ba et al., 2019; Mei & Montanari, 2019; D’Ascoli et al.,  2020).
Revised: More recently, the double descent phenomena has been also studied for more complex models such as two-layer neural networks and random feature models (Ba et al., 2019; Mei & Montanari, 2019; D’Ascoli et al., 2020; Gerace et al., 2020).

Original: The majority of previous work in this direction focuses on understanding the asymptotic behavior of model performance, i.e., where training time t → ∞ .
Revised: The majority of previous work in this direction focuses on understanding the asymptotic behavior of model performance, i.e., where training time t → ∞ .

Original: In recent years, there has been an interest in studying the non-asymptotic (ﬁnite training time) performance, suggesting that several intriguing properties of neural networks can be attributed to different features being learned at different scales.
Revised: In recent years, there has been an interest in studying the non-asymptotic (ﬁnite training time) performance, suggesting that several intriguing properties of neural networks can be attributed to different features being learned at different scales.

Original: Among  limited work studying the particular epoch-wise double descent, Nakkiran et al.
Revised: Among the limited work studying the particular epoch-wise double descent, Nakkiran et al.

Original: (2019) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence uniﬁes both model-wise and epoch-wise double descent.
Revised: (2019) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence uniﬁes both model-wise and epoch-wise double descent.

Original: In an empirical study Heckel & Yilmaz (2020), it is found that thetime evolution dynamics show twoor more overlapping, U-shape bias/variance curves with different minima times, thus leading to non-monotonic test error curves.
Revised: Through a combination of theory and empirical results, Heckel & Yilmaz (2020) ﬁnd that the dynamics of evolution of single and two layer networks under gradient descent, can be perceived to be the superposition of two bias/variance curves with different minima times, thus leading to non-monotonic test error curves.

Original: In this work, we build on (B¨os et al., 1993; B¨os, 1998; Advani & Saxe, 2017; Mei & Montanari, 2019) which analyze model-wise double descent through the lens of linear models, and probe the origins of epoch-wise double descent.
Revised: In this work, we build on B¨os et al. ; B¨os (1998); Advani & Saxe (2017); Mei & Montanari (2019) which analyze model-wise double descent through the lens of linear models, to probe the origins of epoch-wise double descent.

Original: Our main contributions are:
Revised: Particularly,

Original: • In Section 2.2, we introduce a linear teacher-student model which, despite its simplicity, exhibits some of intriguing properties of generalization dynamics in deep neural networks.
Revised: • We introduce a linear teacher-student model which, despite its simplicity, exhibits some of intriguing properties of generalization dynamics in deep neural networks.

Original:  • In the limit of high dimensions, we leverage the replica method developed in statistical physics to derive closed-form expressions for the generalization dynamics of our teacherstudent setup, as a function of training time and regularization strength (Section 2.3).
Revised: (Section 2.1) • In the limit of high dimensions, we leverage the replica method developed in statistical physics to derive closed-form expressions for the generalization dynamics of our teacherstudent setup, as a function of training time and regularization strength.

Original:  • Consistent with recent ﬁndings, we provide an explanation for the existence of epoch-wise double descent through the lens of multi-scale feature learning.
Revised: (Section 2.2) • Consistent with recent ﬁndings, we provide an explanation for the existence of epoch-wise double descent through the lens of multi-scale feature learning.

Original: (Figure 1) •
Revised: (Figure 1) •

Original: We perform simulation experiments to validate our analytical predictions.
Revised: We perform simulation experiments to validate our analytical predictions.

Original: We also conduct experiments with deep networks on CIFAR-10, showing that our teacher-student setup exhibits generalization behavior which is qualitatively similar to that of large nonlinear networks.
Revised: We also conduct experiments with deep networks, showing that our teacher-student setup exhibits generalization behavior which is qualitatively similar to that of deep networks.

Original: (Figure 2)
Revised: (Figure 2)

Original:  A NALYTICAL R ESULTS2. P RELUDE
Revised:  A NALYTICAL R ESULTS

Original: Before diving into the theory, we invite the reader to recall a simple equation from thermodynamics.

Original: Consider an ideal gas in a container with its large number of molecules moving around, colliding with each other, all while obeying Newton’s laws.

Original: While the exact dynamics of each of such molecules is intractable, the system’s macroscopic behavior can be characterized in terms of a handful of scalar quantities, namely, the pressure P , the volume V , and the temperature T .

Original: By averaging over suitable probability measures and applying the principle of free-energy minimization, one arrives at a remarkably simple relationship between these three macroscopic variables, i.e., the well-known PV = nRT ( n : number of moles of gas, R : gas constant) (Reif, 2009).

Original: The same principle can be ported to neural networks.

Original: Stochastic Gradient Descent (SGD) — the de facto optimization algorithm for neural networks — exhibits complex dynamics arising fromthe interactions between a large number of parameters (Kunin et al., 2020).
Revised: Stochastic Gradient Descent (SGD) — the de facto optimization algorithm for neural networks — exhibits complex dynamics arising from a large number of parameters (Kunin et al., 2020).

Original: However, it is possible to describe some aspects of the high-dimensional microscopic dynamics of neural networks in terms of low-dimensional understandable macroscopic entities.
Revised: However, it is possible to describe some aspects of the high-dimensional microscopic dynamics of neural networks in terms of low-dimensional understandable macroscopic entities.

Original: In a series of seminal papers by Gardner (Gardner, 1988; Gardner & Derrida, 1988; 1989), the replica method of statistical physics was adopted to derive expressions describing the generalization behavior of large linear models trained using SGD.
Revised: In a series of seminal papers by Gardner (Gardner, 1988; Gardner & Derrida, 1988; 1989), the replica method of statistical physics was adopted to derive expressions describing the generalization behavior of large linear models trained using SGD.

Original: In this paper, we employ Gardner’s analysis to build upon an established line of work studying linear and generalized linear models (Seung et al., 1992; Kabashima et al., 2009; Krzakala et al., 2012).
Revised: In this paper, we employ Gardner’s analysis to build upon an established line of work studying linear and generalized linear models (Seung et al., 1992; Kabashima et al., 2009; Krzakala et al., 2012).

Original: While most of previous work study the asymptotic  generalization behavior in the limit of large training time, we adapt these methods to study transient learning dynamics of generalization.
Revised: While most of previous work study the asymptotic ( t → ∞ ) generalization behavior, we adapt these methods to study transient learning dynamics of generalization for ﬁnite training time.

Original: We study asimpliﬁed teacher-student model that exhibits key characteristics of modern neural networks.
Revised: In the following, we ﬁrst introduce a teacher-student model that exhibits interesting characteristics of modern neural networks.

Revised: We then adapt the replica method to study the generalization performance as a function of training time and amount of regularization.

Original: 2. A T EACHER -S TUDENT S ETUP
Revised: 2. A T EACHER -S TUDENT S ETUP

Original: Teacher: We study a supervised linear regression problem in which the training labels y , are generated by a noisy linear model (Figure 1),
Revised: Teacher: We study a supervised linear regression problem in which the training labels y , are generated by a noisy linear model (Figure 1),

Original: [Equation]
Revised: [Equation]

Original: where z ∈ R d is the teacher’s input and y ∗ , y ∈ R are the teacher’s noiseless and noisy outputs, respectively.
Revised: where z ∈ R d is the teacher’s input and y ∗ , y ∈ R are the teacher’s noiseless and noisy outputs, respectively.

Original: W ∈ R d represents the (ﬁxed) weights of the teacher and
Revised: W ∈ R d represents the (ﬁxed) weights of the teacher and

Original: (cid:15) ∈ R is the noise.
Revised: (cid:15) ∈ R is the noise.

Original: Both
Revised: Both

Original: W and (cid:15) are drawn i.i.d.
Revised: W and (cid:15) are drawn i.i.d.

Original: from Gaussian distributions with zero mean and variances of 1 and σ 2 (cid:15) , respectively.
Revised: from Gaussian distributions with zero means and variances of 1 and σ 2 (cid:15) , respectively.

Original: Student: A student model is correspondingly chosen to be a similar shallow network with trainable weights ˆ W ∈ R d .
Revised: Student: A student model is correspondingly chosen to be a similar shallow network with trainable weights ˆ W ∈ R d .

Original: The student model is trained on n training pairs { ( x µ , y µ ) } nµ =1 , with the labels y µ being generated by the above teacher network, as,
Revised: The student model is trained on n training pairs { ( x µ , y µ ) } nµ =1 , with the labels y µ being generated by the above teacher network, as,

Original: [Equation]
Revised: [Equation]

Original: where the matrix F ∈ R d × d is a predeﬁned and ﬁxed modulation matrix regulating the student’s access to the true input z .
Revised: where the matrix F ∈ R d × d is a predeﬁned and ﬁxed modulation matrix regulating the student’s access to the true input z .

Original: One can think of z as the latent factors of variation on which the teacher operates, while x can be thought as the pixels that the student learns from.
Revised: One can think of z as the latent factors of variation on which the teacher operates, while x can be thought as the pixels that the student learns from.

Original: Learning paradigm: To train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,
Revised: Learning paradigm: To train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ [0 , ∞ ) is the regularization coefﬁcient.
Revised: where λ ∈ [0 , ∞ ) is the regularization coefﬁcient.

Original: Optimizing Eq.
Revised: Optimizing Eq.

Original: 3 with stochastic gradient descent (SGD) yields the following update rule,
Revised: 3 with stochastic gradient descent (SGD) yields the typical update rule,

Original: [Equation]
Revised: [Equation]

Original: in which t denotes the training step and η is the learning rate.
Revised: in which t denotes the training step and η is the learning rate.

Original: Additionally, ξ ∈ R d models the stochasticity noise of the optimization.
Revised: Additionally, ξ ∼ N (0 , σ 2 ξ ) models the stochasticity noise of the optimization algorithm (Bottou et al., 1991).

Original: ξ ∼ N (0 , 2 β I ) is assumed be Gaussian with zero mean and a covariance of 2 β I where β in thermodynamic terminology is referred to as the inverse temperature.

Original: Macroscopic variables: The quantity of interest in this work, is the expected generalization error of the student, determined by averaging the student’s error over all possible input-target pairs and noise realizations, as,
Revised: Macroscopic variables: The quantity of interest in this work, is the expected generalization error of the student, determined by averaging the student’s error over all possible input-target pairs and noise realizations, as,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: As shown in Stephenson & Lee (2021), if n, d → ∞ with a constant ratio nd  , Eq.
Revised: As shown in B¨os et al. (1993), if n, d → ∞ with a constant ratio nd < ∞ , Eq.

Original: 5 can be written as a function of scalar macroscopic  variables R, Q ∈ R ,
Revised: 5 can be written as a function of two macroscopic scalar variables R, Q ∈ R ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where σ 2 (cid:15) is the variance of the teacher’s output noise and,
Revised: where σ 2 (cid:15) is the variance of the teacher’s output noise and,

Original: [Equation]
Revised: [Equation]

Original:  Both R and Q have clear interpretations; R is the dot-product between the teacher
Revised: See App. B.1 for the proof. Both R and Q have clear interpretations; R is the dot-product between the teacher

Original: ’s weights W and the student’s modulated weights F ˆ W , hence can be interpreted as the alignment between the teacher and the student .
Revised: ’s weights W and the student’s modulated weights F ˆ W , hence can be interpreted as the alignment between the teacher and the student .

Original: Similarly, Q can be interpreted as the student’s modulated norm .
Revised: Similarly, Q can be interpreted as the student’s modulated norm .

Original: The negative sign of R in Eq.
Revised: The negative sign of R in Eq.

Original: 6 suggests that the larger R is, the smaller the generalization error gets.
Revised: 6 suggests that the larger R is, the smaller the generalization error gets.

Original: At the same time, Q appears with a positive sign suggesting the students witha smaller (modulated) norm generalize better.
Revised: At the same time, Q appears with a positive sign suggesting the students with smaller (modulated) norm generalize better.

Original: As  remark, note that both R and Q are functions of ˆ W which itself is a function of training time t and the regularization strength λ .
Revised: As a remark, note that both R and Q are functions of ˆ W which itself is a function of training iteration t and the regularization strength λ .

Original: Therefore, hereafter, we denote the above quantities as L G ( t, λ ) ,
Revised: Therefore, hereafter, we denote the above quantities as L G ( t, λ ) ,

Original: R ( t, λ ) , and Q ( t, λ ) .
Revised: R ( t, λ ) , and Q ( t, λ ) .

Original: 2. M AIN R ESULT
Revised: 2. M AIN R ESULTS

Revised: In this Section, we present our main analytical results, with Section 2.3 containing a sketch of our derivations.

Revised: For brevity of the results, here, we only present the results for σ 2 (cid:15) =

Revised: λ

Original:  General case.
Revised: = 0 . See App. B for the general case and the detailed proofs. General matrix F .

Original: Let Z, X  ∈ R n × d denote the input matrices for the teacher and student, where X := 
Revised: Let Z := [ z µ ] nµ =1 ∈ R n × d and X := [ x µ ] nµ =1 ∈ R n × d denote the input matrices for the teacher and student such that X := ZF .

Original: F T Z . For a general modulation matrix F , thediagonal matrix Λ contains the eigenvalues of the student’s input covariance matrix,
Revised: For a general modulation matrix F , the input covariance matrix has the following singular value decomposition (SVD),

Original: [Equation]
Revised: [Equation]

Original: where in  the last step we have performed a singular value decomposition (SVD).
Revised: in which the diagonal matrix Λ contains the eigenvalues of the student’s input covariance matrix.

Original: Solving the dynamics of gradient descent as in Eq.
Revised: Solving the dynamics of gradient descent as in Eq.

Original: 4, we arrive at the following exact analytical expressions for R ( t, λ ) and Q ( t, λ ) , 
Revised: 4, we arrive at the following exact analytical expressions for R ( t ) and Q ( t ) , [Equation] where,

Original: [Equation]
Revised: [Equation]

Revised: in which Tr ( . ) is the trace operator.

Revised: See App.

Revised: B.2 the proof.

Revised: Remark: The solution in Eqs.

Revised: 9 and 10 are exact, however, they require the empirical computation of the eigenvalues Λ .

Revised: Below, we treat a special case of the dynamics that allow us to derive approximate solutions that do not explicitly depend on Λ .

Original: Special case:Slow and Fast  features.
Revised: Special case: Fast and slow features.

Original: For interpretability, we now study a case where the modulation matrix F has a speciﬁc structure described in Assumption 1. 
Revised: We now study a case where the modulation matrix F has a speciﬁc structure described in Assumption 1. Assumption 1.

Original: Assumption 1 The modulation matrix F with a singular value decomposition F := U Σ V T has two sets of singular values such that the ﬁrst p singular values are equal to σ 1 and the remaining d − p singular values are equal to σ 2 .
Revised: The modulation matrix, F , under a SVD, F := U Σ V T has two sets of singular values such that the ﬁrst p singular values are equal to σ 1 and the remaining d − p singular values are equal to σ 2 .

Original: We let the condition number of F to be denoted by κ :=
Revised: We let the condition number of F to be denoted by κ :=

Original: σ 1 σ 2 > 1 .
Revised: σ 1 σ 2 > 1 .

Original: By employing the replica method of statistical physics (Gardner, 1988; Gardner & Derrida, 1988), we now derive approximate expressions for R ( t, λ ) .
Revised: By employing the replica method of statistical physics (Gardner, 1988; Gardner & Derrida, 1988), we now derive approximate expressions for R ( t ) and Q ( t ) .

Original: We ﬁrst deﬁne the following auxiliary variables,
Revised: To begin with, we ﬁrst deﬁne the following auxiliary variables,

Original: [Equation]
Revised: [Equation]

Original: and also let,
Revised: and also let,

Original: [Equation]
Revised: [Equation]

Original: The closed-from scalar expression for R ( t, λ ) is then given by,
Revised: The closed-from scalar expression for R ( t ) is then given by,

Original: [Equation]
Revised: [Equation]

Original: where ,
Revised: where ,

Original: R 1 := n

Original: [Equation]
Revised: [Equation]

Original: , and ,
Revised: and ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For Q ( t, λ ) , we accordingly deﬁne two more auxiliary variables,
Revised: For Q ( t ) , we accordingly deﬁne two more auxiliary variables,

Original: [Equation]
Revised: [Equation]

Original: fori ∈
Revised: for

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with which the closed-from scalar expression for Q ( t, λ ) reads,
Revised: with which the closed-from scalar expression for Q ( t ) reads,

Original: [Equation]
Revised: [Equation]

Original: where ,
Revised: where ,

Original: [Equation]
Revised: [Equation]

Original: and ,
Revised: and ,

Original: [Equation]
Revised: [Equation]

Original: By plugging Eqs.
Revised: By plugging Eqs.

Original: 12 and 14 into Eq.
Revised: 13 and 15 into Eq.

Original: 6, one obtains a closed-form expression for L G ( t, λ ) as a function of  training time and the regularization strength.
Revised: 6, one obtains a closed-form expression for L G ( t ) as a function of the training time. See App. B.3 for the proof. [Equation]

Revised: [Table]

Original: Note that the speed at which R 1 and R(similarly Q 1 and Q 2 ) evolve is a function of the condition number, κ :=
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: σ 1 σ 2 .
Revised: [Table] [Equation] [Equation] Remark: Eq.

Original: 2. S KETCH OF DERIVATIONS
Revised: 11 indicates that the singular values of F , are directly multiplied by t . That implies that the learning speed of each feature is scaled by the magnitude of its corresponding singular value. As an illustration, the ﬁgure on the right shows the evolution of R 1 , R 2 , and R = R 1 + R 2 for a case where p = d/ 2 , σ 1 = 1 , and σ 1 = 0 .

Revised: 01 implying a condition number of κ = 100 .

Original: In this Section, we sketch the key steps in the derivation of our main results.
Revised: In this Section, we sketch the key steps in the derivation of our main results.

Original: For the sake of simplicity, here we only treat the noiseless teacher.
Revised: For the sake of simplicity, here we only treat the case where σ (cid:15) = λ = 0 .

Original: Detailed proofsand further details are presented in Appendix A
Revised: The general case with detailed proofs are presented in App B.

Original: Exact dynamics of SGD.
Revised: Exact dynamics of SGD.

Original: Recall the gradient descent update rule in Eq. 4.
Revised: Recall the gradient descent update rule in Eq. 4.

Original: For the linear model deﬁned in Eqs.
Revised: For the linear model deﬁned in Eqs.

Original: 1-2, learning is governed by the following discrete-time dynamics,
Revised: 1-2, learning is governed by the following discrete-time dynamics,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: For simplicity, we furthermore assume the student to be initialized at the origin, ˆ W 0{+t =
+} =0 ,where the dynamics admit the following exact closed-form solution,
Revised: With the assumption that ˆ W {+t =
+} = , the dynamics admit the following exact closed-form solution,

Original: [Equation]
Revised: [Equation]

Original: With a SVD on X T X , Eqs.
Revised: With a SVD on X T X , Eqs.

Original: 9 can then be obtained by substituting ˆ W t in Eqs.
Revised: 9-10 can then be obtained by substituting ˆ W t in Eqs.

Original: As a remark, note that one can recover the results of Advani & Saxe (2017) by setting F → Iand η → 0 .
Revised: As a remark, note that one can recover the results of Advani & Saxe (2017) by setting F = I .

Original: In that case, the eigenvalues of X T X follow a Marchenko–Pastur distribution (Marchenko & Pastur, 1967).
Revised: In that case, the eigenvalues of X T X follow a Marchenko–Pastur distribution (Marchenko & Pastur, 1967).

Original: Induced probability density of SGD.
Revised: Induced probability density of SGD.

Original: It is well-known (Kuhn & Bos, 1993) that probability distribution of weight conﬁgurations for network weights ˆ W trained via SGD on a loss L ( ˆ W ) , tend to the Gibbs distribution such that,
Revised: It is well-known (Kuhn & Bos, 1993; Solla, 1995) that probability distribution of weight conﬁgurations for network weights ˆ W trained via SGD on a loss L ( ˆ W ) , tend to the Gibbs distribution such that,

Original: [Equation]
Revised: [Equation]

Original: (17) in which Z β is the partition function (cid:16)(cid:82)
Revised: in which Z β is the partition function (cid:16)(cid:82)

Original: d ˆ W exp( − β L ( ˆ W )) (cid:17) and β is called the inverse temperature of the thermodynamic system modeling our network setup.
Revised: d ˆ W exp( − β L ( ˆ W )) (cid:17) and β is called the inverse temperature and is inversely proportional the stochastic noise of SGD, ξ , deﬁned in Eq. 4.

Original: Intuitively, for small β , the distribution of P ( ˆ W ) is almost uniform, while as β → ∞ , P ( ˆ W ) becomes more concentrated around the minimum of the training loss.
Revised: Intuitively, for small β , the distribution of P ( ˆ W ) is almost uniform, while as β → ∞ , P ( ˆ W ) becomes more concentrated around the minimum of the training loss.

Original: It is important to highlight that Eq.
Revised: It is important to highlight that Eq.

Original: 17 describes the equilibrium distribution of the student network’s weights, i.e., at the end of training ( t → ∞ ).
Revised: 19 describes the equilibrium distribution of the student network’s weights, i.e., at the end of training ( t → ∞ ).

Original: However, we are interested in studying the trajectory of student’s weights during the course of training, i.e., for ﬁnite t .
Revised: However, we are interested in studying the trajectory of student’s weights during the course of training, i [Equation] e., for ﬁnite t .

Original: To that end,  the time-dependent probability density over ˆ W is given by,
Revised: To that end, we derive the timedependent probability density over ˆ W ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where,

Original: [Equation]
Revised: [Equation]

Original: where ˜ W ( t ) is deﬁned in Eq.{+[Equation] [Equation] R = R 1 + R 2 R 
 Rt+}
Revised: {+[Equation] [Equation] R = R 1 + R 2 R 
 Rt+}

Original: To further simplify P ( ˆ W, t ) , we note that ˜ L ( ˆ W, t ) in Eq.
Revised: SGD steps t th step on argmin [Equation] ˆ [Equation]

Original: 18 can be approximated by,

Original: [Equation]
Revised: [Table]

Original: The updated expression for ˜ L  ( ˆ W, t ) enables us to use the replica method for the ﬁnite time learning.
Revised: ˜ L T ( ˆ W, t ) is a modiﬁed loss such that its min- imum (equilibrium distribution) is achieved at the t th iterate of gradient descent on L ( ˆ W ) . The schematic diagram on the right illustrates this equivalence, such that, arg min ˆ W ˜ L T ( ˆ W, t ) =

Original: Free-Energy To determine the average training and generalization error attraining time t , one proceeds by ﬁrst computing the free-energy of the system as,
Revised: ˆ W t , where ˆ W t is the deﬁned in Eq. 4. The typical generalization error. To determine the typical generalization performance at time t , one proceeds by ﬁrst computing the free-energy of the system as,

Original: [Equation]
Revised: [Equation]

Original: and extremize the above entity w.r.t.
Revised: Free-energy is a self-averaging property where its typical/most probable value coincides with its average over proper probability distributions Engel & Van den Broeck (2001).

Original:  the macroscopic variables of interest, R, Q , to determine their average case estimates.
Revised: Therefore, to determine the typical values of R and Q , we extremize the free-energy w.r.t.

Original: The reason behind the evaluation of the free-energy f is that, unlike the (input and teacher averaged) partition function, its average and most probable values coincide.
Revised: those variables.

Original: Due to the logarithm inside the expectation, analytical computation of Eq.
Revised: Due to the logarithm inside the expectation, analytical computation of Eq.

Original: 20 is intractable.
Revised: 24 is intractable.

Original: However, the replica method (M ´ ezard et al., 1987) allows us to tackle this through the following identity,
Revised: However, the replica method (M´ezard et al., 1987) allows us to tackle this through the following identity,

Original: [Equation]
Revised: [Equation]

Original: Computation of the free-energy via this procedure and its subsequent extremization w.r.t R and Q , we arrive at Eqns.
Revised: Computation of the free-energy via replica method and its subsequent extremization w.r.t R and Q , we arrive at Eqs.

Original: 12 and 14.
Revised: 13 and 15.

Original:  To summarize, using the replica method, we are able to cast the high-dimensional dynamics of SGD into simple scalar equations governing R and Q and consequently, the generalization error L G .
Revised: See App. B.3 for more details. To summarize, using the replica method, we are able to cast the high-dimensional dynamics of SGD into simple scalar equations governing R and Q and, consequently, the generalization error L G .

Original: While our analysis is limited to the speciﬁc teacher and student setup , this simple model already exhibits dynamics closely matching those observed in more complex networks, as we now illustrate.
Revised: While our analysis is limited to the speciﬁc teacher and student setup, this simple model already exhibits dynamics qualitatively similar to those observed in more complex networks, as we now illustrate.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this Section, we conduct numerical simulations to validate our analytical results and provide clear insights on the macroscopic dynamics of generalization.
Revised: In this Section, we conduct numerical simulations to validate our analytical results and provide clear insights on the macroscopic dynamics of generalization.

Original: We also conduct experiments on real-world neural networks. We demonstrate that our teacher-student setup exhibits generalization behaviorwhich is qualitatively similar to that ofdeep neural networks.
Revised: We also conduct experiments on realworld neural networks showing a close qualitative match between the generalization behavior of neural networks and our teacher-student setup.

Original: For simulations and evaluation of the theory , a teacher generates n = 300 examples with d = 200 .

Original: The standard deviation of teacher noise is 0 . 1 .

Original: Assumption 1 holds with p = 100 where we vary the condition number κ .

Original: For numerical results, we use SGD with a learning rate of 0 .

Original: 01 and 0 . 0 momentum.

Original: Numerical simulations are averaged over 100 random seeds.

Original: For real-world experiments , we train a ResNet18 (He et al., 2016) with large layer widths [64 , 2 × 64 , 4 × 64 , 8 × 64] .
Revised: For real-world experiments , we train a ResNet18 (He et al., 2016) with large layer widths [64 , 2 × 64 , 4 × 64 , 8 × 64] .

Original: We follow the training setup of Nakkiran et al.
Revised: We follow the training setup of Nakkiran et al.

Original: ; Label noise with a probability 0 .
Revised: ; Label noise with a probability 0 .

Original: 15 randomly assign an incorrect label to training examples.
Revised: 15 randomly assign an incorrect label to training examples.

Original: Noise is samples only once before the training starts.
Revised: Noise is sampled only once before the training starts.

Original: We train using Adam (Kingma & Ba, 2014) with learning rate of 1 e − 4 for 1K epochs.
Revised: We train using Adam (Kingma & Ba, 2014) with learning rate of 1 e − 4 for 1K epochs.

Original: Real-world experiments are averaged over 50 random seeds.
Revised: Real-world experiments are averaged over 50 random seeds.

Original: To ensure reproducibility, we include the complete source code in a GitHub repository as well as an anonymous Collab notebook .
Revised: To ensure reproducibility, we include the complete source code in a GitHub repository as well as an anonymous Collab notebook .

Original: 3. M ATCH BETWEEN THEORY AND REAL - WORLD EXPERIMENTS
Revised: 3. M ATCH BETWEEN THEORY AND REAL - WORLD EXPERIMENTS

Original: We conduct an experiment on the classiﬁcation task of CIFAR-10 (Krizhevsky et al., 2009) with varying amount of weight decay regularization strength λ .
Revised: We conduct an experiment on the classiﬁcation task of CIFAR-10 (Krizhevsky et al., 2009) with varying amount of weight decay regularization strength λ .

Original: We monitor the generalization error (01 test error) during the course of training and visualize a heat-map of the generalization error for different λ ’s in Figure 2 (a).
Revised: We monitor the generalization error (01 test error) during the course of training and visualize a heat-map of the generalization error for different λ ’s in Figure 2 (a)

Original: We also conduct a similar experiment with the teacher-student setup presented in Section 2.2.
Revised: We also conduct a similar experiment with the teacher-student setup presented in Section 2.1.

Original: We visualize a heat-map of the generalization error which is the mean squared error (MSE) over test distribution in Figure 2 (b).
Revised: We visualize a heat-map of the generalization error which is the mean squared error (MSE) over test distribution in Figure 2 (b).

Original: Particularly, we plot Eqs.
Revised: Particularly, we plot Eqs.

Original: 12 and 14 with a constant κ = 100 .
Revised: 13 and 15 with a constant κ = 100 .

Original: As a remark, we note that a κ = 100 implies that some features are learned 100 times faster than other features.
Revised: As a remark, we note that a κ = 100 implies that a subset of features are learned 100 times faster than other features.

Original: Critically, it can be observed that in both experiments a model with intermediate levels of regularization displays a typical overﬁtting behavior where the generalization error decreases ﬁrst and then overﬁts.
Revised: It is observed that in both experiments, a model with intermediate levels of regularization displays a typical overﬁtting behavior where the generalization error decreases ﬁrst and then overﬁts.

Original: This is consistent with Eq. {+
+}
Revised: This is consistent with Eq. {+
+}

Original: 10 which indicates larger amounts of regularization prevent slow feature from being learned as λ and the inverse of time are summed.
Revised: which indicates larger amounts of regularization prevent slow feature from being learned as λ and the inverse of t are summed.

Original: In our particular teacherstudent setup, learning of slow features requiresweights with large magnitudes which are mitigated by the weight-decay regularization.
Revised: In other words, learning of slow features requires large weights, something that is penalized by the weight-decay.

Original: On the other hand, a model with smaller amount of regularization exhibits a larger double descent generalization curve, i e. , with more training, the generalization error will eventually improve.
Revised: On the other hand, a model with smaller amount of regularization exhibits the double descent generalization curve.

Original: We also validate our derived analytical expressions by running numerical simulations which are presented in Figure 4.
Revised: We also validate our derived analytical expressions by running numerical simulations which are presented in Figure 4.

Original: 3. T HE P HASE DIAGRAM
Revised: 3. T HE P HASE DIAGRAM

Original: To further investigate the transition between the two phases of classical single descent and double descent , we explore the phase diagram.
Revised: To further investigate the transition between the two phases of classical single descent and double descent , we explore the phase diagram.

Original: Recall that with Eq.
Revised: Recall that with Eq.

Original: 6, one can fully characterize the evo- lution of the generalization dynamics in terms of two scalar variables instead of the d -dimensional parameter space.
Revised: 6, one can fully characterize the evolution of the generalization dynamics in terms of two scalar variables instead of the d -dimensional parameter space.

Original: R and Q presented in Eq.
Revised: R and Q presented in Eq.

Original: 7 are macroscopic variables where R represents the alignment between the teacher and the student and Q is the student’s (modulated) norm .
Revised: 7 are macroscopic variables where R represents the alignment between the teacher and the student and Q is the student’s (modulated) norm .

Original: Hence, a better generalization performance is achieved with larger R and smaller Q .
Revised: Hence, a better generalization performance is achieved with larger R and smaller Q .

Original: R and Q are not free parameters and both depend on the training dynamics through Eqs.
Revised: R and Q are not free parameters and both depend on the training dynamics through Eqs.

Original: 12 and 14.
Revised: 13 and 15.

Original: Nevertheless, it is instructive to visualize the generalization error for all pairs of ( R, Q )  ∈ [0 . 0 , 0 . 8] ×  [0 . 0 , 1 . 6] .
Revised: Nevertheless, it is instructive to visualize the generalization error for all pairs of ( R, Q ) . In Figure 3, we visualize the RQ -plane for ( R, Q ) ∈ [0 . 0 , 0 . 8] × Q ∈ [0 . 0 , 1 . 6] .

Original: We also visualize the trajectories taken by SGD.

Original: At the time of initialization, ( R, Q ) =
Revised: At the time of initialization, ( R, Q ) =

Original: (0 , 0) as the models are initialized at the origin.
Revised: (0 , 0) as the models are initialized at the origin.

Original: As training time proceeds, values of
Revised: As training time proceeds, values of

Original: R and Q follow the depicted trajectories.
Revised: R and Q follow the depicted trajectories.

Original: In Figure 3, different trajectories correspond to different values of κ , the condition number of the modulation matrix F in Eq.
Revised: In Figure 3, different trajectories correspond to different values of κ , the condition number of the modulation matrix F in Eq.

Original: The condition number implies that some features are learned κ times faster than the others.

Original: It is important to note that the closer a trajectory gets to the lower-right, the smaller the generalization error is.
Revised: It is important to note that the closer a trajectory is to the lower-right, the better the generalization error gets.

Original: The yellow curve which corresponds to the case with large κ = 1 e 5 meaning that some features are extremely slower than the other features that practically do not get learned.
Revised: The yellow curve which corresponds to the case with large κ = 1 e 5 meaning that a subset of features are extremely slower than the others that practically do not get learned.

Original: In that case, generalization error exhibits traditional over-ﬁtting due to over-training.
Revised: In that case, generalization error exhibits traditional over-ﬁtting due to over-training.

Original: On the phase diagram, the yellow trajectory starts at (0 , 0) and moves towards Point A which has the lowest generalization error of this curve.
Revised: On the phase diagram, the yellow trajectory starts at (0 , 0) and moves towards Point A which has the lowest generalization error of this curve.

Original: Then as the training continues, Q increases and as t → ∞ the trajectory lands at Point B which has the worse generalization error.
Revised: Then as the training continues, Q increases and as t → ∞ the trajectory lands at Point B which has the worse generalization error.

Original: The curves in orange, green and blue correspond to trajectories with κ = 1 e 3 , κ = 1 e 2 , κ = 1 e 1 , respectively.
Revised: The curves in orange, green and blue correspond to trajectories with κ = 1 e 3 , κ = 1 e 2 , κ = 1 e 1 , respectively.

Original: They follow the case of κ = 1 e 5 up to the vicinity of Point B, but then the trajectories slowly incline towards another ﬁxed point, Point C signalling a second descent in the generalization error.
Revised: They follow the case of κ = 1 e 5 up to the vicinity of Point B, but then the trajectories slowly incline towards another ﬁxed point, Point C signalling a second descent in the generalization error.

Original: The phase diagram along with the corresponding generalization curves in Figure 2 illustrate that features that are learned on a faster time-scale are responsible for the initial conventional Ushaped generalization curve, while the second descent can be attributed to the features that are learned at a slower time-scale [Equation] [Figure] [Equation]
Revised: The phase diagram along with the corresponding generalization curves in Figure 2 illustrate that features that are learned on a faster time-scale are responsible for the initial conventional U-shaped generalization curve, while the second descent can be attributed to the features that are learned at a slower time-scale.

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: /T2RstCYfhjYzpBh10x7Q/E/r5Zg+7SRChUnCIqPF7UTSTGiw1RoS2jgKPuWMK6FvZXyLtOMo80ua0Pwp1+eJeWjvO/

Original: [Figure]

Original: /evjXOFsEkeG7JJ9ckh8ckIK5JIUSYlw8kieySt5c56cF+fd+Ri3zjmTmR3yB87nD0z

Original: [Figure]

Original: <

Original: [Figure]

Original: latexit

Original: [Figure]

Original: <latexit sha

Original: [Figure]

Original: "RmZQmjrrZdoLIMJsgukYHnuHMkE

Original: [Figure]

Original: [Figure]

Original: AAACCHicbVDLSgNBEJz1GeNr1aMHB4PgKeyKoMegCF6EiOYBSQizk04yZHZ2mekV45KjF3/FiwdFvPoJ3vwbJ

Original: [Figure]

Original: DJhY0FFXddHcFsRQGPe

Original: [Figure]

Original: dGoLPEn5AcmaDYdL

Original: [Figure]

Original: qrYgnISjkkhlT870YGynTKLiEQbaeGIgZ77EO1CxVLATTSEePDOiBVVq0HWlbCulI

Original: [Figure]

Original: T2RstCYfhjYzpBh10x7Q

Original: [Figure]

Original: <

Original: [Figure]

Original:  R ELATED W ORK AND D ISCUSSION
Revised:  R ELATED W ORK AND D ISCUSSION

Original: Double Descent: Although the term(model-wise) double descent has been introduced  recently in reference to the population risk demonstrating a cusp at the interpolation point, a similar phe- nomenon had already been  studied in several decades-old works in the regression setting (Krogh & Hertz, 1992a; Opper, 1995; Opper & Kinzel, 1996; B¨os, 1998) under a statistical physics framework.
Revised: Although the term double descent has been introduced rather recently (Belkin et al., 2019a), similar behaviors had already been observed and studied in several decades-old works form a statistical physics perspective (Krogh & Hertz, 1992a; Opper, 1995; Opper & Kinzel, 1996; B¨os, 1998; Engel & Van den Broeck, 2001).

Original: More recently, these behaviors have been investigated in the context of modern machine learning, both from an empirical (Nakkiran et al., 2019; 
Revised: More recently, these behaviors have been investigated in the context of modern machine learning, both from an empirical (Nakkiran et al., 2019; Amari et al., 2020;

Original: Amari et al., 2020; Yang et al., 2020) and theoretical (Engel & Van den Broeck, 2001; Belkin et al., 2019a; Geiger et al., 2019; Advani & Saxe, 2017; 
Revised: Yang et al., 2020) and theoretical (Belkin et al., 2019a; Geiger et al., 2019; Advani & Saxe, 2017; Mei &

Original: Mei & Montanari, 2019; Gerace et al., 2020; d’Ascoli et al., 2020; Ba et al., 2019) point of view, to determine various limiting behavior of the training and generalization error.
Revised: Montanari, 2019; Gerace et al., 2020; d’Ascoli et al., 2020; Ba et al., 2019; d’Ascoli et al., 2021)

Original: Our work is closely related to that of Heckel & Yilmaz (2020), which provides upper bounds on the risk of single and two layer models in a regression setting during gradient-based training, on input-data with distinct feature variances. In that work, the authors demonstrate empirically that epoch-wise double descent can be viewed as a superposition of two or more bias-variance tradeoff curves due to different parts of the network being learned at different epochs. Furthermore, the authors show that epoch-wise double descent can be eliminated by appropriate selection of learning rates for individual network weights. Consistent with these ﬁndings, our work formalizes this phenomenon in terms of feature learning scales and provides closed-form predictions.
Revised: perspectives

Original: Furthermore, by leveraging tools from statistical physics such as the replica method, we derive explicit equations for the generalization error as a function of regularization strength and training time in a teacher-student setup. We believe our analysis takes a step towards studying the generalization dynamics of neural networks, by characterizing their behavior using a ﬁnite set of informative macroscopic parameters. In summary, while supporting the notion that the interaction of different feature learning speeds causes epoch-wise double descent, our work provides formal predictions of the dynamics that unfold during training. Random matrix theory: Hastie et al.
Revised: Hastie et al.

Original: ; Advani et al.
Revised: ; Advani et al.

Original: (2020); Belkin et al.
Revised: (2020); Belkin et al.

Original: (2020) use random matrix theory (RMT) tools to characterize the asymptotic generalization behavior of overparameterized linear and random feature models.
Revised: (2020) use random matrix theory (RMT) tools to characterize the asymptotic generalization behavior of over-parameterized linear and random feature models.

Original: In an inﬂuential work, Mei & Montanari (2019) extend the same analysis to a random feature model and theoretically derive the model-wise double descent curve for a model with Tikhonov regularization.
Revised: In an inﬂuential work, Mei & Montanari (2019) extend the same analysis to a random feature model and theoretically derive the model-wise double descent curve for a model with Tikhonov regularization.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2020) also study double descent in ridge estimators and show an equivalence to kernel ridge regression.
Revised: (2020) also study double descent in ridge estimators and show an equivalence to kernel ridge regression.

Original: Pennington & Worah (2019) used RMT to study the curvature of single-hidden-layer neural network in an attempt to understand the efﬁcacy of ﬁrst-order optimization methods in training DNNs.
Revised: Pennington & Worah (2019) used RMT to study the curvature of single-hidden-layer neural network in an attempt to understand the efﬁcacy of ﬁrst-order optimization methods in training DNNs.

Original: In addition, Liang & Rakhlin (2020) take a similar approach to investigate implicit regularization in high dimensional ridgeless regression with nonlinear kernels. 
Revised: In addition, Liang & Rakhlin (2020) take a similar approach to investigate implicit regularization in high dimensional ridgeless regression with nonlinear kernels. While most of the related work study the non-monotonicity of the generalization error as a function of the model size or sample size, Nakkiran et al. (2019) introduced the epoch-wise double descent.

Revised: Epoch-wise double descent refers to the phenomenon where the generalization error undergoes two descents as the training time increases.

Revised: There has been limited work on studying of epoch-wise double descent.

Original: Generalization properties of over-parameterized networks: Neyshabur et al.
Revised: Very recently, Heckel & Yilmaz (2020) and Stephenson & Lee (2021) have focused on ﬁnding the roots of this phenomenon. Heckel & Yilmaz (2020) provides upper bounds on the risk of single and two layer models in a regression setting where the input data has distinct feature variances. Heckel & Yilmaz (2020) demonstrate that a superposition of two or more bias-variance tradeoff curves leads to epoch-wise double descent. The authors also show that different layers of the network are learned at different epochs.

Revised: For that reason, epoch-wise double descent can be eliminated by appropriate selection of learning rates for individual network weights.

Revised: Consistent with these ﬁndings, our work formalizes this phenomenon in terms of feature learning scales and provides closed-form predictions.

Revised: Stephenson & Lee (2021) arrives at similar conclusions.

Revised: Authors in Stephenson & Lee (2021) take a random matrix theory approach on a data model that exhibits epoch-wise double descent.

Revised: The data model is constructed so that the noise is explicitly added only to the fast-learning features while slow-learning features remain noise-free.

Revised: Consequently, the fast-learning features are noisy and hence show a U-shaped generalization curve while slow-learning features are noiseless.

Revised: Our ﬁndings and those of Heckel & Yilmaz (2020) and Stephenson & Lee (2021) reinforce one another with a common central ﬁnding that the epoch-wise double descent results from different features/layers being learned at different time-scale.

Revised: However, we also highlight that both Heckel & Yilmaz (2020) and Stephenson & Lee (2021) built upon tool from random matrix theory and study distinct data models from our teacher-student setup.

Revised: We study the same phenomenon from a different perspective.

Revised: By leveraging the replica method from statistical physics, we characterized the generalization behavior using a set of informative macroscopic parameters.

Revised: While supporting the notion that the interaction of different feature learning speeds causes epoch-wise double descent, our work provides formal predictions of the dynamics that unfold during training.

Revised: We believe our theoretical framework sets the stage for further understanding of generalization dynamics in neural networks beyond the double descent.

Revised: A future direction to study is a case in which the ﬁrst descent is strong enough to bring down the training loss to very small values to the point that learning slower features is practically impossible or happens after a very large number of epochs.

Revised: Power et al.

Revised: (2021) reports an instance of such behavior called Grokking where the model abruptly learns to perfectly generalize but long after the training loss has reached very small values.

Revised: Limitations.

Revised: It should be noted that studying ﬁner details of the dynamics would require a more precise model of the neural networks.

Revised: Clearly, our proposed model is not a universal and unique way to model the dynamics of the complex, over-parameterized deep neural networks.

Revised: Social Impact.

Revised: The authors do not foresee a negative social impact speciﬁcally arising from this rather theoretical work.

Revised: A F URTHER R ELATED W ORK AND D ISCUSSION

Revised: If we consider plots where the generalization error on the y -axis is plotted against other quantities on the x -axis, we ﬁnd earlier works that have identiﬁed double descent behavior for quantities such as the number of parameters, the dimensionality of the data, the number of training samples, or the training time on the x -axis.

Revised: In this paper, we studied epoch-wise double descent, i.e. we plot the training time t , or the number of training epochs, on the x -axis.

Revised: Literature displaying double descent phenomena in generalization behavior w.r.t.

Revised: other quantities do so in the limit of t → ∞ .

Revised: From a random matrix theory perspective, Le Cun et al.

Revised: ; Hastie et al.

Revised: ; Advani et al.

Revised: (2020), and Belkin et al.

Revised: (2020) are among works which have analytically studied the spectral density of the Hessian matrix.

Revised: According to their analyses, at intermediate levels of complexity, the presence of small but non-zero eigenvalues in the Hessian matrix results in high generalization error as the inverse of the Hessian is calculated for the pseudo-inverse solution.

Revised: Neyshabur et al.

Original: (2014) demonstrated that over-parameterized networks does not necessarily overﬁt thus suggesting the need of a new form of measure of model complexity other than network size.
Revised: (2014) demonstrated that over-parameterized networks does not necessarily overﬁt thus suggesting the need of a new form of measure of model complexity other than network size.

Original: Subsequently, Neyshabur et al.
Revised: Subsequently, Neyshabur et al.

Original: (2018) suggest a novel complexity measure based on unit-wise capacities which correlates better with the behavior of test error with increasing network size.
Revised: (2018) suggest a novel complexity measure based on unit-wise capacities which correlates better with the behavior of test error with increasing network size.

Original: Chizat & Bach (2020) study the global convergence and superior generalization behavior of inﬁnitely wide two-layer neural networks with logistic loss.
Revised: Chizat & Bach (2020) study the global convergence and superior generalization behavior of inﬁnitely wide two-layer neural networks with logistic loss.

Original: Goldt et al.
Revised: Goldt et al.

Original: (2020) make use of the Gaussian Equivalence Theorem to study the generalization performance of two-layer neural networks and kernel models trained on data drawn from pre-trained generative models.
Revised: (2020) make use of the Gaussian Equivalence Theorem to study the generalization performance of two-layer neural networks and kernel models trained on data drawn from pre-trained generative models.

Original: Bai & Lee (2020) investigated the gap between the empirical performance of over-parameterized networks and their NTK counterparts, ﬁrst proposed by Jacot et al. 
Revised: Bai & Lee (2020) investigated the gap between the empirical performance of over-parameterized networks and their NTK counterparts, ﬁrst proposed by Jacot et al. From the perspective of bias/variance trade-off, Geman et al. (1992), and more recently, Neal et al.

Revised: (2018) empirically observe that while bias is monotonically decreasing, variance could be decreasing too or unimodal as the number of parameters increases, thus manifesting a double descent generalization curve.

Original: In relation to the aspect of different feature learning scales, Rahaman et al. 
Revised: Hastie et al. (2019) analytically study the variance. More recently, Yang et al. (2020) provides a new bias/variance decomposition of bias exhibiting double desc-nt in which the variance follows a bell-shaped curve.

Revised: However, the decrease in variance as the model size increases remains unexplained.

Revised: For high dimensional regression with random features, d’Ascoli et al.

Revised: (2020) provides an asymptotic expression for the bias/variance decomposition and identiﬁes three sources of variance with non-monotonous behavior as the model size or dataset size varies.

Revised: d’Ascoli et al.

Revised: (2020) also employs the analysis of random feature models and identiﬁes two forms of overﬁtting which leads to the so-called sample-wise triple descent.

Revised: More recently, Chen et al.

Revised: (2020) show that as a result of the interaction between the data and the model, one may design generalization curves with multiple descents.

Revised: From a statistical physics perspective, Opper (1995); B¨os et al.

Revised: ; B¨os (1998); Opper & Kinzel (1996) are among the ﬁrst studies which theoretically observe sample-wise double-descent in a ridge regression setup where the solution is obtained by the pseudo-inverse method.

Revised: Most of these studies employ the “Gardner analysis” (Gardner, 1988; Gardner & Derrida, 1988; 1989) for models where the number of parameters and the dimensionality of data are coupled and hence the observed form of double descent is different from that observed in deep neural networks.

Revised: A beautiful extended review of this line of work is provided in Engel & Van den Broeck (2001).

Revised: Among recent works, Gerace et al.

Revised: (2020) also apply the Gardner analysis but to a novel generalized data generating process called the hidden manifold model and derive the model-wise double-descent equations analytically.

Revised: Finally, recall that towards providing an explanation for the epoch-wise double descent, we argue that the epoch-wise double descent can be attributed to different features being learned at different time-scales , resulting in a non-monotonous generalization curve.

Revised: In relation to the aspect of different feature learning scales, Rahaman et al.

Original: (2019) had observed that DNNs have a tendency towards learning simple target functions ﬁrst that can allow for good generalization behavior of various data samples.
Revised: (2019) had observed that DNNs have a tendency towards learning simple target functions ﬁrst that can allow for good generalization behavior of various data samples.

Original: Pezeshki et al.
Revised: Pezeshki et al.

Original: (2020) also identify and provide explanation for a feature learning imbalance exhibited by over-parameterized networks trained via gradient descent on cross-entropy loss, with the networks learning only a subset of the full feature spectrum over training.
Revised: (2020) also identify and provide explanation for a feature learning imbalance exhibited by over-parameterized networks trained via gradient descent on cross-entropy loss, with the networks learning only a subset of the full feature spectrum over training.

Original: More recently though, Zhang & Wu (2020), show that certain DNNs models prioritize learning high-frequency components ﬁrst followed by the learning of slow but informative features, leading to the second descent of the test error as observed in epoch-wise double descent. 
Revised: More recently though, Zhang & Wu (2020), show that certain DNNs models prioritize learning highfrequency components ﬁrst followed by the learning of slow but informative features, leading to the second descent of the test error as observed in epoch-wise double descent. On the difference between model-wise and epoch-wise double descent curves.

Original: Limitations.
Revised: In accordance with its name, model-wise double descent (in the test error) occurs due to an increase in modelsize (number of its parameters), i.e., as the model transitions from an under-parameterized to an over-parameterized regime. A variety of works have tried to understand this phenomenon from the lens of implicit regularization (Neyshabur et al., 2014) or deﬁning novel complexity measures (Neyshabur et al., 2017). On the other hand, epoch-wise double descent (in the test error) as treated in our work, is observed to occur for both over-parameterized (Nakkiran et al., 2019) and underparameterized (Heckel & Yilmaz, 2020) setups. As found in our work along with the latter reference, this phenomenon seems to be a result of different feature learning speeds rather than the extent of model parameterization.

Revised: The overlap of the test-error contributions from the different weights with varying scales of learning henceforth leads to a non-monotonous evolution of the model test error as exempliﬁed by epoch-wise double descent.

Revised: We also note that the peak in model-wise double descent is associated with the model’s capacity to perfectly interpolate the data, we do not think an analogous notion exists for the case of epoch-wise double descent.

Revised: Our understanding of the peak in the latter is that it corresponds to a training time conﬁguration whereby a subclass of features are already learnt (due to a larger associated signal-tonoise-ratio) and are being overﬁtted upon to ﬁt the target.

Revised: As training proceeds further, the remaining set of features are eventually learnt thus allowing for a lowering of the test error.

Revised: On the implicit regularization of SGD and ridge-regularized loss.

Revised: The results presented in Eqs.

Revised: 20-23 have a core dependence on the ﬁndings of Ali et al.

Revised: These works ﬁrst formalize the connection between (continuous-time) GD or SGD-based training of an ordinary least squares (OLS) setup and that of ridge regression, providing bounds on the test error under these algorithms over training time t , in terms of a ridge setup with ridge parameter λ

Revised: = 1 /t .

Revised: We utilize these results in the sense that by evaluating the generalization error L G of our student-teacher setup with explicit ridge regularization, we invoke the connection between the ridge coefﬁcient λ and training time t as described in these works, to obtain the behavior of (ridgeless)

Revised: L G over training.

Revised: This determination of an expression of L G ( t ) is what allows us to study the epoch-wise DD phenomenon.

Revised: B T ECHNICAL P ROOFS B. T HE GENERALIZATION ERROR AS A FUNCTION OF R AND Q (E Q . 6)

Revised: Recall that the teacher is the data generator and is deﬁned as,

Revised: [Equation]

Revised: where

Revised: z ∈ R

Revised: d is the teacher’s input and y ∗ , y ∈ R are the teacher’s noiseless and noisy outputs, respectively.

Revised: W ∈ R d represents the (ﬁxed) weights of the teacher and

Revised: (cid:15) ∈ R is the noise.

Revised: And student is deﬁned as,

Revised: [Equation]

Revised: where the matrix F ∈ R d

Revised: ×

Revised: d is a predeﬁned and ﬁxed modulation matrix regulating the student’s access to the true input z .

Revised: The average generalization error of the student, determined by averaging the student’s error over all possible input-target pairs and noise realizations is given by,

Revised: [Equation]

Revised: (28) in which the variables ( y ∗ , ˆ y ) form a bi-variate Gaussian distribution with zero mean and a covariance of,

Revised: [Equation]

Revised: in which,

Revised: [Equation]

Revised: and,

Revised: [Equation]

Revised: Eq. 29 implies a correlation between y ∗ and ˆ y obstructing the calculation of the average in Eq.

Revised: Following (B¨os, 1998; Krogh & Hertz, 1992a), we deﬁne decoupled variables ˜

Revised: y ∗ and ˜ˆ y as follows,

Revised: [Equation]

Revised: , and

Revised: [Equation]

Revised: The variables ˜ y ∗ and ˜ˆ y are independent Gaussian variables such that < ˜

Revised: y ∗ ,

Revised: ˜ˆ y >

Revised: z = 0 .

Revised: Therefore, two expectations can be applied independently,

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Finally, we note that expectation w.r.t.

Revised: a Gaussian variable x is deﬁned as,

Revised: [Equation]

Revised: B. T HE GENERAL CASE EXACT DYNAMICS (E QS . 9-10)

Revised: Recall that to train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,

Revised: [Equation]

Revised: where λ ∈ [

Revised: [Equation]

Revised: ∞ ) is the regularization coefﬁcient.

Revised: The minimum of the loss function, denoted by W , is achieved at,

Revised: [Equation]

Revised: An exact gradient descent has the following dynamics,

Revised: [Equation]

Revised: which leads to,

Revised: [Equation]

Revised: Assuming ˆ W 0 =

Revised: [Equation]

Revised: we arrive at the following closed-form equation,

Revised: [Equation]

Revised: where W is deﬁned in Eq 40.

Revised: Now back to deﬁnition of R in Eq.

Revised: 84 and by substitution of Eq.

Revised: [Equation]

Revised: we have,

Revised: [Equation]

Revised: Similarly for Q

Revised: [Equation]

Revised: let D :=

Revised: (cid:16) I − (cid:2)

Revised: [Equation]

Revised: I

Revised: [Equation]

Revised: η Λ

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: , then we have,

Revised: [Equation]

Revised: where,

Revised: [Equation]

Revised: and,

Revised: [Equation]

Revised: For simplicity and brevity of the results, in the main text, we only present the results where σ 2 (cid:15) =and λ = 0 .

Revised: Substituting σ 2 (cid:15) =

Revised: λ = 0 leads to the following expressions,

Revised: [Equation]

Revised: where, A := FV (cid:16) I − (cid:2)

Revised: I − η Λ (cid:3) t (cid:17) V T F − 1 , (71) and that concludes the proof.

Revised: B. T HE SPECIAL CASE APPROXIMATE DYNAMICS (E QS . 13 AND 15)

Revised: Recall that the teacher and student are deﬁned as,

Revised: [Equation]

Revised: (72) where (cid:15) ∼ N (0 , σ 2 (cid:15) ) is the label noise, F is the modulation matrix, and || z

Revised: || 22 =

Revised: || W

Revised: ||

Revised: = 1 .

Revised: The training and generalization losses are deﬁned as,

Revised: [Equation]

Revised: According to Eq. 6, the generalization loss can be written in terms of two scalar variables R and Q ,

Revised: [Equation]

Revised: where,

Revised: [Equation]

Revised: Now, applying t steps of SGD on L T results in the following distribution for the student’s weights,

Revised: [Equation]

Revised: in which ˜ L T ( ˆ W, t ) is a modiﬁed loss where its equilibrium coincides with the t th iterate of SGD on the original loss L T ( ˆ W ) .

Revised: In Eq.

Revised: 77 the scalar variable β depends on the noise of SGD and Z β,t is the partition function which is deﬁned as,

Revised: [Equation]

Revised: in which, Q 0 can be perceived to be a target norm the student weights ˆ W are being constrained to and d is the dimensionality of the data.

Revised: It can be interpreted that the partition function Z β,t counts the students.

Revised: We are now interested in ﬁnding R and Q of the typical (most probable) students.

Revised: Therefore, it sufﬁces to ﬁnd the students that dominate the partition function (or more precisely the free-energy).

Revised: The free-energy is deﬁned as,

Revised: [Equation]

Revised: [Equation]

Revised: where W and z are the teacher’s weight and input, respectively.

Revised: Due to the logarithm inside the expectation, analytical computation of Eq.

Revised: 79 is intractable.

Revised: However, the replica method (M´ezard et al., 1987) allows us to tackle this through the following identity,

Revised: [Equation]

Revised: The case where F = I .

Revised: As a ﬁrst step, we ﬁrst study a case where F = I .

Revised: In that case, as derived in B ¨ os (1998), Eq. 79 can be simpliﬁed to,

Revised: [Equation]

Revised: in which the scalar variables G and H are deﬁned as,

Revised: [Equation]

Revised: At this point, in order to ﬁnd the most probable students, one can extremize the free-energy f

Revised: ( R, Q, Q 0 ) in Eq.

Revised: The solution to this extermination is derived in B¨os et al.

Revised: (1993) and reads,

Revised: [Equation]

Revised: [Equation]

Revised: in which,

Revised: [Equation]

Revised: and ,

Revised: [Equation]

Revised: The case where F follows Assumption 1.

Original: It should be noted that studying ﬁner details of the dynamics would require a more precise model of  the neural networks.
Revised: Assumption. The modulation matrix, F , under a SVD, F := U Σ V T has two sets of singular values such that the ﬁrst p singular values are equal to σ 1 and the remaining d − p singular values are equal to σ 2 . We let the condition number of F to be denoted by κ := σ 1 σ 2 > 1 .

Revised: Without loss of generality, we assume that U

Revised: = V =

Revised: I .

Original: Clearly, our proposed model is not a universal andunique way to model the dynamics of the complex, over-parameterized deep neural networks.
Revised: Consequently, the (noiseless) teacher and the student can be written as the composition of two sub-models as following, [Equation] in which z 1 ∈ R p and z 2 ∈ R d − p .

Revised: Let ˆ y

Original: Social Impact.
Revised: i denote the output of the i th component of the student. Also let y ∗ i and y i denote the noiseless and noisy targets, respectively. Therefore, for the student components i ∈ 1 , 2 , we have,

Original: The authors do not foresee a negative social impact speciﬁcally arising from this rather theoretical work.
Revised: [Equation] in which (cid:15) is the explicit noise , added to the teacher’s output while (cid:15) j ( t ) is an implicit variable noise which decreases as the component j (cid:54) =

Revised: i learns to match ˆ y j and y j .

Revised: Accordingly, the variables H i and G i for each component i are re-deﬁned as,

Original: A A PPENDIX A. R EPLICA T RICK
Revised: [Equation] in which R i and Q i are deﬁned as, [Equation] and

Revised: [Equation]

Revised: where σ i denotes the singular values of the matrix F as deﬁned in Assumption 1.

Revised: Rewriting Eqs.

Revised: [Equation]

Revised: and 86 for each of the student’s components, we arrive at,

Revised: [Equation]

Revised: where Q 1 depends on Q 2 and vice versa.

Revised: However, with simple calculations, we can arrive at the following standalone equation.

Revised: Let,

Revised: [Equation]

Revised: and also let,

Revised: [Equation]

Original: In the following
Revised: for i [Equation] [Equation] with which the closed-from scalar expression for Q ( t, λ ) reads,

Original: [Equation]
Revised: [Equation]

Original: we detail the mathematical arguments leading to the replica trick expression. For some r → 0 ,we can write for any scalar x :
Revised: where ,

Original: [Equation]
Revised: [Equation]

Original: A. G ENERALIZATION E RROR
Revised: B. R EPLICA T RICK

Original: The generalization error for our linear teacher-student setup is deﬁned to be the average of the MSE over all possible teacher and student input conﬁgurations, x , i [Equation] e., [Equation]
Revised: In the following, we detail the mathematical arguments leading to the replica trick expression.

Original: [-where, N is the number of input samples, Q 
 = W T Γ W-] ,R = W T ∆ W and | W | 2 = 1 . It is to be noted that in the above, we have used: [Equation]
Revised: [-where, N is the number of input samples, Q 
 = W T Γ W-]For some r → 0 , we can write for any scalar x :

Original: [Equation]
Revised: [Equation]

Original: A. T HERMAL AND S ELF -A VERAGED T RAINING AND G ENERALIZATION E RROR
Revised: C OMPUTATION OF THE FREE - ENERGY

Original: The self-averaged free energy (per unit weight) of our student network, is given by (Engel & Van den
Revised: The self-averaged free energy (per unit weight) of our student network, is given by (Engel & Van den

Original: Broeck, 2001),
Revised: Broeck, 2001),

Original: [Equation]
Revised: [Equation]

Original: Here, β = 1 /T is the inverse temperature parameter corresponding to our statistical ensemble, d the (teacher) student network width [Equation] and Z the partition function of the system deﬁned as ( n : number of training examples), [Equation]
Revised: Here, β = 1 /T is the inverse temperature parameter corresponding to our statistical ensemble, d the (teacher) student network width, and Z the partition function of the system deﬁned as ( n : number of training examples).

Original: with ,

Original: [Equation]

Original: Here, Q 0 can be perceived to be a target norm the student weights W are being constrained to.

Original: Next, by making use of the replica trick (Eq. 22)

Original: [Equation]

Original: we can next write Eq.

Original: 26 as,

Original: [Equation]

Original: Here, Z r corresponds to the combined average of n different replicas of our student network with the same realization of the random inputs.

Original: To evaluate (cid:104)(cid:104) Z r (cid:105)(cid:105) x , W , we begin by introducing the teacher and the student network functions, y ∗ µ and y a,µ ,

Original: [Equation]

Original: as Gaussian variables (with P → ∞ ), in the partition function, to obtain,
Revised: As Gaussian variables (with n, d → ∞ ), in the partition function, to obtain,

Original: [Equation]
Revised: [Equation]

Original: where in the last line above
Revised: where in the last line above

Original: [Equation]
Revised: [Equation]

Original: we have expressed the inserted δ functions using their integral representations.
Revised: we have expressed the inserted δ functions using their integral representations.

Original: Now, since our teacher and student inputs satisfy Eq.

Original: 24, hence, one can write in the limit

Original: P → ∞ , (cid:28)(cid:28)

Original: [Equation]

Original: where, (cid:104) a, b (cid:105) stands for a sum over a and b such that a (cid:54)

Original: = b .

Original: Additionally, in the last line, we have used the relations, W T Γ W = PQ 0 and, W ∗ T W = P .

Original: Therefore, assimilating the above, we can write,

Original: [Equation]

Original: To make further progress
Revised: To make further progress

Original: [Equation]
Revised: [Equation]

Original: we introduce the auxiliary variables
Revised: we introduce the auxiliary variables,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: via the respective δ functions, to arrive at,
Revised: via the respective δ functions, to arrive at,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Repeating the procedure of expressing the above δ functions using their integral representations
Revised: Repeating the procedure of expressing the above δ functions using their integral representations

Original: [Equation]
Revised: [Equation]

Original: we then get ( α = n/d ),
Revised: we then get ( α = n/d ),

Original: [Equation]
Revised: [Equation]

Original: = [Equation] If we now, perform a singular value decomposition of the covariance matrix Γ as, Γ = U T SU =
Revised: If we now, perform a singular value decomposition of the covariance matrix Γ as, Γ = U T SU =

Original: V T V
Revised: V T V

Original: [Equation]
Revised: [Equation]

Original: where S: matrix of singular values of Γ
Revised: where S: matrix of singular values of Γ

Original: [Equation]
Revised: [Equation]

Original: and we have expressed, V = S 1 / 2 U
Revised: and we have expressed [Equation] V = S 1 / 2 U

Original: [Equation]
Revised: [Equation]

Original: then one can proceed to write,
Revised: then one can proceed to write,

Original: [Equation]
Revised: [Equation]

Original: [Equation] having expressed, ˜ W a = V W a [Equation] and identifying ∆ =
Revised: having expressed, ˜ W a = V W a , and identifying ∆ =

Original: S 1 / 2 U from our deﬁnitions.
Revised: S 1 / 2 U from our deﬁnitions.

Revised: Now

Original: Now, since in the above, the W ai integrals factorize in i
Revised: [Equation] since in the above [Equation] the W ai integrals factorize in i

Original: [Equation]
Revised: [Equation]

Original: and similarly the y µa , ˆ y µa and d y ∗ µ factorize in µ , one can proceed to write:
Revised: and similarly the y µa , ˆ y µa and d y ∗ µ factorize in µ , one can proceed to write:

Original: [Equation]
Revised: [Equation]

Original: ) where,
Revised: where,

Original: [Equation]

Original: Now

Original: [Equation]
Revised: [Equation]

Original:  in the limit d → ∞ [Equation] Eq. 38 can be approximated using the saddle-point approach (Bender &
Revised: Now, in the limit d → ∞ , Eq. 101 can be approximated using the saddle-point approach (Bender &

Original: Orszag, 2013),
Revised: Orszag, 2013),

Original: [Equation]
Revised: [Equation]

Original: where [Equation] extr corresponds to extremization of (cid:104)(cid:104) Z n (cid:105)(cid:105) x , W over the respective order parameters.
Revised: where, extr corresponds to extremization of (cid:104)(cid:104) Z n (cid:105)(cid:105) x,W over the respective order parameters.

Original: Performing this extremization over ˆ Q 0 a [Equation] ˆ Q ab and ˆ R a 
Revised: Performing this extremization over ˆ Q 0 a , ˆ Q ab and ˆ R a , then generates an expression of the form,

Original: [Equation]
Revised: [Equation]

Original: then generates an expression of the form,

Original: [Equation]

Original: where we have invoked replica symmetry in the form, Q ab = Q and R a = R [Equation] and that E T=
Revised: where we have invoked replica symmetry in the form, Q ab = Q and R a = R , and that E T

Original:  ( y ∗ − y ) 2 / 2 .
Revised: = ( y ∗ − y ) 2 / 2 .

Original: Plugging this back into Eq. 
Revised: Plugging this back into Eq. ?

Original: 28, then ﬁnally yields,
Revised: , then ﬁnally yields,

Original: [Equation]
Revised: [Equation]

Original: The remaining pair of order parameters generate the following set of transcendental equations on extremization (B¨os, 1998):
Revised: The remaining pair of order parameters generate the following set of transcendental equations on extremization (B ¨ os, 1998):

Original: [Equation]
Revised: [Equation]

Original: where, a = max[1 , α ] for T → 0 .
Revised: where, a = max[1 , α ] for T → 0 .

Original: Now, the above determined values of R, Q and Q 0 can be perceived as the maximally likely values of
Revised: Now, the above determined values of R, Q and Q 0 can be perceived as the maximally likely values of

Original: R, Q and Q 0 of our teacher-student setup, for an inverse temperature β parameterizingthe system.
Revised: R, Q and Q 0 of our teacher-student setup, for an inverse temperature β parameterizing

Original: Generalization Error :

Original: The self-averaged generalization error can be obtained by plugging in the expressions for R ( ∗ ) and Q ( ∗ ) from above, into Eq.

Original: This yields,

Original: [Equation]

Original: [Equation]

Original: Training Error : The self-averaged training error, on the other hand is deﬁned to be, [Equation] B R IDGE REGULARIZATION AND F REE -E NERGY
Revised: C E XTENDED EXPERIMENTS

Original: The addition of a ridge or L 2 regularization term to the square-loss of our teacher student setup coupled with the choice of the regularization matrix F as detailed in 1, then yields,

Original: [Equation] causes a modiﬁcation of our free-energy expression of Eq. 42 to yield, [Equation]
Revised: Figure 4 presents the analytical generalization dynamics for two values of κ and provides comparison between the theory and simulation results of the same model.

Original: where , [Equation] and, [Equation]
Revised: We observe that the theory and

Original: [Equation]

Original: , where ,

Original: [Equation]
Revised: [Table]

Original: 1 b 2 c 2 + b 1 c1 − b 1 b, and ,

Original: Q 1 := b 1 b 2 c [Equation] b 2 c1 − b 1 b.
Revised: γ = 0.

Original: (48) for a i , b

Original: i and c i ( i = 1 , 2 ) deﬁned in the main text. C E XTENDED EXPERIMENTS Figure 4 presents the analytical generalization dynamics for two values of kappa and provides comparison between the theory and simulation results of the same model. We observe that the theory and simulations accurately match.
Revised: γ = 0.

Original: Further experiments are provided in the following anonymous

Original: Collab notebook .

Original: [Table]
Revised: [Table]

Original: +W9mSToR3QoecgZNVZ6uHlKUPEIpemXK27VnYOsEi8nFcjR6Je/eoOYpVmWCap113MT40+pMpwJnJV6qcaEsjEdYtdSSSPU/nS+8YycWWVAwljZJw2Zq78TUxppPYkCOxlRM9LLXib+53VTE176Uy6T1KBki4/CVBATk+x8MuAKmRETSyhT3O5K2IgqyowtqWRL8JZPXiWtWtVzq95drVK/yusowgmcwjl4cAF1uIUGNIGBhGd4hTdHOy/Ou/OxGC04eeYY/sD5/AHQapD9<
Revised: +W9mSToR3QoecgZNVZ6uHlKUPEIpemXK27VnYOsEi8nFcjR6Je/eoOYpVmWCap113MT40+pMpwJnJV6qcaEsjEdYtdSSSPU/nS+8YycWWVAwljZJw2Zq78TUxppPYkCOxlRM9LLXib+53VTE176Uy6T1KBki4/CVBATk+x8MuAKmRETSyhT3O5K2IgqyowtqWRL8JZPXiWtWtVzq95drVK/yusowgmcwjl4cAF1uIUGNIGBhGd4hTdHOy/Ou/OxGC04eeYY/sD5/AHQapD9</latexit>

Original: [Figure]
Revised: [Figure]

Original: latexit
Revised: simulations accurately match. Further experiments are provided in the following anonymous Colab notebook .

Original: [Figure]
Revised: Before diving into the theory, we invite the reader to recall a simple equation from thermodynamics. Consider an ideal gas in a container with its large number of molecules moving around, colliding with each other, all while obeying Newton’s laws. While the exact dynamics of each of such molecules is intractable, the system’s macroscopic behavior can be characterized in terms of a handful of scalar quantities, namely, the pressure P , the volume V , and the temperature T . By averaging over suitable probability measures and applying the principle of free-energy minimization, one arrives at a remarkably simple relationship between these three macroscopic variables, i.e., the well-known PV = nRT ( n : number of moles of gas, R : gas constant) (Reif, 2009).