Original: G RADIENT FLOW IN THE GAUSSIAN COVARIATE MODEL : EXACT SOLUTION OF LEARNING CURVES AND MULTIPLE DESCENT STRUCTURES
Revised: G RADIENT FLOW IN THE GAUSSIAN COVARIATE MODEL : EXACT SOLUTION OF LEARNING CURVES AND MULTIPLE DESCENT STRUCTURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A recent line of work has shown remarkable behaviors of the generalization error curves in simple learning models.
Revised: A recent line of work has shown remarkable behaviors of the generalization error curves in simple learning models.

Original: Even the least-squares regression has shown atypical features such as the model-wise double descent, and further works have observed triple or multiple descents.
Revised: Even the least-squares regression has shown atypical features such as the model-wise double descent, and further works have observed triple or multiple descents.

Original: Another important characteristic are the epoch-wise descent structures which emerge during training.
Revised: Another important characteristic are the epoch-wise descent structures which emerge during training.

Original: The observations of model-wise and epoch-wise descents have been analytically derived in limited theoretical settings (such as the random feature model) and are otherwise experimental.
Revised: The observations of model-wise and epoch-wise descents have been analytically derived in limited theoretical settings (such as the random feature model) and are otherwise experimental.

Original: In this work, we provide a full and uniﬁed analysis of the whole time-evolution of the generalization curve, in the asymptotic large-dimensional regime and under gradient-ﬂow, within a wider theoretical setting stemming from a gaussian covariate model.
Revised: In this work, we provide a full and uniﬁed analysis of the whole time-evolution of the generalization curve, in the asymptotic large-dimensional regime and under gradient-ﬂow, within a wider theoretical setting stemming from a gaussian covariate model.

Original: In particular, we cover most cases already disparately observed in the literature and also provide examples of the existence of multiple descent structures as a function of a model parameter or time.
Revised: In particular, we cover most cases already disparately observed in the literature and also provide examples of the existence of multiple descent structures as a function of a model parameter or time.

Original: Furthermore, we show that our theoretical predictions adequately match the learning curves obtained by gradient descent over realistic datasets.
Revised: Furthermore, we show that our theoretical predictions adequately match the learning curves obtained by gradient descent over realistic datasets.

Original: Technically we compute averages of rational expressions involving random matrices using recent developments in random matrix theory based on "linear pencils".
Revised: Technically we compute averages of rational expressions involving random matrices using recent developments in random matrix theory based on "linear pencils".

Original: Another contribution, which is also of independent interest in random matrix theory is a new derivation of related ﬁxed point equations (and an extension there-off) using Dyson brownian motions
Revised: Another contribution, which is also of independent interest in random matrix theory is a new derivation of related ﬁxed point equations (and an extension there-off) using Dyson brownian motions

Original:  I NTRODUCTION 1. P RELIMINARIES
Revised:  I NTRODUCTION 1. P RELIMINARIES

Original: With growing computational resources, it has become customary for machine learning models to use a huge number of parameters (billions of parameters in Brown et al.
Revised: With growing computational resources, it has become customary for machine learning models to use a huge number of parameters (billions of parameters in Brown et al.

Original: (2020)), and
Revised: (2020)), and

Original: the need for scaling laws has become of utmost importance Hoffmann et al.
Revised: the need for scaling laws has become of utmost importance Hoffmann et al.

Original: Therefore it is of great relevance to study the asymptotic (or "thermodynamic") limit of simple models in which the number of parameters and data samples are sent to inﬁnity.
Revised: Therefore it is of great relevance to study the asymptotic (or "thermodynamic") limit of simple models in which the number of parameters and data samples are sent to inﬁnity.

Original: A landmark progress made by considering these theoretical limits, is the analytical (oftentimes rigorous) calculation of precise double-descent curves for the generalization error starting with Belkin et al.
Revised: A landmark progress made by considering these theoretical limits, is the analytical (oftentimes rigorous) calculation of precise double-descent curves for the generalization error starting with Belkin et al.

Original: ; Hastie et al.
Revised: ; Hastie et al.

Original: ; Mei & Montanari (2019), Advani et al.
Revised: ; Mei & Montanari (2019), Advani et al.

Original: (2020), d’Ascoli et al.
Revised: (2020), d’Ascoli et al.

Original: (2020), Gerace et al.
Revised: (2020), Gerace et al.

Original: (2020), Deng et al.
Revised: (2020), Deng et al.

Original: (2021), Kini &
Revised: (2021), Kini &

Original: Thrampoulidis (2020) conﬁrming in a precise (albeit limited) theoretical setting the experimental phenomenon initially observed in Belkin et al.
Revised: Thrampoulidis (2020) conﬁrming in a precise (albeit limited) theoretical setting the experimental phenomenon initially observed in Belkin et al.

Original: (2019), Geiger et al.
Revised: (2019), Geiger et al.

Original: ; Spigler et al.
Revised: ; Spigler et al.

Original: Nakkiran et al.
Revised: Nakkiran et al.

Original: (2020a).
Revised: (2020a).

Original: Further derivations of triple or even multiple descents for the generalization error have also been performed d’Ascoli et al.
Revised: Further derivations of triple or even multiple descents for the generalization error have also been performed d’Ascoli et al.

Original: ; Nakkiran et al.
Revised: ; Nakkiran et al.

Original: (2020b)
Revised: (2020b)

Original: ; Chen et al.
Revised: ; Chen et al.

Revised: Richards et al.

Revised: ; Wu & Xu (2020).

Revised: Other aspects of multiples descents have been explored in

Revised: Lin & Dobriban (2021); Adlam & Pennington (2020b) also for the Neural tangent kernel in Adlam &

Original:  The tools in use come from modern random matrix theory Pennington & Worah (2017);
Revised: Pennington (2020a). The tools in use come from modern random matrix theory Pennington & Worah (2017);

Original: Rashidi Far et al.
Revised: Rashidi Far et al.

Original: ; Mingo & Speicher (2017), and statistical physics methods such as the replica method Engel & Van den Broeck (2001).
Revised: ; Mingo & Speicher (2017), and statistical physics methods such as the replica method Engel & Van den Broeck (2001).

Original: In this paper we are concerned with a line of research dedicated to the precise time-evolution of the generalization error under gradient ﬂow corroborating, among other things, the presence of epoch-wise descents structures Crisanti & Sompolinsky (2018); Bodin & Macris (2021) observed in
Revised: In this paper we are concerned with a line of research dedicated to the precise time-evolution of the generalization error under gradient ﬂow corroborating, among other things, the presence of epoch-wise descents structures Crisanti & Sompolinsky (2018); Bodin & Macris (2021) observed in

Original: Nakkiran et al.
Revised: Nakkiran et al.

Original: (2020a).
Revised: (2020a).

Original: We consider the gradient ﬂow dynamics for the training and generalisation errors in the setting of a Gaussian Covariate model, and develop analytical methods to track the whole time evolution.
Revised: We consider the gradient ﬂow dynamics for the training and generalisation errors in the setting of a Gaussian Covariate model, and develop analytical methods to track the whole time evolution.

Original: In particular, for inﬁnite times we get back the predictions of the least square estimator which have been thoroughly described in a similar model by Loureiro et al.
Revised: In particular, for inﬁnite times we get back the predictions of the least square estimator which have been thoroughly described in a similar model by Loureiro et al.

Original: In the next paragraphs we set-up the model together with a list of special realizations, and describe our main contributions.
Revised: In the next paragraphs we set-up the model together with a list of special realizations, and describe our main contributions.

Original: 1. M ODEL DESCRIPTION
Revised: 1. M ODEL DESCRIPTION

Original: Generative Data Model: In this paper, we use the so-called Gaussian Covariate model in a teacherstudent setting.
Revised: Generative Data Model: In this paper, we use the so-called Gaussian Covariate model in a teacherstudent setting.

Original: An observation in our data model is deﬁned through the realization of a gaussian vector z ∼ N (0 , 1 d I d ) .
Revised: An observation in our data model is deﬁned through the realization of a gaussian vector z ∼ N (0 , 1 d I d ) .

Original: The teacher and the student obtain their observations (or two different views of the world) with the vectors x ∈ R p  and ˆ x ∈ R p (cid:48) respectively, which are given by the application of two linear operations on z .
Revised: The teacher and the student obtain their observations (or two different views of the world) with the vectors x ∈ R p B and ˆ x ∈ R p A respectively, which are given by the application of two linear operations on z .

Original: In other words there exists two matrices B ∈ R d × p  and A ∈ R d × p (cid:48) such that x = B T z and ˆ x =
Revised: In other words there exists two matrices B ∈ R d × p B and A ∈ R d × p A such that x = B T z and ˆ x =

Original: A T z .
Revised: A T z .

Original: Note that the generated data can also be seen as the output of a generative 1-layer linear network.
Revised: Note that the generated data can also be seen as the output of a generative 1-layer linear network.

Original: In the following, the structure of A and B is pretty general as long as it remains independent of the realization z : the matrices may be random matrices or block-matrices of different natures and structures to capture more sophisticated models.
Revised: In the following, the structure of A and B is pretty general as long as it remains independent of the realization z : the matrices may be random matrices or block-matrices of different natures and structures to capture more sophisticated models.

Original: While the models we treat are deﬁned through appropriate A and B , we will often only need the structure of U
Revised: While the models we treat are deﬁned through appropriate A and B , we will often only need the structure of U

Original: =
Revised: =

Original: AA T and
Revised: AA T and

Original: V = BB T .
Revised: V = BB T .

Original: A direct connection can be made with the Gaussian Covariate model described in Loureiro et al.
Revised: A direct connection can be made with the Gaussian Covariate model described in Loureiro et al.

Original: (2021) which suggests considering directly observations ¯ x = ( x T , ˆ x T ) T ∼ N (0 , Σ) for a given covariance structure Σ .
Revised: (2021) which suggests considering directly observations ¯ x = ( x T , ˆ x T ) T ∼ N (0 , Σ) for a given covariance structure Σ .

Original: The spectral theorem provides the existence of orthonormal matrix O and diagonal D such that Σ
Revised: The spectral theorem provides the existence of orthonormal matrix O and diagonal D such that Σ

Original: =
Revised: =

Original: O T DO and D contains d non-zero eigenvalues in a squared block Dand p  + p (cid:48) − d zero eigenvalues.
Revised: O T DO and D contains d non-zero eigenvalues in a squared block D 1 and p A + p B − d zero eigenvalues.

Original: We can write D =
Revised: We can write D =

Original: J T D 1 J with J =
Revised: J T D 1 J with J =

Original: ( I d | 0 p  + p (cid:48) − d ) .
Revised: ( I d | 0 p A + p B − d ) .

Original: Therefore if we let z = 1 √ d D − 12 1 JO ¯ x which has variance 1 d I d , then upon noticing JJ T = I d and deﬁning ( A | B ) T = √ dO T J T D 12 

Revised: Therefore if we let z = 1 √ d D − 12 1 JO ¯ x which has variance 1 d I d , then upon noticing JJ T = I d and deﬁning ( A | B ) T = √ dO T J T D 12

Original: we ﬁnd ( A | B ) T z ∼ N (0 , Σ) .
Revised: we ﬁnd ( A | B ) T z ∼ N (0 , Σ) .

Original: The Gaussian Covariate model uniﬁes many different models as shown in Table 1.
Revised: The Gaussian Covariate model uniﬁes many different models as shown in Table 1.

Original: These special cases are all discussed in section 3 and Appendix D
Revised: These special cases are all discussed in section 3 and Appendix D

Original: [Table]
Revised: [Table]

Original: Ridgeless regression with signal r and noise σ
Revised: Ridgeless regression with signal r and noise σ

Original: [Table]
Revised: [Table]

Original:  Mismatched ridgeless regression withz signal r and noise σ and mismatch parameter γ with γ + γ (cid:48) =
Revised: Mismatched ridgeless regression withz signal r and noise σ and mismatch parameter γ with γ + γ (cid:48) =

Original: [Table]
Revised: [Table]

Original:  non-isotropic ridgless [Table] noiseless with a α polynomial distorsion of the inputs scalings
Revised: non-isotropic ridgless regression noiseless with a α polynomial distorsion of the inputs scalings

Original: [Table]
Revised: [Table]

Original:  ν (cid:113) dp
Revised: p O N × q O q × p σ (cid:113) dq I q     µ (cid:113) dp W ν (cid:113) dp

Original: I N
Revised: I N

Original: O q × N  
Revised: O q × N  

Original: Random features regression of a noisy linear function with W the random weights and ( µ, ν ) describing a non-linear activation function   [Equation] [Equation]
Revised: Random features regression of a noisy linear function with W the random weights and ( µ, ν ) describing a non-linear activation function    √ ω ··· 0√ ω 2 ··· 0 . ..

Original:  0√ [Equation] [Equation] [Equation]
Revised: ··· √ ω d       √ ω ··· 0√ ω 2 ··· 0 . ..

Original:   

Original: [Equation] [Equation] √ [Equation]
Revised: ··· √ ω d   

Original: [Equation]

Original: [Equation]

Original: d  

Original: Further Kernel methods
Revised: Further Kernel methods

Original: Learning task: We consider the problem of learning a linear teacher function f
Revised: Learning task: We consider the problem of learning a linear teacher function f

Original: d ( x ) =
Revised: d ( x ) =

Original: β ∗ T x with x and ˆ x sampled as deﬁned above, and with β ∗ ∈ R p a column vectors.
Revised: β ∗ T x with x and ˆ x sampled as deﬁned above, and with β ∗ ∈ R p a column vectors.

Original: This hidden vector β ∗ (to be learned) can potentially be a deterministic vector.
Revised: This hidden vector β ∗ (to be learned) can potentially be a deterministic vector.

Original: We suppose that we have n data-points ( z i , y i ) 1 ≤ i ≤ n with x i =
Revised: We suppose that we have n data-points ( z i , y i ) 1 ≤ i ≤ n with x i =

Original: Bz i , ˆ x i =
Revised: Bz i , ˆ x i =

Original: Az i .
Revised: Az i .

Original: This data can be represented as the n × d matrix Z ∈ R n × d where z Ti is the i -th row of Z , and the column vector vector Y ∈ R n with i -th entry
Revised: This data can be represented as the n × d matrix Z ∈ R n × d where z Ti is the i -th row of Z , and the column vector vector Y ∈ R n with i -th entry

Original: y i .
Revised: y i .

Original: Therefore, we have the matrix notation Y = ZBβ ∗ .
Revised: Therefore, we have the matrix notation Y = ZBβ ∗ .

Original: We can also set X = ZB so that Y =
Revised: We can also set X = ZB so that Y =

Original: Xβ ∗ .
Revised: Xβ ∗ .

Original: In the same spirit, we deﬁne the estimator of the student ˆ y β ( z ) =
Revised: In the same spirit, we deﬁne the estimator of the student ˆ y β ( z ) =

Original: β T x = z T Aβ .
Revised: β T x = z T Aβ .

Original: We note that in general the dimensions of β and β ∗ (i.e., p (cid:48) and p  ) are not necessarily equal as this depends on the matrices B and A .
Revised: We note that in general the dimensions of β and β ∗ (i.e., p A and p B ) are not necessarily equal as this depends on the matrices B and A .

Original: We have ˆ Y = ZAβ =
Revised: We have ˆ Y = ZAβ =

Original: ˆ Xβ for ˆ X = ZA .
Revised: ˆ Xβ for ˆ X = ZA .

Original: Training and test error:
Revised: Training and test error:

Original: We will consider the training error E λ train and test errors E gen with a regularization coefﬁcient λ ∈ R ∗ + deﬁned as
Revised: We will consider the training error E λ train and test errors E gen with a regularization coefﬁcient λ ∈ R ∗ + deﬁned as

Original: [Equation]
Revised: [Equation]

Original: It is well known that the least-squares estimator ˆ β = arg min H ( β ) is given by the Thikonov regression formula ˆ β λ
Revised: It is well known that the least-squares estimator ˆ β = arg min H ( β ) is given by the Thikonov regression formula ˆ β λ

Original: =
Revised: =

Original: ( ˆ X T ˆ X + λI ) − 1 ˆ X T Y and that in the limit λ → 0 , this estimator converges towards the ˆ β 0 given by the Moore-Penrose inverse ˆ β 

Revised: ( ˆ X T ˆ X + λI ) − 1 ˆ X T Y and that in the limit λ → 0 , this estimator converges towards the ˆ β 0 given by the Moore-Penrose inverse ˆ β

Original: =
Revised: =

Original: ( ˆ X T ˆ X )
Revised: ( ˆ X T ˆ X )

Original: + ˆ X T Y .
Revised: + ˆ X T Y .

Original: Gradient-ﬂow:
Revised: Gradient-ﬂow:

Original: We use the gradient-ﬂow algorithm to explore the evolution of the test error through time with ∂β t ∂t = − n 2 ∇ β E λ train ( β t ) .
Revised: We use the gradient-ﬂow algorithm to explore the evolution of the test error through time with ∂β t ∂t = − n 2 ∇ β E λ train ( β t ) .

Original: In practice, for numerical calculations we use the discrete-time version, gradient-descent, which is known to converge towards the aforementioned least-squares estimator provided a sufﬁciently small time-step (in the order of 1 λ max where λ max is the maximum eigenvalue of ˆ X T ˆ X ).
Revised: In practice, for numerical calculations we use the discrete-time version, gradient-descent, which is known to converge towards the aforementioned least-squares estimator provided a sufﬁciently small time-step (in the order of 1 λ max where λ max is the maximum eigenvalue of ˆ X T ˆ X ).

Original: The upfront coefﬁcient n on the gradient is used so that the test error scales with the dimension of the model and allows for considering the evolution in the limit n, d,p, p (cid:48) → + ∞ with a ﬁxed ratios nd , pd , p (cid:48) d .
Revised: The upfront coefﬁcient n on the gradient is used so that the test error scales with the dimension of the model and allows for considering the evolution in the limit n, d, p A , p B → + ∞ with a ﬁxed ratios nd , p A d , p B d .

Original: We will note φ = nd .
Revised: We will note φ = nd .

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: We provide a general uniﬁed framework covering multiple models in which we derive, in the asymptotic large size regime, the full time-evolution under gradient ﬂow dynamics of the training and generalization errors for teacher-student settings.
Revised: We provide a general uniﬁed framework covering multiple models in which we derive, in the asymptotic large size regime, the full time-evolution under gradient ﬂow dynamics of the training and generalization errors for teacher-student settings.

Original: In particular , in the inﬁnite time-limit we check 
Revised: In particular, in the inﬁnite time-limit we check that our equations

Original: that our equations reduce to those of Loureiro et al.
Revised: reduce to those of Loureiro et al.

Original: (2021) (as should be expected).
Revised: ) (as should be expected

Original:  Various special cases are illustrated in section 3, and among these a simpler re-derivation of the whole dynamics of the random featuresmodel Bodin & Macris (2021), the full dynamics for kernel methods, and situations exhibiting multiple descent curves both as a function of model parameters and time (See section 3.2 and Appendix D.2).
Revised: But with our results we now have the possibility to explore quantitatively potential advantages of different stopping times: indeed our formalism allows to compute the time derivative of the generalization curve at any point in time. Various special cases are illustrated in section 3, and among these a simpler re-derivation of the whole dynamics of the random features

Original: This analysis enables to detect potential early-stopping advantages in the dynamics of the gradient ﬂow.
Revised: model Bodin &

Original: We show that our equations can also capture the learning curves over realistic datasets such as 
Revised: Macris (2021) , the full dynamics for kernel methods, and situations exhibiting multiple descent curves both as a function

Original: MNIST with gradient descent (See section3.4 and Appendix D.5), extending further the results of
Revised: of model parameters and time (See section and Appendix D

Original:  Loureiro et al.
Revised: In particular, our analysis allows to design multiple descents with respect to the training epochs. We show that our equations can also capture the learning curves over realistic datasets such as MNIST with gradient descent (See section 3. and Appendix D. , extending further the results of Loureiro et al.

Original: (2021) to the time dependence of the curves.
Revised: (2021) to the time dependence of the curves.

Revised: This could be an interesting guideline for deriving scaling laws for large learning models.

Original: We use modern random matrix techniques, namely an improved version of the linear-pencil method - recently introduced in the machine learning community by Adlam et al.
Revised: We use modern random matrix techniques, namely an improved version of the linear-pencil method - recently introduced in the machine learning community by Adlam et al.

Original: (2019) - to derive asymptotic limits of traces of rational expressions involving random matrices.
Revised: - to derive asymptotic limits of traces of rational expressions involving random matrices.

Original: Furthermore we propose a new derivation an important ﬁxed point equation using Dyson brownian motion which, although non-rigorous, should be of independent interest (See Appendix E).
Revised: Furthermore we propose a new derivation an important ﬁxed point equation using Dyson brownian motion which, although non-rigorous, should be of independent interest (See Appendix E).

Original: Notations: We will use Tr d [ · ] ≡ lim d → + ∞ 1 d Tr [ · ] and similarly for Tr n [ · ] .
Revised: Notations: We will use Tr d [ · ] ≡ lim d → + ∞ 1 d Tr [ · ] and similarly for Tr n [ · ] .

Original: We also occasionally use N d ( v ) =
Revised: We also occasionally use N d ( v ) =

Original: lim d → + ∞ 1 d (cid:107) v (cid:107) 2 for a vector v (when the limit exists).
Revised: lim d → + ∞ 1 d (cid:107) v (cid:107) 2 for a vector v (when the limit exists).

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: We resort to the high-dimensional assumptions (see Bodin & Macris (2021) for similar assumptions).
Revised: We resort to the high-dimensional assumptions (see Bodin & Macris (2021) for similar assumptions).

Original: Assumptions 2.1 (High-Dimensional assumptions)
Revised: Assumptions 2.1 (High-Dimensional assumptions)

Original: In the high-dimensional limit, i.e, when d → + ∞ with all ratios nd , pd , p (cid:48) d ﬁxed, we assume the following
Revised: In the high-dimensional limit, i.e, when d → + ∞ with all ratios nd , p A d , p B d ﬁxed, we assume the following

Original: All the traces Tr d [ · ] , Tr n [ · ] concentrate on a deterministic value.
Revised: All the traces Tr d [ · ] , Tr n [ · ] concentrate on a deterministic value.

Original: There exists a sequence of complex contours Γ d ⊂ C enclosing the eigenvalues of the random matrix ˆ X T ˆ X ∈ R d × d but not enclosing − λ , and there exist also a ﬁxed contour Γ enclosing the support of the limiting (when d → + ∞ ) eigenvalue distribution of ˆ X T ˆ X
Revised: There exists a sequence of complex contours Γ d ⊂ C enclosing the eigenvalues of the random matrix ˆ X T ˆ X ∈ R d × d but not enclosing − λ , and there exist also a ﬁxed contour Γ enclosing the support of the limiting (when d → + ∞ ) eigenvalue distribution of ˆ X T ˆ X

Original: but not enclosing − λ .
Revised: but not enclosing − λ .

Original: With these assumptions in mind, we derive the precise time evolution of the test error in the highdimensional limit (see result 2.1) and similarly for the training error (see result 2.4).
Revised: With these assumptions in mind, we derive the precise time evolution of the test error in the highdimensional limit (see result 2.1) and similarly for the training error (see result 2.4).

Original: We will also assume that the results are still valid in the case λ = 0 as suggested in Mei & Montanari (2019).
Revised: We will also assume that the results are still valid in the case λ = 0 as suggested in Mei & Montanari (2019).

Original: 2. T IME EVOLUTION FORMULA FOR THE TEST ERROR
Revised: 2. T IME EVOLUTION FORMULA FOR THE TEST ERROR

Original: Result 2.

Revised: Result 2.

Original: The limiting test error time evolution for a random initialization β 0 such that N d ( β 0 ) =
Revised: The limiting test error time evolution for a random initialization β 0 such that N d ( β 0 ) =

Original: rand E [ β 0 ]
Revised: rand E [ β 0 ]

Original: = 0 is given by the following expression:
Revised: = 0 is given by the following expression:

Original: [Equation]
Revised: [Equation]

Original: with V ∗ =
Revised: with V ∗ =

Original: Bβ ∗ β ∗ T B T and c 0 =
Revised: Bβ ∗ β ∗ T B T and c 0 =

Original: Tr d [ V ∗ ] and:
Revised: Tr d [ V ∗ ] and:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f 1 ( x, y ) =
Revised: where f 1 ( x, y ) =

Original: f 2 ( x ) + f 2 ( y ) + ˜ f 1 ( x, y )
Revised: f 2 ( x ) + f 2 ( y ) + ˜ f 1 ( x, y )

Original: − c 0 and:
Revised: − c 0 and:

Original: [Equation]
Revised: [Equation]

Original:  and ζ z given by the self-consistent equation:
Revised: 7) and ζ z given by the self-consistent equation:

Original: [Equation]
Revised: [Equation]

Original: The former result can be expressed in terms of expectations w.r.t the joint limiting eigenvalue distributions of U and V ∗ when they commute with each other.
Revised: The former result can be expressed in terms of expectations w.r.t the joint limiting eigenvalue distributions of U and V ∗ when they commute with each other.

Original: Result 2.2 Besides, when U and V ∗ commute, let u, v ∗ be jointly-distributed according to U and
Revised: Result 2.2 Besides, when U and V ∗ commute, let u, v ∗ be jointly-distributed according to U and

Original: V ∗ eigenvalues respectively.
Revised: V ∗ eigenvalues respectively.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Notice also that in the limit t → ∞ :
Revised: Notice also that in the limit t → ∞ :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: which leads to the next result.
Revised: which leads to the next result.

Original: Result 2.3 In the limit t → ∞ , the limiting test error is given by ¯ E gen (+ ∞ ) =

Original: ˜ f 1 ( − λ, − λ ) .

Original: Remark 1 Notice that the matrix V ∗ is of rank one depending on the hidden vector β ∗ .
Revised: Remark 1 Notice that the matrix V ∗ is of rank one depending on the hidden vector β ∗ .

Original: However, it is also possible to calculate the average generalization (and training) error over a prior distribution β ∗ ∼ P ∗ .
Revised: However, it is also possible to calculate the average generalization (and training) error over a prior distribution β ∗ ∼ P ∗ .

Original: Averaging E β ∗ ∼P ∗ [ ¯ E gen ] propagates the expectation within E β ∗ ∼P ∗ [ B 0 ( t )] and
Revised: Averaging E β ∗ ∼P ∗ [ ¯ E gen ] propagates the expectation within E β ∗ ∼P ∗ [ B 0 ( t )] and

Original: E β ∗ ∼P ∗
Revised: E β ∗ ∼P ∗

Original: [ B 1 ( t )] , which propagates it further into the traces of E β ∗ ∼P ∗ [ ˜ f 1 ] and E β ∗ ∼P ∗ [ f 2 ] .
Revised: [ B 1 ( t )] , which propagates it further into the traces of E β ∗ ∼P ∗ [ ˜ f 1 ] and E β ∗ ∼P ∗ [ f 2 ] .

Original: In fact we ﬁnd:
Revised: In fact we ﬁnd:

Original: [Equation]
Revised: [Equation]

Original: In conclusion, we ﬁnd that E β ∗ ∼P ∗ [ ¯ E gen ] follows the same equations as ¯ E gen in result 2.1 with E β ∗ ∼P ∗ [ V ∗ ] instead of V ∗ .
Revised: In conclusion, we ﬁnd that E β ∗ ∼P ∗ [ ¯ E gen ] follows the same equations as ¯ E gen in result 2.1 with E β ∗ ∼P ∗ [ V ∗ ] instead of V ∗ .

Original: In the following, we will consider V ∗ without any distinction whether it comes from a speciﬁc vector β ∗ or averaged through a sample distribution P ∗ .
Revised: In the following, we will consider V ∗ without any distinction whether it comes from a speciﬁc vector β ∗ or averaged through a sample distribution P ∗ .

Original: Remark 

Revised: Remark

Original: In the particular case where U is diagonal, the matrix V ∗ can be replaced by the following diagonal matrix ˜ V ∗ which, in fact, commutes with U :[Equation]
Revised: In the particular case where U is diagonal, the matrix V ∗ can be replaced by the following diagonal matrix ˜ V ∗ which, in fact, commutes with U :

Original: [Equation]
Revised: [Equation]

Original: This comes essentially from the fact that given a diagonal matrix D and a non-diagonal matrix A , then [ DA ] ii =
Revised: This comes essentially from the fact that given a diagonal matrix D and a non-diagonal matrix A , then [ DA ] ii =

Original: [ D ] ii [ A ] ii .
Revised: [ D ] ii [ A ] ii .

Original: This is particularly helpful, and shows that in many cases the calculations of ˜ f
Revised: This is particularly helpful, and shows that in many cases the calculations of ˜ f

Original: 1 or f 2 remain tractable even for a deterministic β ∗ (see the example in Appendix D.3) .
Revised: 1 or f 2 remain tractable even for a deterministic β ∗ (see the example in Appendix D.3) .

Original: Remark 

Revised: Remark

Original: Sometimes U = AA T and V = BB T are more difﬁcult to handle than their dual counterparts U (cid:63) =
Revised: Sometimes U = AA T and V = BB T are more difﬁcult to handle than their dual counterparts U (cid:63) =

Original: φA T A and V (cid:63)
Revised: φA T A and V (cid:63)

Original: = φB T B together with the additional matrix Ξ = φA T B .
Revised: = φB T B together with the additional matrix Ξ = φA T B .

Original: The following expressions are thus very useful (See Appendix C):
Revised: The following expressions are thus very useful (See Appendix C):

Original: [Equation]
Revised: [Equation]

Original: In fact

Original: [Equation] when x = y = − λ (which corresponds to the limit when t → ∞ ) [Equation] these are the same expressions as (59) in Loureiro et al.
Revised: In fact, when x = y = − λ (which corresponds to the limit when t → ∞ ), these are the same expressions as (59) in Loureiro et al.

Original: (2021) with the appropriate change of variable λ (1 + V )
Revised: (2021) with the appropriate change of variable λ (1 + V )

Original: → ζ and ˜ f 1 → ρ + q − 2 m .
Revised: → ζ and ˜ f 1 → ρ + q − 2 m .

Original: 2. T IME EVOLUTION FORMULA FOR THE TRAINING ERROR
Revised: 2. T IME EVOLUTION FORMULA FOR THE TRAINING ERROR

Original: Result 2.

Revised: Result 2.

Original: The limiting training error time evolution is given by the following expression:
Revised: The limiting training error time evolution is given by the following expression:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  with:
Revised: 18) with:

Original: [Equation]
Revised: [Equation]

Original: where h 1 ( x, y ) =
Revised: where h 1 ( x, y ) =

Original: h 2 ( x ) + h 2 ( y ) + ˜ h 1 ( x, y ) − c 0 and with η z = − zζ z :
Revised: h 2 ( x ) + h 2 ( y ) + ˜ h 1 ( x, y ) − c 0 and with η z = − z ζ z :

Original: [Equation]
Revised: [Equation]

Original: Eventually, in the limit t → ∞ we ﬁnd:
Revised: Eventually, in the limit t → ∞ we ﬁnd:

Original: [Equation]
Revised: [Equation]

Original: Result 2.

Revised: Result 2.

Original: In the limit t → ∞ , we have the relation ¯ E 0 train (+ ∞ ) =
Revised: In the limit t → ∞ , we have the relation ¯ E 0 train (+ ∞ ) =

Original: η 2 − λ ¯ E gen (+ ∞ )
Revised: η 2 − λ ¯ E gen (+ ∞ )

Original: We notice the same proportionality factor η 2 − λ
Revised: We notice the same proportionality factor η 2 − λ

Original: = (cid:16) λ ζ ( − λ ) (cid:17) 2 as already stated in Loureiro et al.
Revised: = (cid:16) λ ζ ( − λ ) (cid:17) 2 as already stated in Loureiro et al.

Original: (2021), however interestingly, in the time evolution of the training error, such a factor is not valid as we have h 2 ( z ) (cid:54) =
Revised: (2021), however interestingly, in the time evolution of the training error, such a factor is not valid as we have h 2 ( z ) (cid:54) =

Original: η z f 2 ( z ) .
Revised: η z f 2 ( z ) .

Original:  A PPLICATIONS AND EXAMPLES
Revised:  A PPLICATIONS AND EXAMPLES

Original: We discuss some of the models provided in table 1 and some others in Appendix D.
Revised: We discuss some of the models provided in table 1 and some others in Appendix D.

Original: 3. R IDGELESS REGRESSION OF A NOISY LINEAR FUNCTION
Revised: 3. R IDGELESS REGRESSION OF A NOISY LINEAR FUNCTION

Original: Target function Consider the following noisy linear function y ( x ) =
Revised: Target function Consider the following noisy linear function y ( x ) =

Original: rx T β ∗ 0 + σ(cid:15) for some constant σ ∈ R + and (cid:15) ∼ N (0 , 1) , and a hidden vector β ∗ 0 ∼ N (0 , I p ) .
Revised: rx T β ∗ 0 + σ(cid:15) for some constant σ ∈ R + and (cid:15) ∼ N (0 , 1) , and a hidden vector β ∗ 0 ∼ N (0 , I p ) .

Original: Assume we have a data matrix
Revised: Assume we have a data matrix

Original: X ∈ R n × p .
Revised: X ∈ R n × p .

Original: In order to incorporate the noise in our structural matrix B , we consider an additional parameter q ( d ) that grows linearly with d and such that d = p + q .
Revised: In order to incorporate the noise in our structural matrix B , we consider an additional parameter q ( d ) that grows linearly with d and such that d = p + q .

Original: Let φ 

Revised: Let φ

Original: = np .
Revised: = np .

Original: Therefore φ = nd = np pd =
Revised: Therefore φ = n d = np p d

Original:  φ 0 ψ .
Revised: = φ 0 ψ .

Original: Also, we let β ∗ T =
Revised: Also, we let β ∗ T =

Original: ( β ∗ T 

Revised: ( β ∗ T

Original: | β T 1 ) ∼ N (0 , I p + q ) and we consider an average V ∗ over β ∗ .
Revised: | β T 1 ) ∼ N (0 , I p + q ) and we consider an average V ∗ over β ∗ .

Original: We construct the following block-matrix B and compute the averaged V ∗ as follow:
Revised: We construct the following block-matrix B and compute the averaged V ∗ as follow:

Original: [Equation]
Revised: [Equation]

Original: Now let’s consider the random matrix Z ∈ R n × d and split it into two sub-blocks Z = (cid:0)(cid:112) p d X | (cid:112) q d Σ (cid:1) .
Revised: Now let’s consider the random matrix Z ∈ R n × d and split it into two sub-blocks Z = (cid:0)(cid:112) pd X | (cid:112) qd Σ (cid:1) .

Original: The framework of the paper yields the following output vector:
Revised: The framework of the paper yields the following output vector:

Original: [Equation]
Revised: [Equation]

Original: 24) where ξ = Σ β ∗ 1 is used as a proxy for the noise (cid:15) .
Revised: where ξ = Σ β ∗ 1 is used as a proxy for the noise (cid:15) .

Original: Estimator Now let’s consider the linear estimator
Revised: Estimator Now let’s consider the linear estimator

Original: ˆ y t
Revised: ˆ y t

Original: = x T β t .
Revised: = x T β t .

Original: To capture the structure of this model, we use the following block-matrix A and compute the resulting matrix U :
Revised: To capture the structure of this model, we use the following block-matrix A and compute the resulting matrix U :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Therefore, it is straightforward to check that we have indeed: ˆ Y t = ZAβ t =
Revised: Therefore, it is straightforward to check that we have indeed: ˆ Y t = ZAβ t =

Original: Xβ t .
Revised: Xβ t .

Original: Analytic result In this speciﬁc example, U and V ∗ obviously commute and the result 2.2 can thus be used.
Revised: Analytic result In this speciﬁc example, U and V ∗ obviously commute and the result 2.2 can thus be used.

Original: First we derive the joint-distribution of the eigenvalues:
Revised: First we derive the joint-distribution of the eigenvalues:

Original: [Equation]
Revised: [Equation]

Original: In this speciﬁc example, we focus only on rederiving the high-dimensional generalization error without any regularization term ( λ = 0 ) for the minimum least-squares estimator.
Revised: In this speciﬁc example, we focus only on rederiving the high-dimensional generalization error without any regularization term ( λ = 0 ) for the minimum least-squares estimator.

Original: So we calculate ζ = ζ (0) as follows: ζ = ψ ζ 1 ψ φψ + ζ + 0 implies
Revised: So we calculate ζ = ζ (0) as follows: ζ = ψ ζ 1 ψ φψ + ζ + 0 implies

Original: ζ 2 + φ 0 ζ = ζ so ζ ∈ { 0 , 1 − φ 0 } .
Revised: ζ 2 + φ 0 ζ = ζ so ζ ∈ { 0 , 1 − φ 0 } .

Original: For ˜ f 1 we get:
Revised: For ˜ f 1 we get:

Original: [Equation]
Revised: [Equation]

Original: In fact, the expression can be simpliﬁed as follow (without the constants φ, ψ ):
Revised: In fact, the expression can be simpliﬁed as follow (without the constants φ, ψ ):

Original: [Equation]
Revised: [Equation]

Original: Using both solutions ζ = 0 or ζ = 1 − φ 0 yields the same results as in Hastie et al.
Revised: Using both solutions ζ = 0 or ζ = 1 − φ 0 yields the same results as in Hastie et al.

Original: ; Belkin et al.
Revised: ; Belkin et al.

Original: (2020) using 2.3:
Revised: (2020) using 2.3:

Original: [Equation]
Revised: [Equation]

Original: 3. N ON - ISOTROPIC RIDGELESS REGRESSION OF A NOISELESS LINEAR MODEL
Revised: 3. N ON - ISOTROPIC RIDGELESS REGRESSION OF A NOISELESS LINEAR MODEL

Original: Two double descents have been found in the non-isotropic ridge regression model in  Nakkiran et al.
Revised: Non-isotropic models have been studied in Dobriban & Wager (2018) and then also Wu & Xu (2020); Richards et al. ; Nakkiran et al.

Original: (2020b).
Revised: (2020b)

Revised: ; Chen et al.

Revised: (2021) where multiple-descents curve have been observed or engineered.

Original: In this section, we extend this idea to show that any number of descents can be generated and derive the precise curve of the generalization error as in Figure 1.
Revised: In this section, we extend this idea to show that any number of descents can be generated and derive the precise curve of the generalization error as in Figure 1.

Original: Target function We use the standard linear model y ( z ) =
Revised: Target function We use the standard linear model y ( z ) =

Original: z T β ∗ for a random β ∗ ∼ N (0 , I d ) .
Revised: z T β ∗ for a random β ∗ ∼ N (0 , I d ) .

Original: Therefore, we consider the matrix B
Revised: Therefore, we consider the matrix B

Original: = I d and thus V ∗ =
Revised: = I d and thus V ∗ =

Original: I d such that Y = ZBβ ∗ =
Revised: I d such that Y = ZBβ ∗ =

Original: Zβ ∗ .
Revised: Zβ ∗ .

Original: Estimator: Following the structure provided in table 1, the design a matrix A is a scalar matrix with p ∈ N ∗ sub-spaces of different scales spaced by a polynomial progression α − 12 i .
Revised: Estimator: Following the structure provided in table 1, the design a matrix A is a scalar matrix with p ∈ N ∗ sub-spaces of different scales spaced by a polynomial progression α − 12 i .

Original: In other words, the student is trained on a dataset with different scalings.
Revised: In other words, the student is trained on a dataset with different scalings.

Original: We thus have U =
Revised: We thus have U =

Original: A 2 and ˆ Y t = ZAβ t .
Revised: A 2 and ˆ Y t = ZAβ t .

Original: Analytic results We refer the reader to the Appendix D.2 for the calculation.
Revised: Analytic results We refer the reader to the Appendix D.2 for the calculation.

Original: Depending if φ is above or below 1 , ζ is the solution of the following equations: ζ = 0 or 1 = 1 p (cid:80) p − 1 i = {+
 φ+}
Revised: Depending if φ is above or below 1 , ζ is the solution of the following equations: ζ = 0 or 1 = 1 p (cid:80) p − 1 i = {+
 φ+}

Original: [Equation] ζ .
Revised: + α i ζ .

Original: In the over-parameterized regime ( φ < 1 ), the generalisation error is fully characterized by the equation:
Revised: In the over-parameterized regime ( φ < 1 ), the generalisation error is fully characterized by the equation:

Original: [Equation]
Revised: [Equation]

Original:  In the asymptotic limit α → ∞ , ζ can be approximated and thus we can derive an asymptotic expansion of ¯ E gen (+ ∞ ) for φ ∈ [0 , 1] \ kp Z where clearly, the multiple descents appear as roots of the denominator of the sum:
Revised: (cid:33) [Equation] In the asymptotic limit α → ∞ , ζ can be approximated and thus we can derive an asymptotic expansion of ¯ E gen (+ ∞ ) for φ ∈ [0 , 1] \ kp Z where clearly, the multiple descents appear as roots of the denominator of the sum:

Original: [Equation]
Revised: [Equation]

Original: Interestingly, we can see how these peaks are being formed with the time-evolution of the gradient ﬂow as in Figure 2 with one peak close to φ = 13 and the second one at φ = 23 .
Revised: Interestingly, we can see how these peaks are being formed with the time-evolution of the gradient ﬂow as in Figure 2 with one peak close to φ = 13 and the second one at φ = 23 .

Original: (Note that small λ requires more computational resources to have ﬁner resolution at long times, hence here the second peak develops fully after t = 10 4 ).
Revised: (Note that small λ requires more computational resources to have ﬁner resolution at long times, hence here the second peak develops fully after t = 10 4 ).

Original: It is worth noticing also the existence of multiple time-descent, in particular at φ = 1 with some "ripples" that can be observed even in the training error.
Revised: It is worth noticing also the existence of multiple time-descent, in particular at φ = 1 with some "ripples" that can be observed even in the training error.

Original: The eigenvalue distribution (See Appendix D.2.1) provides some insights on the existence of these phenomena.
Revised: The eigenvalue distribution (See Appendix D.2.1) provides some insights on the existence of these phenomena.

Original: As seen in Figure 3, the emergence of a spike is related to the rise of a new "bulk" of eigenvalues, which can be clearly seen around φ = 13 and φ = 23 here.
Revised: As seen in Figure 3, the emergence of a spike is related to the rise of a new "bulk" of eigenvalues, which can be clearly seen around φ = 13 and φ = 23 here.

Original: Note that there is some analogy for the generic double-descent phenomena described in Hastie et al.
Revised: Note that there is some analogy for the generic double-descent phenomena described in Hastie et al.

Original: (2019) where instead of two bulks, there is a mass in 0 which is arising.
Revised: (2019) where instead of two bulks, there is a mass in 0 which is arising.

Original: Furthermore, the existence of multiple bulks allow for multiple evolution at different scales (with the e − ( z + λ ) t terms) and thus enable the emergence of multiple epoch-wise peaks.
Revised: Furthermore, the existence of multiple bulks allow for multiple evolution at different scales (with the e − ( z + λ ) t terms) and thus enable the emergence of multiple epoch-wise peaks.

Original: 3. R ANDOM FEATURES REGRESSION
Revised: 3. R ANDOM FEATURES REGRESSION

Original: In this section, we show that we can derive the learning curves for the random features model introduced in Rahimi & Recht (2008), and as in Bodin & Macris (2021).
Revised: In this section, we show that we can derive the learning curves for the random features model introduced in Rahimi & Recht (2008), and we consider the setting described in Bodin & Macris (2021).

Original: We consider for W ∈ R p × N where ψ 0 =
Revised: In this setting, we deﬁne the random weight-matrix W ∈ R p × N where ψ 0 =

Original: Np such that W ij ∼ N (0 , 1 p ) and d = p + N + q and φ = nd , ψ = pd , and φ 0 = np = nd dp =
Revised: Np such that W ij ∼ N (0 , 1 p ) and d = p + N + q and φ = nd , ψ = pd , and φ 0 = np = nd dp =

Original: φψ (thus qd = 1 − (1 + ψ 0 ) ψ ).
Revised: φψ (thus qd = 1 − (1 + ψ 0 ) ψ ).

Original: So with Z = (cid:0)(cid:112) pd X | (cid:112) pd Ω | (cid:112) qd ξ (cid:1) , using the structures A and B from table 1 we have: ZA = µXW + ν Ω and ZB =
Revised: So with Z = (cid:0)(cid:112) pd X | (cid:112) pd Ω | (cid:112) qd ξ (cid:1) , using the structures A and B from tablewe have: ZA = µXW + ν Ω and ZB =

Original: X + σξ , hence the model:
Revised: X + σξ , hence the model:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The Gaussian equivalence principle described in Péché (2019) argues that in the high-dimensional limit, this model is similar to the point-wise application of some centered non-linear activation function σ with ˆ Y = σ ( XW ) β . With further calculation the can be found in Appendix D.4, a similar complete time derivation of the random feature regression can be performed without the large linear-pencil  suggested in Bodin & Macris (2021). 
Revised: With further calculation that can be found in Appendix D.4, a similar complete time derivation of the random feature regression can be performed with a much smaller linear-pencil than the one suggested in Bodin & Macris (2021). As stated in this former work, the curves derived from this formula track the same training and test error in the high-dimensional limit as the model with the point-wise application of a centered non-linear activation function f ∈ L 2 ( e − x 2 2 d x ) with ˆ Y = 1 √ p f ( √ pXW ) β .

Revised: More precisely, with the inner-product deﬁned such that for any function g ∈ L 2 ( e − x 2 2 d x ) , (cid:104) f, g (cid:105) =

Original:  3. T OWARDS REALISTIC DATASETS
Revised: E x ∼N (0 , 1) [ f ( x ) g ( x )] , we derive the equivalent model parameters ( µ, ν ) with µ = (cid:104) f, H e 1 (cid:105) , ν 2 = (cid:104) f, f (cid:105)− µ 2 while having the centering condition (cid:104) f, H e 0 (cid:105) = 0 where ( H e n ) is the Hermite polynomial basis. This transformation is dubbed the Gaussian equivalence principle and has been observed and rigorously proved under weaker conditions in Pennington & Worah (2017); Péché (2019); Hu & Lu (2022), and since then has been applied more broadly for instance in Adlam & Pennington (2020a). 3. T OWARDS REALISTIC DATASETS

Original: As stated in Loureiro et al.
Revised: As stated in Loureiro et al.

Original: (2021), the training and test error of realistic datasets can also be captured.
Revised: (2021), the training and test error of realistic datasets can also be captured.

Original: In this example we track the MNIST dataset and focus on learning the parity of the images ( y = + for even numbers and y = − 1 for odd-numbers).
Revised: In this example we track the MNIST dataset and focus on learning the parity of the images ( y = + for even numbers and y = − 1 for odd-numbers).

Original: We refer to Appendix D.5 for the technical details to obtain Figures 4 and 5, and other examples.
Revised: We refer to Appendix D.5 for thorough discussions of Figures 4 and 5 as well as technical details to obtain them, and other examples.

Original: Besides the learning curve proﬁle at t = + ∞ , the full theoretical time evolution is predicted and matches the experimental runs.
Revised: Besides the learning curve proﬁle at t = + ∞ , the full theoretical time evolution is predicted and matches the experimental runs.

Original: In particular, the rise of the double-descent phenomenon is observed through time.
Revised: In particular, the rise of the double-descent phenomenon is observed through time.

Original:  A G RADIENT FLOW CALCULATIONS
Revised:  C ONCLUSION

Revised: The time-evolution can also be investigated using the dynamical mean ﬁeld theory (DMFT) from statistical mechanics. We refer the reader to the book Parisi et al. (2020) and a series of recent works

Revised: Sompolinsky et al.

Revised: ; Crisanti & Sompolinsky (2018);

Revised: Agoritsas et al.

Revised: ; Mignacco et al.

Revised: (2020; 2021) for an overview of this tool.

Revised: This method is a priori unrelated to ours and yields a set of non-linear integro-differential equations for time correlation functions which are in general not solvable analytically and one has to resort to a numerical solution.

Revised: It would be interesting to understand if for the present model the DMFT equations can be reduced to our set of algebraic equations.

Revised: We believe it can be a fruitful endeavor to compare in detail the two approaches: the one based on DMFT and the one based on random matrix theory tools and Cauchy integration formulas.

Revised: Another interesting direction which came to our knowledge recently is the one taken in Lu & Yau (2022);

Revised: Hu & Lu (2022) and in Misiakiewicz (2022); Xiao & Pennington (2022), who study the high-dimensional polynomial regime where n ∝ d κ for a ﬁxed κ .

Revised: In particular, it is becoming notorious that changing the scaling can yield additional descents.

Revised: This regime is out of the scope of the present work but it would be desirable to explore if the linear-pencils and the random matrix tools that we extensively use in this work can extend to these cases.

Revised: A G RADIENT FLOW CALCULATIONS

Original: In this section, we derive the main equations for the gradient ﬂow algorithm, and derive and set of
Revised: In this section, we derive the main equations for the gradient ﬂow algorithm, and derive and set of

Original: Cauchy integration formula involving the limiting traces of large matrices.
Revised: Cauchy integration formula involving the limiting traces of large matrices.

Original: The calculation factoring out Z in the limit d → ∞ is pursued in the next section.
Revised: The calculation factoring out Z in the limit d → ∞ is pursued in the next section.

Original: First, we recall and expand the training error function in 1:
Revised: First, we recall and expand the training error function in 1:

Original: [Equation]
Revised: [Equation]

Original: Let K =
Revised: Let K =

Original: ( ˆ X T ˆ X + λI ) − 1 =
Revised: ( ˆ X T ˆ X + λI ) − 1 =

Original: ( A T Z T ZA + λI ) − 1 which is invertible for λ > 0 .
Revised: ( A T Z T ZA + λI ) − 1 which is invertible for λ > 0 .

Original: Therefore, we can write the gradient of the training error for any β as:
Revised: Therefore, we can write the gradient of the training error for any β as:

Original: [Equation]
Revised: [Equation]

Original: The gradient ﬂow equations reduces to a ﬁrst order ODE
Revised: The gradient ﬂow equations reduces to a ﬁrst order ODE

Original: [Equation]
Revised: [Equation]

Original: The solution can be completely expressed using L t =
Revised: The solution can be completely expressed using L t =

Original: ( I − exp( − tK − 1 )) as
Revised: ( I − exp( − tK − 1 )) as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the following two subsections, we will focus on deriving an expression of the time evolution of the test error and training error using these equations averaged over the a centered random vector βsuch that r
Revised: In the following two subsections, we will focus on deriving an expression of the time evolution of the test error and training error using these equations averaged over the a centered random vector βsuch that r

Original: 20 = N d ( β 0 ) 2 .
Revised: 20 = N d ( β 0 ) 2 .

Original: A. T EST ERROR
Revised: A. T EST ERROR

Original: As above, the test error can be expanded using the fact that on N 0 = N (0 , 1 d ) , we have the identity
Revised: As above, the test error can be expanded using the fact that on N 0 = N (0 , 1 d ) , we have the identity

Original: E z ∼N 0 [
Revised: E z ∼N 0 [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: So expanding the ﬁrst term yields
Revised: So expanding the ﬁrst term yields

Original: [Equation]
Revised: [Equation]

Original: 47) while the second term yields
Revised: 47) while the second term yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let’s consider now the high-dimensional limit ¯ E gen ( t ) =
Revised: Let’s consider now the high-dimensional limit ¯ E gen ( t ) =

Original: lim d → + ∞ E gen ( β t ) .
Revised: lim d → + ∞ E gen ( β t ) .

Original: We further make the underlying assumption that the generalisation error concentrates on its mean with β 0 , that is to say: ¯
Revised: We further make the underlying assumption that the generalisation error concentrates on its mean with β 0 , that is to say: ¯

Original: [Equation]
Revised: [Equation]

Original: Let V ∗ =
Revised: Let V ∗ =

Original: Bβ ∗ β ∗ T B T and c 0 =
Revised: Bβ ∗ β ∗ T B T and c 0 =

Original: Tr d [ V ∗ ] , then using the former expanded terms in 

Revised: Tr d [ V ∗ ] , then using the former expanded terms in

Original: we ﬁnd the expression
Revised: we ﬁnd the expression

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: So ¯ E gen ( t ) =
Revised: So ¯ E gen ( t ) =

Original: c 0 + r 20 B 0 ( t ) + B 1 ( t ) with:
Revised: c 0 + r 20 B 0 ( t ) + B 1 ( t ) with:

Original: [Equation]
Revised: [Equation]

Original: Let K ( z ) =
Revised: Let K ( z ) =

Original: ( ˜ X T ˜ X − zI ) − 1 the resolvent of ˜ X T ˜ X , and let’s have the convention K = K ( − λ ) to remain consistent with the previous formula.
Revised: ( ˜ X T ˜ X − zI ) − 1 the resolvent of ˜ X T ˜ X , and let’s have the convention K = K ( − λ ) to remain consistent with the previous formula.

Original: Then for any holomorphic functional f : U → C deﬁned on an open set U which contains the spectrum of ˜ X T ˜ X , with Γ a contour in C enclosing the spectrum of ˜ X T ˜ X but not the poles of f , we have with the extension of f onto C n × n : f ( ˜ X T ˜ X )
Revised: Then for any holomorphic functional f : U → C deﬁned on an open set U which contains the spectrum of ˜ X T ˜ X , with Γ a contour in C enclosing the spectrum of ˜ X T ˜ X but not the poles of f , we have with the extension of f onto C n × n : f ( ˜ X T ˜ X )

Original: =
Revised: =

Original: −2 iπ (cid:72) Γ f ( z ) K ( z )d z .
Revised: −2 iπ (cid:72) Γ f ( z ) K ( z )d z .

Original: For instance, we can apply it for the following expression:
Revised: For instance, we can apply it for the following expression:

Original: [Equation]
Revised: [Equation]

Original: So we can generalize this idea to each trace and rewrite B 1 ( t ) and B 0 ( t ) with
Revised: So we can generalize this idea to each trace and rewrite B 1 ( t ) and B 0 ( t ) with

Original: [Equation]
Revised: [Equation]

Original: where we introduce the set of functions f 1 ( x, y ) , f 2 ( z ) and f 0 ( z )
Revised: where we introduce the set of functions f 1 ( x, y ) , f 2 ( z ) and f 0 ( z )

Original: [Equation]
Revised: [Equation]

Original: Let G ( x ) =
Revised: Let G ( x ) =

Original: ( UZ T Z − xI ) −
Revised: ( UZ T Z − xI ) −

Original: [Equation]
Revised: [Equation]

Original: using the push-through identity
Revised: using the push-through identity

Original: [Equation]
Revised: [Equation]

Original: it is straightforward that AK ( z ) A
Revised: it is straightforward that AK ( z ) A

Original: =
Revised: =

Original: G ( z ) U =
Revised: G ( z ) U =

Original: UG ( z ) T .
Revised: UG ( z ) T .

Original: This help us reduce further the expression of f 1 into smaller terms which will be easier to handle with linear-pencils later on
Revised: This help us reduce further the expression of f 1 into smaller terms which will be easier to handle with linear-pencils later on

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly with f 2 and f 0 , they can be rewritten as
Revised: Similarly with f 2 and f 0 , they can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Hence in fact the deﬁnition ˜ f
Revised: Hence in fact the deﬁnition ˜ f

Original: 1 ( x, y ) =
Revised: 1 ( x, y ) =

Original: xy
Revised: xy

Original: Tr d (cid:2) G ( x ) V ∗ G ( y ) T (cid:3) such that
Revised: Tr d (cid:2) G ( x ) V ∗ G ( y ) T (cid:3) such that

Original: [Equation]
Revised: [Equation]

Original: At this point, the equations provided by 57 are valid for any realization Z in the limit d → ∞ .
Revised: At this point, the equations provided by 57 are valid for any realization Z in the limit d → ∞ .

Original: We will see in the next section how to simplify these terms by factoring out Z .
Revised: We will see in the next section how to simplify these terms by factoring out Z .

Original: A. T RAINING ERROR
Revised: A. T RAINING ERROR

Original: Similar formulas can be derived for the training error.
Revised: Similar formulas can be derived for the training error.

Original: For the sake of simplicity, we provide a formula to track the training error without the regularization term, that is to say E 0 train ( β t ) (as in Loureiro et al.
Revised: For the sake of simplicity, we provide a formula to track the training error without the regularization term, that is to say E 0 train ( β t ) (as in Loureiro et al.

Original: (2021)) while still minimizing the loss E λ train ( β t ) .
Revised: (2021)) while still minimizing the loss E λ train ( β t ) .

Original: So using the expanded expression 34, and considering the high-dimensional assumption with concentration ¯ E 0 train ( t ) :=
Revised: So using the expanded expression 34, and considering the high-dimensional assumption with concentration ¯ E 0 train ( t ) :=

Original: lim d → + ∞ E train ( β t ) =
Revised: lim d → + ∞ E train ( β t ) =

Original: lim d → + ∞ E β 

Revised: lim d → + ∞ E β

Original: [ E train ( β t )] we have
Revised: [ E train ( β t )] we have

Original: [Equation]
Revised: [Equation]

Original: First of all
Revised: First of all

Original: [Equation]
Revised: [Equation]

Original: standard random matrix results (for instance see Rubio & Mestre (2011)) assert the result
Revised: standard random matrix results (for instance see Rubio & Mestre (2011)) assert the result

Original: [Equation]
Revised: [Equation]

Original: [ V ∗ ] = φc 0 .
Revised: [ V ∗ ] = φc 0 .

Original: This result can also be derived under our random matrix theory framework, for completeness we provide this calculation in C.2.
Revised: This result can also be derived under our random matrix theory framework, for completeness we provide this calculation in C.2.

Original: Therefore, we can deﬁne H 0 ( t ) and H 1 ( t ) such that
Revised: Therefore, we can deﬁne H 0 ( t ) and H 1 ( t ) such that

Original: [Equation]
Revised: [Equation]

Original: where we have the traces
Revised: where we have the traces

Original: [Equation]
Revised: [Equation]

Original: And using the functional calculus argument with Cauchy integration formula over the same contour Γ we ﬁnd
Revised: And using the functional calculus argument with Cauchy integration formula over the same contour Γ we ﬁnd

Original: [Equation]
Revised: [Equation]

Original: Where we use the traces (which only contain algebraic expression of matrices):
Revised: Where we use the traces (which only contain algebraic expression of matrices):

Original: [Equation]
Revised: [Equation]

Original: The expression of h 1 can be reduced to smaller terms as before with f
Revised: The expression of h 1 can be reduced to smaller terms as before with f

Original: [Equation]
Revised: [Equation]

Original: 87) and similarly with h
Revised: 87) and similarly with h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 91) and similarly with h
Revised: 91) and similarly with h

Original: [Equation]
Revised: [Equation]

Original: We can also deﬁne the term ˜ h 1 ( x, y ) =
Revised: We can also deﬁne the term ˜ h 1 ( x, y ) =

Original: xy Tr n (cid:2) ZG ( y ) V ∗ G ( x ) T Z T (cid:3) so that:
Revised: xy Tr n (cid:2) ZG ( y ) V ∗ G ( x ) T Z T (cid:3) so that:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B T EST ERROR AND TRAINING ERROR LIMITS WITH LINEAR PENCILS
Revised: B T EST ERROR AND TRAINING ERROR LIMITS WITH LINEAR PENCILS

Original: In this section we compute a set of self-consistent equation to derive the high-dimensional evolution of the training and test error.
Revised: In this section we compute a set of self-consistent equation to derive the high-dimensional evolution of the training and test error.

Original: We refer to Appendix E for the deﬁnition and result statements concerning the linear pencils.
Revised: We refer to Appendix E for the deﬁnition and result statements concerning the linear pencils.

Original: We will derive essentially two linear-pencils of size 6 × 6 and 4 × 4 which will enable us to calculate the limiting values for ˜ f 1 , f 2 , f 0 for the test error, and ˜ h 1 , h 2 , h 0 for the training error.
Revised: We will derive essentially two linear-pencils of size 6 × 6 and 4 × 4 which will enable us to calculate the limiting values for ˜ f 1 , f 2 , f 0 for the test error, and ˜ h 1 , h 2 , h 0 for the training error.

Revised: Note that these block-matrices are derived essentially by observing the recursive application of the block-matrix inversion formula and manipulating it so as to obtain the desired result.

Revised: Compared to other works such as Bodin & Macris (2021); Adlam & Pennington (2020a), our approach yields smaller sizes of linear-pencils to handle, which in turn yields a smaller set of algebraic equations.

Revised: One of the ingredient of our method consists in considering a multiple-stage approach where the trace of some random blocks can be calculated in different parts (See the random feature model for example in Appendix D.4).

Revised: However, the question of ﬁnding the simplest linear-pencil remains open and interesting to investigate.

Original: B. L IMITING TRACES OF THE TEST ERROR L IMITING TRACE FOR ˜ f 1 AND f
Revised: B. L IMITING TRACES OF THE TEST ERROR L IMITING TRACE FOR ˜ f 1 AND f

Original: We construct a linear-pencil M 1 as follow (with Z the random matrix into consideration)
Revised: We construct a linear-pencil M 1 as follow (with Z the random matrix into consideration)

Original: M 1 =      − yI
Revised: M 1 =     − yI

Original: Z T
Revised: Z T

Original: Z
Revised: Z

Original: I
Revised: I

Original: U
Revised: U

Original: I − xI
Revised: I − xI

Original: U − xyV ∗
Revised: U − xyV ∗

Original: Z T
Revised: Z T

Original: I
Revised: I

Original: Z
Revised: Z

Original: I       (96)
Revised: I      (96)

Original: The inverse of this block-matrix contains the terms in the traces of ˜ f
Revised: The inverse of this block-matrix contains the terms in the traces of ˜ f

Original: 1 and f 0 .
Revised: 1 and f 0 .

Original: To see this, let’s calculate the inverse of M 1 by splitting it ﬁrst into other "ﬂattened" blocks:M [Equation] = (cid:18)
Revised: To see this, let’s calculate the inverse of M 1 by splitting it ﬁrst into other "ﬂattened" blocks:

Original: [Equation]

Original: D (cid:19)

Original: [Equation]

Original: = (cid:18)

Original: [Equation]

Original: B

Original: [Equation]
Revised: [Equation]

Original: (cid:19) (97)

Original: Where B x and D are given by
Revised: Where B x and D are given by

Original: B x =   − xI U Z T I
Revised: [Equation]

Original: Z I   D = (cid:32) − xyV ∗ 0 0 0(cid:33) (98) then to calculate the inverse of B x , notice ﬁrst its lower right-hand sub-block has inverse(cid:18) Z T
Revised: then to calculate the inverse of B x , notice ﬁrst its lower right-hand sub-block has inverse

Original: [Equation]
Revised: [Equation]

Original: Which lead us to the following inverse using the block-matrix inversion formula (the dotted terms aren’t required):
Revised: Which lead us to the following inverse using the block-matrix inversion formula (the dotted terms aren’t required):

Original: [Equation]
Revised: [Equation]

Original: With g (cid:104) ij [Equation] d the trace of the squared sub-block ( M − 1 1 ) (cid:104) ij (cid:105) divided by the size of the block ( ij ) , we ﬁnd the desired functions
Revised: With g (cid:104) ij (cid:105) d the trace of the squared sub-block ( M − 1 1 ) (cid:104) ij (cid:105) divided by the size of the block ( ij ) , we ﬁnd the desired functions

Original: [Equation]
Revised: [Equation]

Original: Let’s now consider g the limiting value of g d , and calculate the mapping η ( g ) :
Revised: Let’s now consider g the limiting value of g d , and calculate the mapping η ( g ) :

Original: η(
Revised: η

Original: [Equation]
Revised: [Equation]

Original: =    {+0 
+}
Revised: =    {+0 
+}

Original: [Equation] [Equation]{+φg (cid:104) 26 (cid:105)   
+}
Revised: {+φg (cid:104) 26 (cid:105)   
+}

Original: [Equation] (cid:105)  0    [-
 [Equation]-] (cid:105)  

Revised: g (cid:104) 15 (cid:105)  0    [-
 [Equation]-]0  φg (cid:104) 62 (cid:105) 0

Original: [Equation]

Original:  (cid:104) [Equation] (cid:105)  g (cid:104) {+51 (cid:105) 
+}
Revised: φg (cid:104) 22 (cid:105)  g (cid:104) {+51 (cid:105) 
+}

Original: [Equation]

Original: [-(cid:105) 
-] g (cid:104) 11 (cid:105)   (103)
Revised: [-(cid:105) 
-] g (cid:104) 11 (cid:105)   (103)

Original: So we can calculate the matrix Π( M 1 ) such that the elements of g are the limiting trace of the squared sub-blocks of (Π( M 1 ))
Revised: So we can calculate the matrix Π( M 1 ) such that the elements of g are the limiting trace of the squared sub-blocks of (Π( M 1 ))

Original: − 1 (divided by the block-size) following the steps of the result in App.
Revised: − 1 (divided by the block-size) following the steps of the result in App.

Original: E: Π( M 1 ) =     − yI − φg (cid:104) 26 (cid:105)I      (
Revised: E: Π( M 1 ) =     − yI − φg (cid:104) 26 (cid:105)

Original: [Equation] g (cid:104) 15 (cid:105) )
Revised: I      (1 − g (cid:104) 15 (cid:105) )

Original: I
Revised: I

Original: U
Revised: U

Original: I − xI
Revised: I − xI

Original: U −

Original: [Equation] ∗  [Equation] 62 (cid:105)
Revised: U − xyV ∗  − φg (cid:104) 62 (cid:105)

Original: I
Revised: I

Original: I −

Original: [Equation] (cid:104) 22 (cid:105) I  ( [Equation] g (cid:104) 51 (cid:105) )
Revised: I − φg (cid:104) 22 (cid:105) I  (1 − g (cid:104) 51 (cid:105) )

Original: I   − g (cid:104) 11 (cid:105) I   (104)
Revised: I   − g (cid:104) 11 (cid:105) I   (104)

Original: Therefore, there remains to compute the inverse of Π( M 1 ) .
Revised: Therefore, there remains to compute the inverse of Π( M 1 ) .

Original: We split again Π( M 1 ) as ﬂattened sub-blocks to make the calculation easier
Revised: We split again Π( M 1 ) as ﬂattened sub-blocks to make the calculation easier

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: With the three block-matrices˜
Revised: With the three block-matrices

Original: [Equation]
Revised: [Equation]

Original: I

Original: [Equation]

Original: [Equation]

Original: xI

Original: U − g (cid:104) 26 (cid:105) φI

Original: I (1 − g (cid:104) 15 (cid:105) )

Original: I 

Original: [Equation]

Original: A straightforward application of the block-matrix inversion formula yields inverse of ˜ B x
Revised: A straightforward application of the block-matrix inversion formula yields inverse of ˜ B x

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we retrieve the following close set of equations:
Revised: we retrieve the following close set of equations:

Original: [Equation]
Revised: [Equation]

Original: [Equation] These equations can be simpliﬁed slightly by removing g (cid:104) [Equation]
Revised: These equations can be simpliﬁed slightly by removing g (cid:104) 22 (cid:105) , g (cid:104) 26 (cid:105) and introducing q (cid:104) 15 (cid:105) :

Original: [Equation]
Revised: [Equation]

Original: and introducing q [Equation] [Equation]
Revised: Let ζ x = − xq (cid:104) 15 (cid:105)

Original: [Equation]
Revised: [Equation]

Original:  − 
Revised: or by symmetry ζ y = − yq (cid:104) 51 (cid:105)

Original: [Equation]
Revised: [Equation]

Original: Let ζ x = − xq (cid:104) 15 (cid:105) , or by symmetry ζ y = − yq (cid:104) 51 (cid:105) , then using the fact that ˜ f 1 ( x, y )
Revised: then using the fact that ˜ f 1 ( x, y )

Original: = − g (cid:104) 11 (cid:105) and f 0 ( x )
Revised: = − g (cid:104) 11 (cid:105) and f 0 ( x )

Original: = − g (cid:104) 15 (cid:105)
Revised: = − g (cid:104) 15 (cid:105)

Original: we ﬁnd the system of equations
Revised: we ﬁnd the system of equations

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: Remark:
Revised: Remark:

Original: As a byproduct of this analysis, notice the term g (cid:104) 62 (cid:105) =
Revised: As a byproduct of this analysis, notice the term g (cid:104) 62 (cid:105) =

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: xζ x .
Revised: xζ x .

Original: In fact we have:
Revised: In fact we have:

Original: [Equation]
Revised: [Equation]

Original: So if we let m ( x ) =
Revised: So if we let m ( x ) =

Original: Tr n (cid:104) ( ˆ X ˆ X T − xI ) − 1 (cid:105) the trace of the resolvent of the student data matrix, we ﬁnd that m ( x ) =
Revised: Tr n (cid:104) ( ˆ X ˆ X T − xI ) − 1 (cid:105) the trace of the resolvent of the student data matrix, we ﬁnd that m ( x ) =

Original: ζ − 1 x .
Revised: ζ − 1 x .

Original: This can be useful for analyzing the eigenvalues as in Appendix D.2.1.
Revised: This can be useful for analyzing the eigenvalues as in Appendix D.2.1.

Original: L IMITING TRACE FOR f
Revised: L IMITING TRACE FOR f

Original: As before, we construct a second linear-pencil M 2 with Z the random matrix component into consideration
Revised: As before, we construct a second linear-pencil M 2 with Z the random matrix component into consideration

Original: M
Revised: M

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The former ﬂattened block B z can be recognized in the lower right-hand side of M 2 , thus we can use the block matrix-inversion formula and get:
Revised: The former ﬂattened block B z can be recognized in the lower right-hand side of M 2 , thus we can use the block matrix-inversion formula and get:

Original: [Equation]
Revised: [Equation]

Original:  (125)
Revised:  (125)

Original: Now it is clear that we can express f 2 ( z )
Revised: Now it is clear that we can express f 2 ( z )

Original: =
Revised: =

Original: c 0 + lim d
Revised: c 0 + lim d

Original: → + ∞ g (cid:104) 21 (cid:105) d .
Revised: → + ∞ g (cid:104) 21 (cid:105) d .

Original: Following the steps of App.
Revised: Following the steps of App.

Original: E we calculate the mapping
Revised: E we calculate the mapping

Original: [Equation]
Revised: [Equation]

Original: Which in returns enable us to calculate Π( M 2 )
Revised: Which in returns enable us to calculate Π( M 2 )

Original: [Equation]
Revised: [Equation]

Original: To compute the inverse of Π( M 2 ) , the block-matrix is ﬁrst split with the sub-block ˜ B z deﬁned as follow
Revised: To compute the inverse of Π( M 2 ) , the block-matrix is ﬁrst split with the sub-block ˜ B z deﬁned as follow

Original: [Equation]
Revised: [Equation]

Original: A straightforward application of the block-matrix inversion formula yields the inverse of ˜ B z :
Revised: A straightforward application of the block-matrix inversion formula yields the inverse of ˜ B z :

Original: [Equation]
Revised: [Equation]

Original: Hence we can derive the inverse
Revised: Hence we can derive the inverse

Original: [Equation]
Revised: [Equation]

Original: Eventually, using the ﬁxed-point result on linear-pencils, we derive the set of equations
Revised: Eventually, using the ﬁxed-point result on linear-pencils, we derive the set of equations

Original: [Equation]
Revised: [Equation]

Original: In fact, it is a straightforward to see that g (cid:104) 23 (cid:105) , g (cid:104) 34 (cid:105) follows the same equations as the former g (cid:104) 15 (cid:105) , g (cid:104) 26 (cid:105) in the previous subsection, therefore g (cid:104) 23 (cid:105) =
Revised: In fact, it is a straightforward to see that g (cid:104) 23 (cid:105) , g (cid:104) 34 (cid:105) follows the same equations as the former g (cid:104) 15 (cid:105) , g (cid:104) 26 (cid:105) in the previous subsection, therefore g (cid:104) 23 (cid:105) =

Original: g (cid:104) 15 (cid:105) =
Revised: g (cid:104) 15 (cid:105) =

Original: 1 − q (cid:104) 15 (cid:105) =
Revised: 1 − q (cid:104) 15 (cid:105) =

Original: 1 + ζ z z , and thus g (cid:104) 34 (cid:105) =
Revised: 1 + ζ z z , and thus g (cid:104) 34 (cid:105) =

Original: − zζ z Eventually we get g (cid:104) 21 (cid:105) =
Revised: − zζ z Eventually we get g (cid:104) 21 (cid:105) =

Original: − Tr d (cid:2) ζ z V ∗ ( φU + ζ z I ) − 1 (cid:3) so in the limit d → ∞ :
Revised: − Tr d (cid:2) ζ z V ∗ ( φU + ζ z I ) − 1 (cid:3) so in the limit d → ∞ :

Original: [Equation]
Revised: [Equation]

Original: B. L IMITING TRACES FOR THE TRAINING ERROR L IMITING TRACE FOR h
Revised: B. L IMITING TRACES FOR THE TRAINING ERROR L IMITING TRACE FOR h

Original: A careful attention to the linear-pencil M 1 shows that the terms in the trace of ˜ h 1 are actually given by the location g (cid:104) 22 (cid:105) .
Revised: A careful attention to the linear-pencil M 1 shows that the terms in the trace of ˜ h 1 are actually given by the location g (cid:104) 22 (cid:105) .

Original: We have to be careful also of the fact that ( M − 1 1 ) (cid:104) 22 (cid:105) is a block matrix of size n × n , so it is already divided by the size n (and not d ).
Revised: We have to be careful also of the fact that ( M − 1 1 ) (cid:104) 22 (cid:105) is a block matrix of size n × n , so it is already divided by the size n (and not d ).

Original: Hence we simply have with η z = − zζ z :
Revised: Hence we simply have with η z = − zζ z :

Original: [Equation]
Revised: [Equation]

Original: L IMITING TRACE FOR h
Revised: L IMITING TRACE FOR h

Original: In the case of h 2 , we need the speciﬁc term provided by the linear-pencil M 2 by the location g (cid:104) 41 (cid:105) with φh 2 ( z ) =
Revised: In the case of h 2 , we need the speciﬁc term provided by the linear-pencil M 2 by the location g (cid:104) 41 (cid:105) with φh 2 ( z ) =

Original: c 0 φ + g (cid:104) 

Revised: c 0 φ + g (cid:104)

Original: (cid:105)
Revised: (cid:105)

Original: For h 2 we use the linear pencil for f 2 , but instead of using g (cid:104) 21 (cid:105) we use h 2 = c 0 φ + g (cid:104) 41 (cid:105) .
Revised: For h 2 we use the linear pencil for f 2 , but instead of using g (cid:104) 21 (cid:105) we use h 2 = c 0 φ + g (cid:104) 41 (cid:105) .

Original: We ﬁnd:
Revised: We ﬁnd:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence:
Revised: Hence:

Original: [Equation]
Revised: [Equation]

Original: Finally for h 0 we use again the linear pencil M 2 with:
Revised: Finally for h 0 we use again the linear pencil M 2 with:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C O THER LIMITING EXPRESSIONS
Revised: C O THER LIMITING EXPRESSIONS

Original: In this section we bring the sketch of proofs of additional expressions seen in the main results.
Revised: In this section we bring the sketch of proofs of additional expressions seen in the main results.

Original: C. E XPRESSION WITH DUAL COUNTERPART MATRICES U (cid:63) AND V (cid:63)
Revised: C. E XPRESSION WITH DUAL COUNTERPART MATRICES U (cid:63) AND V (cid:63)

Original: The former functionals f 2 and ˜ f 1 can be rewritten as:
Revised: The former functionals f 2 and ˜ f 1 can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: With similar steps using:
Revised: With similar steps using:

Original: [Equation]
Revised: [Equation]

Original: We ﬁnd:
Revised: We ﬁnd:

Original: [Equation]
Revised: [Equation]

Original: Hence in fact:
Revised: Hence in fact:

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have using the push-through identity and the cyclicity of the trace:
Revised: we have using the push-through identity and the cyclicity of the trace:

Original: [Equation]
Revised: [Equation]

Original: C. L IMITING TRACE OF Z T ZV ∗
Revised: C. L IMITING TRACE OF Z T ZV ∗

Original: Here we show another way in which our random matrix result can be used to infer the result on the limiting trace
Revised: Here we show another way in which our random matrix result can be used to infer the result on the limiting trace

Original: Tr d (cid:2) Z T ZV ∗ (cid:3) .
Revised: Tr d (cid:2) Z T ZV ∗ (cid:3) .

Original: To this end, we can design the linear-pencil:
Revised: To this end, we can design the linear-pencil:

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I   (163)
Revised: I   (163)

Original: It is straightforward to calculate the inverse of the sub-matrix:  
Revised: It is straightforward to calculate the inverse of the sub-matrix:  

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: Z
Revised: Z

Original: I 
Revised: I 

Original:  −=  
Revised:  −=  

Original: I − Z T
Revised: I − Z T

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: I − Z
Revised: I − Z

Original: [Equation]
Revised: [Equation]

Original: So that:
Revised: So that:

Original: [Equation]
Revised: [Equation]

Original: At this point, it is clear that the quantity of interest is provided by the term g (cid:104) 14 (cid:105) of the linear-pencil
Revised: At this point, it is clear that the quantity of interest is provided by the term g (cid:104) 14 (cid:105) of the linear-pencil

Original: M 3 .
Revised: M 3 .

Original: We ﬁnd calculate further: η ( g ) =
Revised: We ﬁnd calculate further: η ( g ) =

Original:   0  0 φg (cid:104) 33 (cid:105) 

Revised:   0  0 φg (cid:104) 33 (cid:105)

Original: 0 g (cid:104) 42 (cid:105) 

Revised: 0 g (cid:104) 42 (cid:105)

Original:    (166)
Revised:    (166)

Original: Based on the inverse of M 3 , we can already predict that g (cid:104) 33 (cid:105) =
Revised: Based on the inverse of M 3 , we can already predict that g (cid:104) 33 (cid:105) =

Original: 1 and g (cid:104) 42 (cid:105) = 0 .
Revised: 1 and g (cid:104) 42 (cid:105) = 0 .

Original: Hence: Π( M 3 ) =  
Revised: Hence: Π( M 3 ) =  

Original: I − V ∗
Revised: I − V ∗

Original: I − φI
Revised: I − φI

Original: I
Revised: I

Original: I   =
Revised: I   =

Original: ⇒ Π( M 3 ) − 1 =  
Revised: ⇒ Π( M 3 ) − 1 =  

Original: I
Revised: I

Original: V ∗ φV ∗
Revised: V ∗ φV ∗

Original: I φI
Revised: I φI

Original: I
Revised: I

Original: I   (167)
Revised: I   (167)

Original: Finally we obtain g (cid:104) 14 (cid:105) =
Revised: Finally we obtain g (cid:104) 14 (cid:105) =

Original: Tr d [ φV ∗ ] , and hence Tr d (cid:2) Z T ZV ∗ (cid:3) =
Revised: Tr d [ φV ∗ ] , and hence Tr d (cid:2) Z T ZV ∗ (cid:3) =

Original: φ Tr d [ V ∗ ] .
Revised: φ Tr d [ V ∗ ] .

Original: D A PPLICATIONS AND CALCULATION DETAILS D. M ISMATCHED R IDGELESS REGRESSION OF A NOISY LINEAR FUNCTION
Revised: D A PPLICATIONS AND CALCULATION DETAILS D. M ISMATCHED R IDGELESS REGRESSION OF A NOISY LINEAR FUNCTION

Original: Target function Here we consider a slightly more complicated version of the former example where we let y ( x 0 , x 1 ) =
Revised: Target function Here we consider a slightly more complicated version of the former example where we let y ( x 0 , x 1 ) =

Original: r (cid:0) x T 0 β ∗ 

Revised: r (cid:0) x T 0 β ∗

Original: + x T 1 β ∗ 1 (cid:1) + σ(cid:15) and still averaged over β 0 ∼ N (0 , I γp ) and β 1 ∼ N (0 , I (1 − γ ) p ) with x 0 ∈ R
Revised: + x T 1 β ∗ 1 (cid:1) + σ(cid:15) and still averaged over β 0 ∼ N (0 , I γp ) and β 1 ∼ N (0 , I (1 − γ ) p ) with x 0 ∈ R

Original: γp , x 1 ∈ R (1 − γ ) p .
Revised: γp , x 1 ∈ R (1 − γ ) p .

Original: We let again d = p + q and ψ = pd and φ 0 = pq .
Revised: We let again d = p + q and ψ = pd and φ 0 = pq .

Original: Therefore the former relation still holds φ = nd = np pd =
Revised: Therefore the former relation still holds φ = nd = np pd =

Original: φ 0 ψ .
Revised: φ 0 ψ .

Original: Similarly, we derive a block-matrix B and compute V ∗ :
Revised: Similarly, we derive a block-matrix B and compute V ∗ :

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: So that with the splitting Z = (cid:0)(cid:112) pd X 0 | (cid:112) pd X 1 | (cid:112) qd
Revised: So that with the splitting Z = (cid:0)(cid:112) pd X 0 | (cid:112) pd X 1 | (cid:112) qd

Original: Σ (cid:1) , and β ∗ T = (cid:0) β ∗ T 0 | β ∗ T 1 | β ∗ T 2 (cid:1) , and with ξ = Σ β ∗ 2 :
Revised: Σ (cid:1) , and β ∗ T = (cid:0) β ∗ T 0 | β ∗ T 1 | β ∗ T 2 (cid:1) , and with ξ = Σ β ∗ 2 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Estimator Following the same steps, we construct A and U with
Revised: Estimator Following the same steps, we construct A and U with

Original: [Equation]
Revised: [Equation]

Original: So that we get the linear estimator ˆ Y t
Revised: So that we get the linear estimator ˆ Y t

Original: [Equation]
Revised: [Equation]

Original: Analytic result as U and V ∗ commute again, the joint probability distribution can be derived:
Revised: Analytic result as U and V ∗ commute again, the joint probability distribution can be derived:

Original: [Equation]
Revised: [Equation]

Original: Therefore, in the regime λ = 0 , with κ = φ 0 γ , a calculation leads to the following result (dubbed the "mismatched model" in Hastie et al.
Revised: Therefore, in the regime λ = 0 , with κ = φ 0 γ , a calculation leads to the following result (dubbed the "mismatched model" in Hastie et al.

Original: [Equation]
Revised: [Equation]

Original: D. N ON ISOTROPIC MODEL
Revised: D. N ON ISOTROPIC MODEL

Original: We have the joint probabilities P ( u = α − i , v = 1) = 1 p = γ for i ∈ { 0 , . . .
Revised: We have the joint probabilities P ( u = α − i , v = 1) = 1 p = γ for i ∈ { 0 , . . .

Original: , p − 1 } and λ = 0 .
Revised: , p − 1 } and λ = 0 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: So either ζ = 0 and thus ˜ f 1 = 0 , or ζ (cid:54) = 

Revised: So either ζ = 0 and thus ˜ f 1 = 0 , or ζ (cid:54) =

Original: and:
Revised: and:

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Writing further down ( α i ζ ) 

Revised: Writing further down ( α i ζ )

Original: = ( α i ζ + φ − φ ) 2 =
Revised: = ( α i ζ + φ − φ ) 2 =

Original: ( α i ζ + φ ) 2 − 2 φ ( α i ζ + φ ) + φ 2 we get: p p − 1 (cid:88) i =
Revised: ( α i ζ + φ ) 2 − 2 φ ( α i ζ + φ ) + φ 2 we get: p p − 1 (cid:88) i =

Original: ( α i ζ )( φ + α i ζ )
Revised: ( α i ζ )( φ + α i ζ )

Original: [Equation]
Revised: [Equation]

Original: 1 − 2 φ p p − 1 (cid:88) i =
Revised: 1 − 2 φ p p − 1 (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =  ( φ
Revised: =  ( φ

Original: [Equation]
Revised: [Equation]

Original: i ζ )(181) = 1 −
Revised: i ζ )(181) = 1 −

Original: [Equation]
Revised: [Equation]

Original: + φ
Revised: + φ

Original: [Equation]
Revised: [Equation]

Original: 1 p p − 1 (cid:88) i =  ( φ + α i ζ )(182) =
Revised: 1 p p − 1 (cid:88) i =  ( φ + α i ζ )(182) =

Original: (1 − φ ) −
Revised: (1 − φ ) −

Original: [Equation]
Revised: [Equation]

Original: So:
Revised: So:

Original: [Equation]
Revised: [Equation]

Original: Now injecting the expression for ζ :
Revised: Now injecting the expression for ζ :

Original: [Equation]
Revised: [Equation]

Original: Hence the formula
Revised: Hence the formula

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Asymptotic limit: Let’s consider the behavior of the generalisation error when α → ∞ .
Revised: Asymptotic limit: Let’s consider the behavior of the generalisation error when α → ∞ .

Original: Let’s consider the potential solution for some k ∈ { 0 , . .
Revised: Let’s consider the potential solution for some k ∈ { 0 , . .

Original: . , p − 1 } :
Revised: . , p − 1 } :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for some constant c k .
Revised: for some constant c k .

Original: Then:
Revised: Then:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence we choose:
Revised: Hence we choose:

Original: [Equation]
Revised: [Equation]

Original: Because E gen ( ∞ ) ≥ 0 , we need to enforce ζ k > 0 which leads to the condition 1 pφ − k − 1 ≥ 0 , that is 1 ≥ pφ − k > 0 .
Revised: Because E gen ( ∞ ) ≥ 0 , we need to enforce ζ k > 0 which leads to the condition 1 pφ − k − 1 ≥ 0 , that is 1 ≥ pφ − k > 0 .

Original: So in fact it implies φ ∈ (cid:105) kp , k +1 p (cid:105) , so ζ k can only be a solution for φ in this range.
Revised: So in fact it implies φ ∈ (cid:105) kp , k +1 p (cid:105) , so ζ k can only be a solution for φ in this range.

Original: Therefore we can consider the solution ζ ( φ )
Revised: Therefore we can consider the solution ζ ( φ )

Original: = (cid:80) p − 1 i =0 1 ] kp ; k +1 p
Revised: = (cid:80) p − 1 i =0 1 ] kp ; k +1 p

Original: [( φ ) ζ k ( φ ) .
Revised: [( φ ) ζ k ( φ ) .

Original: Then notice:
Revised: Then notice:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and thus for φ ∈ [0 , 1] \ kp Z :
Revised: and thus for φ ∈ [0 , 1] \ kp Z :

Original: [Equation]
Revised: [Equation]

Original: So we clearly see that in the limit of α large, the test error approaches a function with two roots at the denominator.
Revised: So we clearly see that in the limit of α large, the test error approaches a function with two roots at the denominator.

Original: Evolution:
Revised: Evolution:

Original: [Equation]
Revised: [Equation]

Original: In particular f 2 is given by:
Revised: In particular f 2 is given by:

Original: [Equation]
Revised: [Equation]

Original: and ˜ f 1 is given by:
Revised: and ˜ f 1 is given by:

Original: [Equation]
Revised: [Equation]

Original: D.2. E IGENVALUE DISTRIBUTION
Revised: D.2. E IGENVALUE DISTRIBUTION

Original: In our ﬁgures, we look at the log-eigenvalue distribution of the student data ρ log λ as it provides the most natural distributions on a log-scale basis.
Revised: In our ﬁgures, we look at the log-eigenvalue distribution of the student data ρ log λ as it provides the most natural distributions on a log-scale basis.

Original: So in fact, if we plot the curve y ( x ) = ρ log λ ( x ) we have:
Revised: So in fact, if we plot the curve y ( x ) = ρ log λ ( x ) we have:

Original: [Equation]
Revised: [Equation]

Original: So in a log-scale basis we have ρ log λ (log x )
Revised: So in a log-scale basis we have ρ log λ (log x )

Original: =
Revised: =

Original: xρ λ ( x ) .
Revised: xρ λ ( x ) .

Original: It is interesting to notice the connection with η x for running computer simulations:
Revised: It is interesting to notice the connection with η x for running computer simulations:

Original: [Equation]
Revised: [Equation]

Original: It is work mentioning that the bulks are further "detached" as α grows as it can be seen in ﬁgure 6.
Revised: It is work mentioning that the bulks are further "detached" as α grows as it can be seen in ﬁgure 6.

Original: Furthermore, bigger α makes the spike more distringuisable.
Revised: Furthermore, bigger α makes the spike more distringuisable.

Original: K ERNEL M ETHODS
Revised: K ERNEL M ETHODS

Original: Kernel methods are equivalent to solving the following linear regression problem:
Revised: Kernel methods are equivalent to solving the following linear regression problem:

Original: [Equation]
Revised: [Equation]

Original: Where φ ( x ) =
Revised: Where φ ( x ) =

Original: ( φ i ( x ))
Revised: ( φ i ( x ))

Original: i ∈ N = ( √ ω
Revised: i ∈ N = ( √ ω

Original: i e i ( x )) for some orthogonal basis ( e i ) i ∈ N .
Revised: i e i ( x )) for some orthogonal basis ( e i ) i ∈ N .

Original: In fact we can consider:
Revised: In fact we can consider:

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 203) and z i = ( e 1 ( x i ) , . .
Revised: 203) and z i = ( e 1 ( x i ) , . .

Original: , e d ( x i )) .
Revised: , e d ( x i )) .

Original: Then let’s consider the following linear regression problem:
Revised: Then let’s consider the following linear regression problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This problem is identical to the kernel methods in the situation with a speciﬁc β ∗ T =
Revised: This problem is identical to the kernel methods in the situation with a speciﬁc β ∗ T =

Original: ( θ 01 , . .
Revised: ( θ 01 , . .

Original: , θ 0 d ) .
Revised: , θ 0 d ) .

Original: Although V ∗ and U don’t commute with each other, Notice that with x = y = − λ , due to the diagonal structure of U :
Revised: Although V ∗ and U don’t commute with each other, Notice that with x = y = − λ , due to the diagonal structure of U :

Original: [Equation]
Revised: [Equation]

Original: So in fact we ﬁnd the self-consistent set of equation with E gen (+ ∞ [Equation]
Revised: So in fact we ﬁnd the self-consistent set of equation with E gen (+ ∞ ) =

Original: = ˜ f 1 :
Revised: ˜ f 1 :

Original: [Equation]
Revised: [Equation]

Original: This is precisely the results from equation (78) in Loureiro et al.
Revised: This is precisely the results from equation (78) in Loureiro et al.

Original: (2021) (see also Bordelon et al.
Revised: (2021) (see also Bordelon et al.

Original: with the change of variables λ (1 + V ) →
Revised: with the change of variables λ (1 + V ) →

Original: ζ and ρ + q − 2 m → ˜ f 1 .
Revised: ζ and ρ + q − 2 m → ˜ f 1 .

Original: D. R ANDOM FEATURES EXAMPLE
Revised: D. R ANDOM FEATURES EXAMPLE

Original: We get the following matrices U, V with ˜
Revised: We get the following matrices U, V with ˜

Original: [Equation]
Revised: [Equation]

Original: In fact
Revised: In fact

Original: [Equation]
Revised: [Equation]

Original: the matrices U and V do not commute with each other, so we have more involved calculations.
Revised: the matrices U and V do not commute with each other, so we have more involved calculations.

Original: First we consider the subspace F =
Revised: First we consider the subspace F =

Original: Ker ( V − ˜
Revised: Ker ( V − ˜

Original: σ 2 I q ) ⊥ .
Revised: σ 2 I q ) ⊥ .

Original: Let’s deﬁne the matrices:
Revised: Let’s deﬁne the matrices:

Original: [Equation]
Revised: [Equation]

Original: Then, although U and V can’t be diagonalized in the same basis
Revised: Then, although U and V can’t be diagonalized in the same basis

Original: [Equation]
Revised: [Equation]

Original: they are still both block-diagonal matrices in the same direct-sum space R d = F ⊕ F ⊥
Revised: they are still both block-diagonal matrices in the same direct-sum space R d = F ⊕ F ⊥

Original: [Equation]
Revised: [Equation]

Original: so in fact the following split between the two subspaces F and F ⊥ holds:
Revised: so in fact the following split between the two subspaces F and F ⊥ holds:

Original: [Equation]
Revised: [Equation]

Original: Now let’s deﬁne κ 1 , κ 2 , κ 3 such that:
Revised: Now let’s deﬁne κ 1 , κ 2 , κ 3 such that:

Original: [Equation]
Revised: [Equation]

Original: That is to say, we get directly ˜ f
Revised: That is to say, we get directly ˜ f

Original: 1 = ( r 2 κ 1 + σ 2 κ 3 ) κ 2 and by deﬁnition:
Revised: 1 = ( r 2 κ 1 + σ 2 κ 3 ) κ 2 and by deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: So we already know that κ 3 = 1 .
Revised: So we already know that κ 3 = 1 .

Original: Let’s focus on κ 1 , we can deal with a linear pencil M such that we would get the desired term.
Revised: Let’s focus on κ 1 , we can deal with a linear pencil M such that we would get the desired term.

Original: First we deﬁne similarly A TF , the restriction of A T on the subspace F :
Revised: First we deﬁne similarly A TF , the restriction of A T on the subspace F :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, following the structure of M 1 we can construct the following linear-pencil M :
Revised: Then, following the structure of M 1 we can construct the following linear-pencil M :

Original: [Equation]
Revised: [Equation]

Original: So that:
Revised: So that:

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]
Revised: [Equation]

Original: In the above matrices
Revised: In the above matrices

Original: [Equation]
Revised: [Equation]

Original: the sub-blocks A F and V F are implicitly ﬂattened
Revised: the sub-blocks A F and V F are implicitly ﬂattened

Original: [Equation]
Revised: [Equation]

Original: so in fact M is given completely by:
Revised: so in fact M is given completely by:

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: and therefore, one has to pay attention on the quantity of interest which is given by a sum of two terms:
Revised: and therefore, one has to pay attention on the quantity of interest which is given by a sum of two terms:

Original: [Equation]
Revised: [Equation]

Original: Using a Computer-Algebra-System
Revised: Using a Computer-Algebra-System

Original: [Equation]
Revised: [Equation]

Original: we get the equations with γ x
Revised: we get the equations with γ x

Original: [Equation]
Revised: [Equation]

Original: γ y
Revised: γ y

Original: [Equation]
Revised: [Equation]

Original: δ x
Revised: δ x

Original: [Equation]
Revised: [Equation]

Original: δ y deﬁned such that g
Revised: δ y deﬁned such that g

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: So:
Revised: So:

Original: [Equation]
Revised: [Equation]

Original: and:
Revised: and:

Original: [Equation]
Revised: [Equation]

Original: Hence the result:
Revised: Hence the result:

Original: [Equation]
Revised: [Equation]

Original: Also there remain to use the last equation regarding ζ x using the fact that:
Revised: Also there remain to use the last equation regarding ζ x using the fact that:

Original: [Equation]
Revised: [Equation]

Original: Notice that we have
Revised: Notice that we have

Original: [Equation]
Revised: [Equation]

Original: So because A TF A F = A T A :
Revised: So because A TF A F = A T A :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: For κ 2 we can calculate the following expression - which in fact is general and doesn’t depend on the speciﬁc design of U :
Revised: For κ 2 we can calculate the following expression - which in fact is general and doesn’t depend on the speciﬁc design of U :

Original: [Equation]
Revised: [Equation]

Original: Hence the general formula:
Revised: Hence the general formula:

Original: [Equation]
Revised: [Equation]

Original: One can check that the same formula applies for instance for the mismatched ridgeless regression.
Revised: One can check that the same formula applies for instance for the mismatched ridgeless regression.

Original: Also, we assume that it can be replaced by its continuous limit in y → x in the situation x = y .
Revised: Also, we assume that it can be replaced by its continuous limit in y → x in the situation x = y .

Original: Finally for f 2 , we ﬁnd
Revised: Finally for f 2 , we ﬁnd

Original: [Equation]
Revised: [Equation]

Original: where we use ˜ g associated to a slightly different linear-pencil ˜ M :
Revised: where we use ˜ g associated to a slightly different linear-pencil ˜ M :

Original: [Equation]
Revised: [Equation]

Original: from which we get using a Compute-Algebra-System
Revised: from which we get using a Compute-Algebra-System

Original: [Equation]
Revised: [Equation]

Original: Another more straightforward way for obtaining the same result without the need for an additional linear-pencil is to notice that if we let E 

Revised: Another more straightforward way for obtaining the same result without the need for an additional linear-pencil is to notice that if we let E

Original: = ( I p | 0 p × N ) such that V F = ˜
Revised: = ( I p | 0 p × N ) such that V F = ˜

Original: r 2 E 1 E T
Revised: r 2 E 1 E T

Original: [Equation]
Revised: [Equation]

Original: then we have:
Revised: then we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore reusing the deﬁnition of δ x and the former linear-pencil M :
Revised: Therefore reusing the deﬁnition of δ x and the former linear-pencil M :

Original: [Equation]
Revised: [Equation]

Original: Conclusion we have the following equations
Revised: Conclusion we have the following equations

Original: [Equation]
Revised: [Equation]

Original: D. R EALISTIC DATASETS
Revised: D. R EALISTIC DATASETS

Original: For the realistic datasets, we capture the time evolution for two different datasets: MNIST and
Revised: For the realistic datasets, we capture the time evolution for two different datasets: MNIST and

Original: Fashion-MNIST.
Revised: Fashion-MNIST.

Original: To capture the dynamics over a realistic dataset X ∈ R n tot × d , it is more convenient to use the dual matrices U (cid:63) , V (cid:63) , Ξ .
Revised: To capture the dynamics over a realistic dataset X ∈ R n tot × d , it is more convenient to use the dual matrices U (cid:63) , V (cid:63) , Ξ .

Original: We only need to estimate U (cid:63) and Ξ β ∗ with U (cid:63) (cid:39) 1 n tot X T X and Ξ β ∗ (cid:39) 1 n tot X T Y .
Revised: We only need to estimate U (cid:63) and Ξ β ∗ with U (cid:63) (cid:39) 1 n tot X T X and Ξ β ∗ (cid:39) 1 n tot X T Y .

Original: In both cases, we sill sample a subset of n < n tot data-samples for the training set.
Revised: In both cases, we sill sample a subset of n < n tot data-samples for the training set.

Original: The scope of the theoretical equations is still subject to the high-dimensional limit assumption, in other words we need n and d "large enough", that is to say 1 (cid:28) n .
Revised: The scope of the theoretical equations is still subject to the high-dimensional limit assumption, in other words we need n and d "large enough", that is to say 1 (cid:28) n .

Original: At the same time, the approximation of U (cid:63) and Ξ β ∗ hints at n tot sufﬁciently large compared to the number of considered samples n .
Revised: At the same time, the approximation of U (cid:63) and Ξ β ∗ hints at n tot sufﬁciently large compared to the number of considered samples n .

Original: Hence we need also n (cid:28) n tot .
Revised: Hence we need also n (cid:28) n tot .

Revised: Numerically, for the two following datasets and as per assumptions 2.1, the theoretical prediction rely on a contour enclosing the spectrum Sp ( ˜ X T ˜ X ) of ˜ X T ˜ X , but not enclosing − λ .

Revised: Therefore, in order to proceed with our computations, we take a symmetric rectangle around the x-axis crossing the axis at the particular values − λ 2 and 1 . 2 max Sp ( ˜ X T ˜ X ) after a preliminary computation of the spectrum.

Revised: For the need of our experiments, we commonly discretized the contour and ran a numerical integration over the discretized set of points.

Original: MNIST Dataset: we consider the MNIST dataset with n tot = 70 (cid:48)
Revised: MNIST Dataset: we consider the MNIST dataset with n tot = 70 (cid:48)

Original: 000 images of size 28 × 28 of numbers between 0 and 9 .
Revised: 000 images of size 28 × 28 of numbers between 0 and 9 .

Original: In our setting, we consider the problem of estimating the parity of the number, that is the vector Y with Y i = 1 if image i represents an even number and Y i = − 1 for an odd-number.
Revised: In our setting, we consider the problem of estimating the parity of the number, that is the vector Y with Y i = 1 if image i represents an even number and Y i = − 1 for an odd-number.

Original: The dataset X ∈ R n tot × d is further processed by centering each column to its mean, and normalized by the global standard-deviation of X (in other words the standard deviation of X seen as a ﬂattened n tot × d vector) and further by √ d (for consistency with the theoretical random matrix Z ). 
Revised: The dataset X ∈ R n tot × d is further processed by centering each column to its mean, and normalized by the global standard-deviation of X (in other words the standard deviation of X seen as a ﬂattened n tot × d vector) and further by √ d (for consistency with the theoretical random matrix Z ). The results that we obtain are shown in Figure 4.

Original: As it can be seen on the left graph ofFigure 4, the learning curves can be slightly off compared to the theoretical predictions on the learning proﬁle, in particular for smaller n , but also around the double-descent peak  for small λ .
Revised: On the ﬁgure on the left side we show the theoretical prediction of the training and test error with the minimum least-squares estimator (or alternatively the limiting errors at t = + ∞ ). We make the following observations which in fact relates to the same ones as in Figure 4 in Loureiro et al. • There is an apparent larger deviation in the test error for smaller n which tends to heal with increasing number of data samples • A bias between the mean observation of the test error and the theoretical prediction emerges around the double-descent peak between n = 100 and n = 1000 , in particular, the experiments are slightly above the given prediction. We notice that this bias is even more pronounced for smaller values of λ .

Original: This can also be the case for the test error evolution as it can be seen at n = 700 with the ﬁgure on the right side where the experimental runs are consistently above the prediction, although we observe a good match overall.
Revised: • Although it is not visible on the ﬁgure, increasing n further tends to create another divergence between the theoretical prediction and the experimental runs - as it is expected with n getting closer to n tot . Besides the limiting error, we chose to draw the time-evolution of the training and test error around at n = 700 around the double descent on the right side of Figure 4. This time, a gradient descent algorithm is executed for each 10 experimental runs with a constant learning-rate dt = 0 . Due to the log-scale of the axis, it is interesting to notice that with such a basic non-adaptive learning-rate, each tick on the graph entails 10 times more computational time to update the weights.

Revised: By contrast, the theoretical curves can be calculated at any point in time much farther away.

Revised: Overall we see a good agreement between the evolution of the experimental runs with the theoretical predictions.

Revised: However, as it is expected around the double-descent spike, learning-curves of the experimental runs appear slightly biased and above the theoretical curves.

Original: Fashion-MNIST Dataset: We provide another example with MNIST-Fashion dataset with d =and n tot = 70 (cid:48) 000 .
Revised: Fashion-MNIST Dataset: We provide another example with MNIST-Fashion dataset with d =and n tot = 70 (cid:48) 000 .

Original: The dataset X is processed as for the MNIST dataset.
Revised: The dataset X is processed as for the MNIST dataset.

Original: We take the output vector
Revised: We take the output vector

Original: Y such that Y i = 1 for items i above the waist, and Y i = − 1 otherwise.
Revised: Y such that Y i = 1 for items i above the waist, and Y i = − 1 otherwise.

Original: We provide the results in Figure 7 where the training set is sampled randomly with n elements in n tot and the test set is sampled in the remaining examples.
Revised: We provide the results in Figure 7 where the training set is sampled randomly with n elements in n tot and the test set is sampled in the remaining examples.

Original: As it can be seen, the test error is slightly above the prediction for n < 10 3 but ﬁts well with the predicted values for larger n .
Revised: As it can be seen, the test error is slightly above the prediction for n < 10 3 but ﬁts well with the predicted values for larger n .

Original: Furthermore, the learning curves through time in Figure 8 are different compared to the MNIST dataset in Figure 4 and we still observe a good match with the theoretical predictions.
Revised: Furthermore, the learning curves through time in Figure 8 are different compared to the MNIST dataset in Figure 4 and we still observe a good match with the theoretical predictions.

Original: However the mismatch in the learning curves seems to increase in the speciﬁc case when λ is lower, increasing thereby the effect of the double descent.
Revised: However the mismatch in the learning curves seems to increase in the speciﬁc case when λ is lower, increasing thereby the effect of the double descent.

Original: E L INEAR -P ENCILS FIXED POINT EQUATION
Revised: E L INEAR -P ENCILS FIXED POINT EQUATION

Original: In general, traces of algebraic expressions of large random matrices can be difﬁcult to compute (See for instance appendix in Pennington & Worah (2017)).
Revised: In general, traces of algebraic expressions of large random matrices can be difﬁcult to compute (See for instance appendix in Pennington & Worah (2017)).

Original: A modern approach consists in assembling a block of large random matrices such that the block-inversion formula (otherwise called the Schur complement) yields the desired algebraic expression of random matrices in some sub-blocks.
Revised: A modern approach consists in assembling a block of large random matrices such that the block-inversion formula (otherwise called the Schur complement) yields the desired algebraic expression of random matrices in some sub-blocks.

Original: Then, using the correlation structure of the sub-blocks of the assembled block-matrix, a ﬁxed-point equation can be derived that yields a set of algebraic equations whose solutions provide the traces of the sub-blocks of the inverted block-matrix.
Revised: Then, using the correlation structure of the sub-blocks of the assembled block-matrix, a ﬁxed-point equation can be derived that yields a set of algebraic equations whose solutions provide the traces of the sub-blocks of the inverted block-matrix.

Original: This idea initially emerged in Rashidi Far et al.
Revised: This idea initially emerged in Rashidi Far et al.

Original: (2006), then has been described further in Mingo & Speicher (2017) and Helton et al.
Revised: (2006), then has been described further in Mingo & Speicher (2017) and Helton et al.

Original: (2018) which coined the term "Linear-Pencils".
Revised: (2018) which coined the term "Linear-Pencils".

Original: Since then, it has recently been introduced in the machine learning community in a more geneal form in Adlam & Pennington (2020), and also recently in Bodin & Macris (2021) where a non-rigorous proof is provided using the replica symmetry tool from statistical physics.
Revised: Since then, it has recently been introduced in the machine learning community in a more geneal form in Adlam & Pennington (2020a), and also recently in Bodin & Macris (2021) where a non-rigorous proof is provided using the replica symmetry tool from statistical physics.

Original: Here we propose to generalize even further the ﬁxed-point equation where we let the sub-blocks be potentially of any form, and provide a non-rigorous proof of the proposition following the steps proposed in Bun et al.
Revised: Here we propose to generalize even further the ﬁxed-point equation where we let the sub-blocks be potentially of any form, and provide a non-rigorous proof of the proposition following the steps proposed in Bun et al.

Original: Potters & Bouchaud (2020) using Dyson brownian motions and Itô
Revised: Potters & Bouchaud (2020) using Dyson brownian motions and Itô

Original: Lemma to derive the ﬁxed-point equation.
Revised: Lemma to derive the ﬁxed-point equation.

Original: E. N OTATIONS AND MAIN STATEMENT
Revised: E. N OTATIONS AND MAIN STATEMENT

Original: Let’s consider an invertible self-adjoint complex block matrix M ∈ C N × N with N = p 1 + . . .
Revised: Let’s consider an invertible self-adjoint complex block matrix M ∈ C N × N with N = p 1 + . . .

Original: + p n such that M (cid:104) ij (cid:105) is the sub-matrix of size p
Revised: + p n such that M (cid:104) ij (cid:105) is the sub-matrix of size p

Original: i × p j .
Revised: i × p j .

Original: We assume that p 1 , . . .
Revised: We assume that p 1 , . . .

Original: , p n → ∞ when N → ∞ such that we have the ﬁxed ratios γ i =
Revised: , p n → ∞ when N → ∞ such that we have the ﬁxed ratios γ i =

Original: lim N →∞ p
Revised: lim N →∞ p

Original: i N , and let’s deﬁne the inverse G =
Revised: i N , and let’s deﬁne the inverse G =

Original: M − 1 .
Revised: M − 1 .

Original: Now let S = { ij | p i = p j } .
Revised: Now let S = { ij | p i = p j } .

Original: We deﬁne for ( ij ) ∈ S (and deﬁned to 0 outside of this set):
Revised: We deﬁne for ( ij ) ∈ S (and deﬁned to 0 outside of this set):

Original: [Equation]
Revised: [Equation]

Original: We further decompose M as the sum of two components M = M 0 + 1 √ N H with M 0 and H both self-adjoint, M 0 is also invertible and where H is a block of random matrices independent of M 0 .
Revised: We further decompose M as the sum of two components M = M 0 + 1 √ N H with M 0 and H both self-adjoint, M 0 is also invertible and where H is a block of random matrices independent of M 0 .

Original: In particular, Re( H ) and Im( H ) are independent element-wise with each-other, and we leave the possibility that the sub-blocks of Re( H ) and Im( H ) be either a Wigner random-matrix, Wishart random matrix, the adjoint of a Wishart random matrix, or a (real-)weighted sum of any of the three.
Revised: In particular, Re( H ) and Im( H ) are independent element-wise with each-other, and we leave the possibility that the sub-blocks of Re( H ) and Im( H ) be either a Wigner random-matrix, Wishart random matrix, the adjoint of a Wishart random matrix, or a (real-)weighted sum of any of the three.

Original: For the sake of simplicity, we will consider that the elements Re H [Equation] ij [Equation]
Revised: For the sake of simplicity, we will consider that the elements Re H (cid:104) ij (cid:105) uv

Original: uv or Im H

Original: [Equation] ij [Equation] uv within the block ij are gaussian and identically distributed although the gaussian assumption can certainly be weakened.
Revised: or Im H (cid:104) ij (cid:105) uv within the block ij are gaussian and identically distributed although the gaussian assumption can certainly be weakened.

Original: Now let’s deﬁne σ klij the covariance between the elements of the sub-matrices H (cid:104) ij (cid:105) and
Revised: Now let’s deﬁne σ klij the covariance between the elements of the sub-matrices H (cid:104) ij (cid:105) and

Original: H (cid:104) kl (cid:105) , that is for ( il, jk ) ∈ S 2 on the off-diagonal position uv on transposed element-locations:
Revised: H (cid:104) kl (cid:105) , that is for ( il, jk ) ∈ S 2 on the off-diagonal position uv on transposed element-locations:

Original: [Equation]
Revised: [Equation]

Original: Also, there can be some covariances on similar element-locations which we deﬁne with ¯ σ for ( ik, jl ) ∈ S 2 :
Revised: Also, there can be some covariances on similar element-locations which we deﬁne with ¯ σ for ( ik, jl ) ∈ S 2 :

Original: [Equation]
Revised: [Equation]

Original: Notice that σ klij =
Revised: Notice that σ klij =

Original: σ ijkl and ¯ σ klij = ¯ σ jilk by symmetry, and also when H is real, we always have σ klij =
Revised: σ ijkl and ¯ σ klij = ¯ σ jilk by symmetry, and also when H is real, we always have σ klij =

Original: ¯ σ klij .
Revised: ¯ σ klij .

Original: So overall the random matrix H has to satisfy the following property at any off-diagonal locations ( uv ) , ( xy ) and blocks ( ij, kl ) :
Revised: So overall the random matrix H has to satisfy the following property at any off-diagonal locations ( uv ) , ( xy ) and blocks ( ij, kl ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally we deﬁne the mapping η : C n × n → C n × n :
Revised: Finally we deﬁne the mapping η : C n × n → C n × n :

Original: [Equation]
Revised: [Equation]

Original:  then let Π( M ) =
Revised: (272) then let Π( M ) =

Original: M 0 − η
Revised: M 0 − η

Original: ( g ) ⊗ I with the notation ( η ( g ) ⊗ I ) ij =
Revised: ( g ) ⊗ I with the notation ( η ( g ) ⊗ I ) ij =

Original: η ( g ) ij I p i
Revised: η ( g ) ij I p i

Original: when ij ∈ S and ( η ( g ) ⊗ I ) ij = 

Revised: when ij ∈ S and ( η ( g ) ⊗ I ) ij =

Original: p i ×
Revised: p i ×

Original: q j the null-matrix when ij / ∈ S .
Revised: q j the null-matrix when ij / ∈ S .

Original: Similarly as G , with Π( G ) =
Revised: Similarly as G , with Π( G ) =

Original: Π( M ) − 1 and with:
Revised: Π( M ) − 1 and with:

Original: [Equation]
Revised: [Equation]

Original: 274) we state that:

Original: [Equation]
Revised: [Equation]

Revised: we state that:

Original: [Equation]
Revised: [Equation]

Original: Remark 1: When M 

Revised: Remark 1: When M

Original: =
Revised: =

Original: Z ⊗ I such that Z ij = 0 if ij / ∈ S , then we get Π( M ) =
Revised: Z ⊗ I such that Z ij = 0 if ij / ∈ S , then we get Π( M ) =

Original: ( Z − η ( g )) ⊗ I , then Π( M ) − 

Revised: ( Z − η ( g )) ⊗ I , then Π( M ) −

Original: =
Revised: =

Original: ( Z − η ( g ))
Revised: ( Z − η ( g ))

Original: − 1 ⊗ I .
Revised: − 1 ⊗ I .

Original: Therefore: g =
Revised: Therefore: g =

Original: ( Z − η ( g )) − 1 , or re-adjusting the terms, we ﬁnd back the equation from Adlam & Pennington (2020); Bodin & Macris (2021): 
Revised: ( Z − η ( g )) − 1 , or re-adjusting the terms, we ﬁnd back the equation from Adlam & Pennington (2020a); Bodin & Macris (2021): [Equation]

Original: [Equation]
Revised: [Equation]

Original: Remark 2: When considering the linear pencil of a block-matrix M (cid:63) such this is not necessarily self-adjoint however still invertible, the ampliﬁed matrix M can be considered:
Revised: Remark 2: When considering the linear pencil of a block-matrix M (cid:63) such this is not necessarily self-adjoint however still invertible, the ampliﬁed matrix M can be considered:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies that:
Revised: This implies that:

Original: [Equation]
Revised: [Equation]

Original: So g will also be of the form:
Revised: So g will also be of the form:

Original: [Equation]
Revised: [Equation]

Revised: (cid:19) (279)

Original: So in fact, the same equation still holds with g (cid:104) ij 
Revised: So in fact, the same equation still holds with g (cid:104) ij (cid:105) (cid:63)

Original: [Equation]
Revised: [Equation]

Original:  (cid:63) [Equation] and thus, the self-adjoint constraints can be relaxed.
Revised: C (cid:63) − η ( g (cid:63) ) ⊗ I ) (cid:104) ij (cid:105) (cid:3) and thus, the self-adjoint constraints can be relaxed.

Original: E. N ON - RIGOROUS PROOF VIA D YSON BROWNIAN MOTIONS
Revised: E. N ON - RIGOROUS PROOF VIA D YSON BROWNIAN MOTIONS

Original: In order to show the former result, we extend the sketch of proof provided in Bun et al.
Revised: In order to show the former result, we extend the sketch of proof provided in Bun et al.

Original: Potters & Bouchaud (2020).
Revised: Potters & Bouchaud (2020).

Original: First we introduce a time t and a matrix Z ∈ C n × n with M ( t, Z ) =
Revised: First we introduce a time t and a matrix Z ∈ C n × n with M ( t, Z ) =

Original: Z ⊗ I + M 0 + 1 √ N H ( t ) with H a Dyson brownian motion.
Revised: Z ⊗ I + M 0 + 1 √ N H ( t ) with H a Dyson brownian motion.

Original: Therefore, the matrix that we are interested in is actually M = M (1 , 0 n × n ) .
Revised: Therefore, the matrix that we are interested in is actually M = M (1 , 0 n × n ) .

Original: In order for H (1) to satisfy the property 271 we must have:
Revised: In order for H (1) to satisfy the property 271 we must have:

Original: [Equation]
Revised: [Equation]

Original: Itô’s lemma provides the stochastic differential equation
Revised: Itô’s lemma provides the stochastic differential equation

Original: [Equation]
Revised: [Equation]

Original: Using simple algebraic manipulations and the fact that G is analytic in M (cid:104) ij (cid:105) uv as a rational function
Revised: Using simple algebraic manipulations and the fact that G is analytic in M (cid:104) ij (cid:105) uv as a rational function

Original: [Equation]
Revised: [Equation]

Original: we can rewrite the above partial derivatives as:
Revised: we can rewrite the above partial derivatives as:

Original: [Equation]
Revised: [Equation]

Original: And applying the same formula twice:
Revised: And applying the same formula twice:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Injecting it in (281) we get for p =

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: q

Original: [Equation]
Revised: [Equation]

Original: So considering
Revised: So considering

Original: [Table]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Table] [Equation] [Table] [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]
Revised: [Equation]

Original: The matrix Z is now helpful upon noticing that (using again the analyticity of G )
Revised: The matrix Z is now helpful upon noticing that (using again the analyticity of G )

Original: [Equation]
Revised: [Equation]

Original: Hence (using the fact that σ klij
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: As it would require more in-depth analysis [Equation] we make the following two assumptions:
Revised: As it would require more in-depth analysis, we make the following two assumptions:

Original: We assume that g (cid:104) αβ (cid:105) N concentrates towards a constant value g (cid:104) αβ (cid:105) when N → ∞ 2.
Revised: We assume that g (cid:104) αβ (cid:105) N concentrates towards a constant value g (cid:104) αβ (cid:105) when N → ∞ 2.

Original: That (cid:15) (cid:104) αβ
Revised: That (cid:15) (cid:104) αβ

Original: (cid:105)
Revised: (cid:105)

Original: N concentrates towards 0 when N → ∞
Revised: N concentrates towards 0 when N → ∞

Original: With these assumptions in mind we obtain the partial differential equation:
Revised: With these assumptions in mind we obtain the partial differential equation:

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] using the change of variable ˆ g ( s ) =
Revised: Finally, using the change of variable ˆ g ( s ) =

Original: g (ˆ t ( s ) , ˆ Z ( s ))
Revised: g (ˆ t ( s ) , ˆ Z ( s ))

Original: =
Revised: =

Original: g ( t + s, Z + sη ( g ( t, Z )))
Revised: g ( t + s, Z + sη ( g ( t, Z [Equation]

Original: we ﬁnd:
Revised: we ﬁnd:

Original: [Equation]
Revised: [Equation]

Original: So ˆ g (cid:104) αβ (cid:105) ( s ) is constant so: ˆ g (cid:104)αβ
Revised: So ˆ g (cid:104) αβ (cid:105) ( s ) is constant so: ˆ g (cid:104)

Original: [Equation]
Revised: [Equation]

Original: − t)
Revised: − t

Revised: [Equation]

Original: = ˆ g
Revised: = ˆ g

Revised: [Equation]

Revised: αβ

Original: [Equation]
Revised: [Equation]

Original: 0) which implies:
Revised: 0) which implies:

Original: [Equation]
Revised: [Equation]

Original: Hence for ( t, Z ) =
Revised: Hence for ( t, Z ) =

Original: (1 , 0 n × n ) we have:
Revised: (1 , 0 n × n ) we have:

Original: [Equation]

Original: g

Original: [Equation]
Revised: [Equation]

Original: Hence the expected result:
Revised: Hence the expected result:

Original: [Equation]
Revised: [Equation]

Original: E. E XAMPLES W IGNER SEMICIRCLE LAW
Revised: E. E XAMPLES W IGNER SEMICIRCLE LAW

Original: Let’s consider n = 1 and the symmetric random matrix
Revised: Let’s consider n = 1 and the symmetric random matrix

Original: H ∈
Revised: H ∈

Original: R
Revised: R

Original: N
Revised: N

Original: and
Revised: and

Original: M = H √ N − zI .
Revised: M = H √ N − zI .

Original: We ﬁnd that η ( g ) =
Revised: We ﬁnd that η ( g ) =

Original: g and using (276) we ﬁnd directly
Revised: g and using (276) we ﬁnd directly

Original: [Equation]
Revised: [Equation]

Original: M ARCHENKO P ASTUR LAW
Revised: M ARCHENKO P ASTUR LAW

Original: Let’s consider n = 2 and the random matrix X ∈ R d × N with φ
Revised: Let’s consider n = 2 and the random matrix X ∈ R d × N with φ

Original: =
Revised: =

Original: Nd = γ 1 γ 2 and γ 1 = NN + d , γ 2 = dN + d and the random symmetric block matrix:
Revised: Nd = γ 1 γ 2 and γ 1 = NN + d , γ 2 = dN + d and the random symmetric block matrix:

Original: [Equation]
Revised: [Equation]

Original: Using Schur complement, it can be seen that g (cid:104) 
Revised: Using Schur complement, it can be seen that g (cid:104) 11 (cid:105) N =

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original:  which is precisely the trace that is being looked for.
Revised: 1 (cid:21) which is precisely the trace that is being looked for.

Original: A careful analysis shows that σ 1212 = σ 2121 = 1 γ 1 while the rest is null and thus [ η ( g )] 11 = γ 2 γ 1 g (cid:104) 22 (cid:105) =
Revised: A careful analysis shows that σ 1212 = σ 2121 = 1 γ 1 while the rest is null and thus [ η ( g )] 11 = γ 2 γ 1 g (cid:104) 22 (cid:105) =

Original: g (cid:104) 22 (cid:105) φ and [ η ( g )] 22 = g (cid:104) 11 (cid:105) .
Revised: g (cid:104) 22 (cid:105) φ and [ η ( g )] 22 = g (cid:104) 11 (cid:105) .

Original: With (276) we obtain the system of algebraic equations
Revised: With (276) we obtain the system of algebraic equations

Original: [Equation]
Revised: [Equation]

Original: Therefore, injecting the solution of the second equation g (cid:104) [Equation] = − {+
+}
Revised: Therefore, injecting the solution of the second equation g (cid:104) 22 (cid:105) = − {+
+}

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: in the ﬁrst equation
Revised: in the ﬁrst equation

Original: [Equation]
Revised: [Equation]

Original: Hence the Marcheko Pastur result:
Revised: Hence the Marcheko Pastur result:

Original: [Equation]
Revised: [Equation]