S
PECTRAL M ULTIPLICITY E NTAILS S AMPLE - WISE
M ULTIPLE D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we study the generalization risk of ridge and ridgeless linear regression.
We assume that the data features follow a multivariate normal distribution and that the spectrum of the covariance matrix consists of a given set of eigenvalues of proportionally growing multiplicity.
We characterize the limiting bias and variance when the dimension and the number of training samples tend to inﬁnity proportionally.
Exact formulae for the bias and variance are derived using the random matrix theory and convex Gaussian min-max theorem.
Based on these formulae, we study the sample-wise multiple descent phenomenon of the generalization risk curve, i.e., with more data, the generalization risk can be non-monotone, and speciﬁcally, can increase and then decrease multiple times with more training data samples.
We prove that sample-wise multiple descent occurs when the spectrum of the covariance matrix is highly ill-conditioned.
We also present numerical results to conﬁrm the values of the bias and variance predicted by our theory and illustrate the multiple descent of the generalization risk curve.
Moreover, we theoretically show that the ridge estimator with optimal regularization can result in a monotone generalization risk curve and thereby eliminate multiple descent under some assumptions.
I NTRODUCTION
The double/multiple descent phenomenon attracted recent research attention due to (Belkin et al., 2019).
This line of work focuses on the parameter-wise double/multiple descent phenomenon of the risk curve (Bartlett et al., 2020; Tsigler & Bartlett, 2020; Belkin et al., 2019; 2020;
Chen et al., 2020a; Liang et al., 2020; Advani et al., 2020; B ¨ os & Opper, 1998; Krogh & Hertz, 1992; Le Cun et al., 1991; Mei & Montanari, 2019; Opper et al., 1990;
Vallet et al., 1989;
Watkin et al., 1993).
The classical learning theory shows that when the number of parameters (which reﬂects the model complexity) increases, the test error (generalization risk) ﬁrst decreases due to more ﬁtting power, and then increases due to overﬁtting.
The generalization risk attains a peak at the interpolation threshold (the number of parameters equals the number of data points so that the model interpolates the data).
This results in a U-shaped risk curve if we plot the test error versus the number of parameters.
The double descent risk curve posits that the risk will decrease (again) if one further increases the model complexity beyond the interpolation threshold (Belkin et al., 2019).
Thus there is a second descent in addition to the ﬁrst one in the U-shaped stage of the curve.
Belkin et al.
(2019) presented empirical results and showed the existence of such double descent behavior in the random Fourier features model, the fully connected neural network, and the random forest model.
Prior to (Belkin et al., 2019), earlier studies of the shape and features of the risk curve in a number of contexts include (Vallet et al., 1989; Opper et al., 1990; Le Cun et al., 1991; Krogh & Hertz, 1992; B¨os & Opper, 1998;
Watkin et al., 1993; Advani et al., 2020).
Loog et al.
(2020) presented a prehistory of the double descent phenomenon.
Belkin et al.
(2020) proved the double descent curve in the Gaussian model and the Fourier series model.
Mei & Montanari (2019) theoretically established the double descent curve of the random features regression.
Bartlett et al.
; Tsigler & Bartlett (2020) characterized the conditions for ridgeless and ridge linear regression problems, respectively, under which the minimum-norm interpolants achieve near-optimal generalization risk.
Liang et al.
(2020) showed that the test error of the minimum-norm interpolator of data in reproducing kernel Hilbert space is upper bounded by a multiple descent curve as the model complexity increases.
They also presented a numerical result supporting that the test error itself exhibits a multiple descent curve.
Chen et al.
(2020a) proved that the multiple descent curve does exist for the minimum-norm interpolator in linear regression and that the curve can be even designed.
Following the parameter-wise double descent, research interest extended to epoch-wise and samplewise double descent (Nakkiran et al., 2020;
Chen et al., 2020b; Min et al., 2021; Nakkiran et al., 2021).
Nakkiran et al.
(2020) observed from their numerical result that the generalization risk experiences a double descent as one keeps the model size ﬁxed and increases the training time.
They called this observation epoch-wise double descent.
Nakkiran et al.
(2020) also noted sample-wise non-monotonicity, which means that more data can hurt generalization.
Nakkiran et al.
(2021) proved that for isotropic features, optimally regularized ridge regression yields a monotonic generalization risk curve with more samples.
Nakkiran et al.
(2021) also showed that if the features are formed by projecting high-dimensional isotropic data to a random low-dimensional space (say, ddimensional), the optimally regularized ridge regression has a monotonic generalization risk curve with increasing d (the model size).
Sample-wise non-monotonicity and double descent was also observed in (Chen et al., 2020b; Min et al., 2021) in adversarially trained models.
C ompared to (Wu & Xu, 2020; ichi Amari et al., 2021; Dobriban & Wager, 2018; Richards et al., 2021), in what follows, we highlight our contributions and the differences from them.
First, our major contribution is providing a rigorous proof for the existence of sample-wise (test error vs. the number of training samples) double and multiple descent in linear regression.
However, (Richards et al., 2021) only mentioned parameter-wise double descent (test error vs. model capacity) in their related work.
(ichi Amari et al., 2021) only mentioned epoch-wise (test error vs. training time) double descent in Appendix A.2.
Neither (Richards et al., 2021) nor (ichi Amari et al., 2021) mentioned multiple descent.
Second, we made and theoretically proved the observation that an ill-conditioned covariance matrix is a sufﬁcient condition for the existence of sample-wise multiple descent. To the best of our knowledge, our work is the ﬁrst paper that pointed this out. Third, we solved the Stieltjes transform explicitly and derived explicit formulae for the risk and variance in our setup. In addition, we also provided rigorous treatment to the ridgeless setting and also obtained explicit formulae for it.
Fourth, there is another difference between our paper and the papers that the reviewer mentioned.
(Wu & Xu, 2020; ichi Amari et al., 2021; Dobriban & Wager, 2018; Richards et al., 2021) assumed a prior on the true linear model and takes expectation over the prior.
In our paper, we do not assume a prior on the true linear model and our risk does not take the expectation over a random true linear model.
In the setting of generally anisotropic features, this paper gives an asymptotic characterization of the generalization risk curve with more samples.
The asymptotic regime is an approximation for large n, d and can also shed light on practical machine learning problems.
We ﬁrst introduce our problem setup.
1. P ROBLEM S ETUP
Data Distribution Let
Σ ∈ R d × d be a positive semi-deﬁnite matrix which is termed the covariance matrix , and let θ ∗ ∈ R d .
The eigenvalues of Σ are λ 1 , . . .
, λ m with multiplicity d 1 , . . .
, d m , respectively.
We have d = (cid:80) mi =1 d i .
Assume that λ 1 , . . .
, λ m are ﬁxed, distinct, all positive, and do not depend on d (i.e., for all d , the eigenvalue of Σ are always λ 1 , . . .
, λ m ).
We assume the following data distribution D for ( x, y ) ∈
R d × R :
[Equation]
where x and (cid:15) are independent and (cid:15) ∼ N (0 , σ 2 ) .
In practice, there are natural random variables x that satisfy our assumption.
For example, assume that we want to use machine A to measure the length of several objects and use machine B to measure their temperature.
The measured lengths and temperatures follow an i.i.d.
Gaussian distribution.
However, the variance of measurement of machine A is different from that of machine B. Then we consider the random vector formed by the measurements x = ( l 1 , . . . , l n , t 1 , . . . , t n ) , where l i and t i are the length and temperature of object i , respectively.
This results in a block-structured covariance matrix.
When we measure more objects, the size of the covariance matrix tends to inﬁnity.
Second, the motivation came from (Nakkiran et al., 2021).
(Nakkiran et al., 2021) observed empirically in their Figure 2 that when the covariance matrix has a block structure (speciﬁcally, there are only two ﬁxed different eigenvalues 10 and 1 ), the expected excess risk exhibits multiple descent.
We quantitatively studied this observation and obtained the related formulae.
The excess risk of an estimator θ ∈ R d is given by
[Equation]
Assume that the training data { ( x i , y i ) } ni
=1 ⊆ R d × R is drawn i.i.d.
from D .
Write
[Equation]
We have y =
Xθ ∗ + (cid:15) , where (cid:15) ∼ N (0 , σ 2 I n ) .
Ridge Estimator and Minimum-Norm Estimator
Deﬁnition 1 (Ridge estimator) .
The ridge estimator ˆ θ λ,n,d ∈ R d ( λ > 0 ) solves the following minimization problem
[Equation]
Deﬁnition 2 (Minimum-norm estimator) .
The minimum-norm estimator (also known as the ridgeless estimator) ˆ θ 0 ,n,d ∈ R d solves the following minimization problem
[Equation]
such that
[Equation]
We are interested in the expected excess risk of ˆ θ λ,n,d , which is given by
[Equation]
The expectation is taken over the randomness of the training data { ( x i , y i ) } ni =1 .
Asymptotic Regime Let Π
i
∈
R
d × d be the orthogonal projection to the eigenspace of λ i .
This paper focuses on the asymptotic behavior of the expected excess risk of ˆ θ λ,n,d where n, d i → + ∞ , d i /n
→ z i ( z i is a ﬁxed positive constant), and (cid:107) Π
i θ ∗ (cid:107) 2 → η i .
In other words, we are interested in [Equation] i
[-→ z i (cid:107) Π i θ ∗ (cid:107) 
 →-][Equation]
i /n
[Equation]
1. O UR C ONTRIBUTIONS
Our contributions are summarized as follows.
We obtain the formulae for the limiting bias and variance, and thereby the limiting risk.
We use two methods to obtain these formulae.
Speciﬁcally, we obtain the limiting bias and variance by solving the Stieltjes transform and computing its derivatives and antiderivatives.
We also use convex Gaussian min-max theorem (CGMT)
(Thrampoulidis et al., 2015) to compute the limiting variance.
The advantage of the CGMT method is that it is more mathematically tractable for the ridgeless estimator Through
the CGMT approach , we obtain a closed-form formula for the variance in the underparameterized regime and simplify the formula for the variance in the overparameterized regime.
Moreover, based on the simpliﬁed formula, we deduce a closed-form expression for the variance if the covariance matrix of the data distribution has two different eigenvalues.
We ﬁnd and theoretically prove that sample-wise multiple descent happens when the covariance matrix has eigenvalues of very different orders of magnitude (thus the covariance matrix is highly ill-conditioned).
We show that if the true linear model θ ∗ satisﬁes (cid:107) Π i θ ∗ (cid:107) 2 = (cid:113) d i d , optimal regularization (i.e., pick λ that minimizes the generalization risk of ˆ θ λ,n,d ) results in a monotone generalization risk curve—in other words, with optimal regularization, more data samples always improve generalization.
Thus there is no sample-wise double or multiple descent.
This provides a theoretical proof of a phenomenon observed in (Nakkiran et al., 2021) that optimal regularization can mitigate double descent for anisotropic data.
Note that without regularization, there will be a blow-up in expected excess risk when n = d (the linear model exactly interpolates the data) and therefore, there is no samplewise descent across the under- and over-parameterized regimes.
P RELIMINARIES
Notation Write [ m ] for { 1 , 2 , . . .
, m } .
Let i denote the imaginary unit.
If x ∈ R n and Σ ∈ R n × n is a positive semideﬁnite matrix, write (cid:107) x (cid:107) Σ (cid:44) √ x (cid:62) Σ x .
For a vector x , let (cid:107)·(cid:107) 1 and (cid:107)·(cid:107) 2 denote the (cid:96) 1 and (cid:96) 2 norm, respectively.
Let (cid:12) denote the Hadamard (entry-wise) product between vectors.
Write (cid:107) · (cid:107) 2 and (cid:107) · (cid:107) F for the spectral matrix norm and Frobenius matrix norm, respectively.
Let (cid:52) denotes the Loewner order.
For two square matrices A and B of the same size, write A (cid:52) B if B − A is positive semideﬁnite.
Deﬁne spec ( A ) as the set of all eigenvalues of A .
Let O ( d ) =
(cid:8)
A ∈ R d × d | AA (cid:62) =
A (cid:62)
A
= I d (cid:9) denote the set of d × d orthogonal matrices.
Deﬁne S d − 1 ( r ) (cid:44) { x ∈ R d | (cid:107) x (cid:107) 2 = r } .
Denote almost sure convergence by a.s. → , and convergence in probability plim and P → .
Ridge Estimator and Minimum-Norm Estimator We begin with the equivalent characterizations of the ridge and minimum-norm estimator.
An equivalent characterization of the ridge estimator ˆ θ λ,n,d is
[Equation]
The second equality in Equation (2) is because of the Sherman–Morrison–Woodbury formula.
A proof of Equation (2) can be found in (Tsigler & Bartlett, 2020).
An equivalent deﬁnition of the minimum-norm estimator ˆ θ 0 ,n,d is that ˆ θ 0 ,n,d solves the following minimization problem
[Equation]
such that X (cid:62) Xθ
= X (cid:62) y .
Thus we have
[Equation]
where A + denotes the pseudo-inverse of A .
The second and third equalities are because of the identity X + = (cid:0) X (cid:62) X (cid:1) + X (cid:62) =
X (cid:62)
[Equation]
XX
[Equation]
(cid:1)
+ .
The minimum-norm estimator is the limit of the ridge estimator ˆ θ λ,n,d as λ → 0 + :
[Equation]
This is because of the identity
[Equation]
Bias-Variance Decomposition of Expected Excess Risk
We ﬁrst show that the excess risk of an estimator θ equals the norm of θ − θ ∗ :
[Equation]
For the ridge estimator
[Equation]
the expected excess risk is
[Equation]
For the minimum-norm estimator, the expected excess risk is
[Equation]
B [Equation] We call B λ,d,n and B 0 ,d,n the bias term, and call V λ,d,n and V 0 ,d,n the variance term.
The bias and variance for the minimum-norm estimator are the limit of their counterpart for the ridge estimator as λ → 0 + , i.e., lim λ
→ 0 + B λ,d,n = B 0 ,d,n and lim λ → 0 + V λ,d,n = V 0 ,d,n (this can be shown by Lebesgue’s dominated convergence theorem, see our proof in Lemma 5 and Lemma 6, respectively).
M AIN R ESULTS 3. L IMITING R ISK AND S AMPLE - WISE M ULTIPLE D ESCENT
We study the limiting bias and variance for a linear regression problem in which the data distribution follows a multivariate normal distribution, the spectrum of the covariance matrix exhibits a block structure and tends to a discrete distribution.
Thanks to the random matrix theory, we obtain the formulae (presented in Theorem 1) for the limiting bias and variance, and thereby the total risk.
We use two methods to obtain these formulae.
The ﬁrst method is through the Stieltjes transform of the matrix 1 n XX (cid:62) .
The central quantity for computing the limiting bias and variance through the ﬁrst method is the solution ρ ∗ to the optimization problem Equation (5) in Item 1 of Theorem 1.
Item 1 guarantees the existence of a solution and determines its optimality condition Equation (6).
Item 2 computes the Jacobian matrix of ρ ∗ with respect to λ i and provides a closed-form formula to compute the Jacobian matrix.
Equation (9) and Equation (10) in Item 4 give the formulae for the limiting bias obtained by the ﬁrst method.
Equation (11) and Equation (12) give the limiting variance.
The second method is through the convex Gaussian min-max theorem (CGMT)
(Thrampoulidis et al., 2015).
The central quantity is the solution r ∗ to the minimax optimization problem Equation (8) in Item 3.
We use CGMT to obtain the formulae for the variance term.
They are presented in Equation (13) and Equation (14) in Item 4.
Theorem 1.
The following statements hold:
There exists a minimizer ρ ∈ R m + that solves
[Equation]
The minimizer ρ ∗ satisﬁes
[Equation]
Let ρ ∗ ∈ R
m be a minimizer of Equation (5) and J = ∂ρ ∗ ∂ λ ∈ R m × m be the Jacobian matrix J ij =
∂ρ ∗
i ∂λ j .
Then J is given by
[Equation]
and the matrix (cid:0) diag ( λ ) + (cid:0) λ + λ (cid:62) ρ ∗
[Equation]
I m − ( z − ρ ∗ ) λ (cid:62) (cid:1) is always invertible.
Deﬁne r = ( r 1 , . . .
, r m ) , λ = ( λ 1 , . . . , λ m ) , and ϑ ( r t , r , λ ) = 2 r t (cid:115) 1 + (cid:88)
[Equation]
For any K t ≥ 2 λ and K u ≥ 2 λ + ( 2+ √ γ ) λ , we have
[Equation]
and the above optimization problem has a solution.
Let r ∗ =
( r ∗ 1 , . .
, r ∗ m ) solve Equation (8) .
Deﬁne q = (cid:0) η 21 /z 1 , . . .
, η 2 m /z m (cid:1) (cid:62) and view λ
= ( λ 1 , . .
λ m ) [Equation] as a column vector.
The limiting bias is given by
[Equation]
The limiting variance is given by
[Equation]
Figure 1 illustrates the theoretical and numerical values of the bias, variance, and total risk.
We observe a triple descent in Figure 1a where the covariance matrix has three blocks, and a quadruple descent in Figure 1b where the covariance has four blocks.
In the three-block example, we set λ 3 (cid:29) λ
(cid:29) λ 1 ( λ 1 = 1 , λ 2 = 100 , λ 3 = 1000 ).
In the four-block example, we set λ 4 (cid:29) λ 3 (cid:29) λ 2 (cid:29) λ
( λ 1 = 1 , λ 2 = 100 , λ 3 = 10 4 , λ 4 = 10 7 ).
For the values of other parameters, please refer to the caption of Figure 1 Our ﬁndings provide an explanation for the occurrence of sample-wise multiple descent: it occurs when the covariance matrix is highly ill-conditioned.
Moreover, we ﬁnd that the generalization risk curve is continuous in ridge regression ( λ > 0 ) while it blows up at n = d in ridgeless regression ( λ = 0 ).
We can see the singularity (at n = d = 200 ) of the ridgeless generalization risk curve in Figure 2a.
Following Theorem 1, we focus on the variance in the ridgeless case ( λ = 0 ) and further study the expressions in Equation (13) and Equation (14).
We ﬁnd that the variance exhibits sharply different behaviors in the underparameterized and overparameterized regimes.
Recall that we will let n, d i → + ∞ and keep d i /n
→ z i .
Then d/n → (cid:80) i ∈ [ m ] z i .
If lim d/n = (cid:80) i ∈ [ m ] z i > 1 , we are in the underparameterized regime.
In this regime, the bias vanishes and therefore the risk equals the variance.
If lim d/n < 1 , we are in the overparameterized regime.
Theorem 2.
If d/n → (cid:80) i ∈ [ m ] z i > 1 and r ∗ =
( r ∗ 1 , . .
, r ∗ m ) solves min
[Equation]
i
[Equation]
(cid:88)
[Equation]
subject to (cid:115)
[Equation]
then we have an optimality condition for r ∗ :
[Equation]
[Figure] We specify the parameters that we used as follows. are Var1” is obtained by Equation (11) of
Theorem 1.
“T. Var2” is obtained by Equation (13).
where A ∗ = (cid:113)(cid:80) i ∈ [ m ] r ∗ 2 i + 1 . Moreover, we have lim n,d i → + ∞ d i /n → z i
V 0 ,d,n
[Equation] σ [Equation] λ →
[Equation] + [Equation] mi
[Equation]
1 r ∗ 2 i .
If d/n → (cid:80) i ∈
[ m ] z i < 1 , then we have
[Equation]
Corollary 1.
If m = 1 and d/n → z 1 > 1 , we have lim n,d i → + ∞ d
[Equation]
Proof.
In the case m = 1 , we have r ∗ 1 solves min r 1 ≥ 0 1 λ 1 r 21 subject to (cid:112) r 21 + 1 = √
z
1 r 1 .
The equality constraint gives r ∗ 21 = 1 z 1 − 1 .
Then by Theorem 2, the limiting variance is σ 2 r ∗ 21 = σ 2 1 z 1 − 1 .
In Theorem 2, we ﬁnd that in the underparameterized regime, r ∗ solves an equality-constrained minimization problem.
In the proof of Theorem 2, we see that the equality constraint is feasible in the underparameterized regime but infeasible in the overparameterized regime.
Moreover, we present an optimality condition for r ∗ , which will be used in Theorem 3 to study the two-block ( m = 2 ) case.
If the data distribution is isotropic (which means that the covariance matrix is a scalar matrix), Collorary 1 shows that the limiting variance is σ 2 1 z 1 − 1 , which agrees with (Hastie et al., 2019, Theorem 1).
In the overparameterized regime, however, we ﬁnd that the limiting variance does not depend on the spectrum { λ 1 , . . .
, λ m } of the covariance matrix and only depends on the noise intensity σ and the ratios z i =
lim d i /n .
This agrees with (Hastie et al., 2019, Proposition 2).
In Theorem 3, we study the case m = 2 and present a concrete closed-form formula for the limiting variance in the overparameterized regime.
Recall that the limiting variance in the underparameterized regime has a closed-form σ 2 (cid:80) i ∈
[ m ] z i 1 − (cid:80) i ∈ [ m ] z i for general m , as shown in Theorem 2.
Theorem 3.
If m = 2 and d/n → z 1 + z 2 > 1 , we have
[Equation]
[Figure]
where
[Equation]
We illustrate the theoretical values predicted by Theorem 3 (overparameterized regime) and Theorem 2 (underparameterized regime) in Figure 2a and compare it to the numerical values.
Corollary 2 (Triple descent in the two-block case) .
Assume m = 2 , z 1 =
z 2 , d/n → ζ = 2 z 1 , and λ 2 /λ 1 = (cid:37) .
Deﬁne f (cid:37) ( ζ ) =
lim n,d i → + ∞ d i /n
→ z i
V 0 ,d,n .
We have
[Equation]
There exists ζ 1 , ζ 2 , ζ 3 , ζ 4 and (cid:37) 0 such that for all (cid:37) > (cid:37) 0 , we have f (cid:48) (cid:37) ( ζ 1 ) < 0 , f (cid:48) (cid:37) ( ζ 2 ) > 0 , f (cid:48) (cid:37) ( ζ 3 ) < 0 , and f (cid:48) (cid:37) ( ζ 4 ) < 0 .
Proof.
The case ζ < 1 is already given in Theorem 2.
In the sequel, assume ζ > 1 .
Deﬁne q as in
Equation (16).
We have
[Equation]
Recall Theorem 3, we get
[Equation]
Direct calculation yields
[Equation]
The function g ( ζ ) > 0 if ζ ∈ (cid:0) √ 2 , 2 (cid:1) and we have g ( ζ ) <
if ζ < √ 2 or ζ > 2 .
Pick ζ 1 > 2 > ζ 2 > √ 2 > ζ 3 > 1 > ζ 4 .
Then we have g ( ζ 1 ) < 0 , g ( ζ 2 ) > 0 , g ( ζ 3 ) < 0 , and g ( ζ 4 ) > 0 .
There exists (cid:37) 0 such that for all (cid:37) > (cid:37) 0 , we have f (cid:48) (cid:37) ( ζ 1 ) < 0 , f (cid:48) (cid:37) ( ζ 2 ) > 0 , f (cid:48) (cid:37) ( ζ 3 ) < 0 , and f (cid:48) (cid:37) ( ζ 4 ) < 0 .
Collorary 2 theoretically proves that there exists triple descent when m =
and λ 2 (cid:29) λ 1 .
Note that a larger ζ = lim d/n reﬂects a relatively smaller n .
If f (cid:48) (cid:37) ( ζ ) < 0 , then f (cid:37) ( ζ ) decreases on a neighborhood of ζ and therefore the limiting variance increases with a relatively larger n .
As n becomes relatively larger, we see an increasing stage, a decreasing stage, and ﬁnally an increasing stage in order in the overparameterized regime ( n < d ).
When we further increase n and enter the underparameterized regime, we observe a decreasing stage.
We illustrate f ( ζ ) in Figure 2b.
In
Figure 2b, we observe two singularities at ζ = 1 and ζ = 2 .
3. O PTIMAL R EGULARIZATION M ONOTONIZES G ENERALIZATION R ISK C URVE
Recall the deﬁnition of the ridge estimator in Deﬁnition 1.
Since this subsection concerns samplewise monotonicity, we add a subscript n to X and y (they are deﬁned by Equation (1) in Section 1.1) to emphasize that they consist of n data items.
Therefore we write
[Equation]
In this subsection, under an assumption, we show that optimal regularization (i.e., pick λ that minimizes the generalization risk of ˆ θ λ,n,d ) results in a monotone generalization risk curve—in other words, with optimal regularization, more data always reduces the generalization risk.
The assumption is that (cid:107) Π i θ ∗ (cid:107)
=
(cid:113)
d i d , i.e., the squared norm of the projection of θ ∗ onto each eigenspace of the covariance matrix is proportional to the dimension of that eigenspace.
(Nakkiran et al., 2021) showed by numerical results that optimal regularization can mitigate double descent for anisotropic data distribution.
We give a partial theoretical proof of their observed phenomenon.
To ease the notation, we use γ i (cid:44) lim n/d i rather than z i (cid:44)
lim
d i /n in Theorem 4 because a larger γ reﬂects a relatively larger n (in the limit).
Theorem 4 shows that with the optimal regularization, the limiting risk is an increasing function of γ 1 , . . .
, γ m .
Theorem 4 (Optimal regularization) .
If
[Equation]
then there exists a function g ( γ 1 , . .
γ m ) such that g ( γ 1 , . .
, γ m ) is increasing in every γ i and
[Equation]
C ONCLUSION
We studied the generalization risk (test error) versus the number of training samples in ridgeless regression.
Under the assumption that the data distribution is Gaussian and the spectrum distribution of its covariance matrix converges to a discrete distribution, we obtained the exact formulae for the limiting bias and variance terms using the random matrix theory when the dimension and the number of training samples go to inﬁnity in a proportional manner.
Using these formulae, we proved the sample-wise multiple descent phenomenon of the generalization risk curve.
Moreover, we theoretically showed that the ridge estimator with optimal regularization can result in a monotone generalization risk curve and thereby eliminate multiple descent under some assumptions.
A E IGENDECOMPOSITION AND M ORE N OTATION
Write Σ = P Λ P (cid:62) , where P is an orthogonal matrix and Λ = diag ( λ 1 I d 1 , . . .
λ m I d m ) ∈
R d × d is a diagonal matrix.
Write λ − = min i ∈ [ m ] λ i and λ + = max i ∈ [ m ] λ i .
We can generate x 1 , . . .
, x n from standard normal random vector
z i ∼ N (0 , I d ) by setting x i = P Λ 1 / 2 z i .
Therefore, if Z = ( z. . .
z n ) ∈ R d × n ,we get
X (cid:62) =
( x. . .
x n ) =
P Λ 1 / 2 ( z. . .
z n ) =
P Λ 1 / 2 Z .
Take the transpose gives X = Z (cid:62) Λ 1 / 2 P (cid:62) .
Note that every entry of Z ∈ R d × n follows i.i.d.
N (0 , 1) .
Write Z in a row-partitioned form
Z =  
Z (cid:62)...
Z (cid:62) m   , where Z i ∈ R n × d i .
Write P in a column-partitioned form
P
[Equation] [Equation]
P m )
[Equation]
where P i ∈ R
d × d i .
Recall that Π
i ∈ R
d × d denotes the orthogonal projection to the eigenspace of λ i .
We have Π i
= P i P (cid:62) i .
Deﬁne θ (cid:48) (cid:44) P (cid:62) θ ∗ and write it in a row-partitioned form
[Equation]
(19) where θ (cid:48) i ∈
R d i .
Then (cid:107) θ (cid:48)
i (cid:107) 2 =
(cid:13)(cid:13) P (cid:62) i θ ∗ (cid:13)(cid:13)
= (cid:13)(cid:13) P i P (cid:62) i θ ∗ (cid:13)(cid:13)
= (cid:107) Π
i θ ∗ (cid:107) 2 .
We summarize part of the notation above in Table 1.
B B IAS AND V ARIANCE U NDER E IGENDECOMPOSITION
Lemma 1 characterizes the smallest and largest eigenvalue of Z (cid:62) Zd (if n/d → γ < 1 ) and ZZ (cid:62) n
(if
n/d → γ > 1 ).
Recall that we study the asymptotic regime d i /n
→ z i .
Therefore γ = 1 (cid:80) j ∈ [ m ] z j .
Lemma 1 ((Bai & Yin, 2008, Theorem 2)) .
Let Z ∈ R
d
× n be a random matrix whose entries are i.i.d.
N (0 , 1) random variables.
As n, d → + ∞ , n/d → γ ∈ (0 , 1) , we have
[Equation]
[Equation] [Equation]
almost surely.
If γ ∈ (1 , ∞ ) , as n, d → + ∞ , n/d → γ , we have
[Equation]
almost surely.
Lemma 2 (Corollary 5.35 (Vershynin, 2010)) .
Let A be an N × n matrix whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − 2 exp (cid:0) − t 2 / 2 (cid:1) one has
[Equation]
where s min ( A ) and s max ( A ) are the smallest and largest singular value of A .
Lemma 3.
Let Z ∈ R
d
× n be a random matrix whose entries are i.i.d.
N (0 , 1) random variables, where d = d ( n ) satisﬁes lim n →∞ nd ( n ) = γ .
There exists universal positive constants C 1 , C 2 , N such that for all n > N , we have
[Equation]
Proof.
Since d ( n ) (cid:16) n , with loss of generality, we assume n/d → γ ∈ (0 , 1) .
Take t = c 1 √ n in
Lemma 2, where c 1 = 12 (cid:16) 1 √ γ − 1 (cid:17) > 0 .
With probability at least 1 − 2 e − c 21 n/ 2 , we have
[Equation]
Therefore, we deduce (cid:32)(cid:114)
[Equation]
Deﬁne C 1 = 18 (cid:16) 1 √ γ − 1 (cid:17) 2 > 0 and C 2 = (cid:16) 3 √ γ + 1 (cid:17) 2 .
Then there exists a universal constant Nsuch that for all n > N 1 , with probability at least 1 − 2 e − c 21 n/ 2 , we have
[Equation]
Deﬁne event E n = (cid:8) C 1 < 1 n s 2min ( Z ) ≤ 1 n s 2max ( Z ) <
C 2 (cid:9) c .
Then we have Pr { E n } ≤ 2 e − c 21 n/ 2 .
Since (cid:80) n ≥ 1 Pr { E n } ≤ (cid:80) n ≥ 1 2 e − c 21 n/ 2 < ∞ , then the probability that inﬁnitely many of E n occur is 0 , i.e.,
Pr (cid:26) lim sup n
E n (cid:27) = 0 .
Therefore, there exists a universal constant N 2 such that for all n > N 2 , E n does not happen, in other words,
[Equation]
holds.
Lemma 4.
Let Z ∈ R
d
× n be a random matrix whose entries are i.i.d.
N (0 , 1) random variables, and let p be a ﬁxed positive integer which is viewed as a constant and hidden in (cid:46) .
If n (cid:16) d , we have E tr (cid:0) ZZ (cid:62) (cid:1) (cid:16) n 2 , E tr (cid:0) ZZ (cid:62) (cid:1) 2 (cid:16) n 3 , and E (cid:107) Z (cid:107) p 2 (cid:46) n p/ 2 .
Proof.
We have
E tr (cid:0)
[Equation]
=
[Equation]
(cid:88)
[Equation]
E
[Equation]
Write Z = 

 z (cid:62)...
z (cid:62) d  
 , where z i ∈ R n and z i ∼ N (0 , I n ) .
We have E (cid:0) z (cid:62)
i z i (cid:1) 2 = E (cid:107)
z i (cid:107) 4 2 = n ( n + 2) .
For i (cid:54) =
j , we deduce E (cid:0) z (cid:62)
i
z j (cid:1)
= E (cid:0) (cid:107) z i (cid:107) 2 (cid:107) z
j (cid:107) 2 u (cid:62) v (cid:1) 2 where u, v ∼ Unif (cid:0) S n − 1 (cid:1) and (cid:107) z
i (cid:107) 2 , (cid:107) z
j (cid:107) 2 , u, v are independent.
Then we get
E (cid:0) (cid:107) z
i (cid:107) (cid:107) z
j (cid:107) s (cid:62)
i s j (cid:1)
= E (cid:107) z i (cid:107) 22 (cid:107) z
j (cid:107) 2 2 (cid:0) s (cid:62)
i s j (cid:1) 2 = n 2 E u 21 = n 2 · 1 n = n .
As a result, we have
[Equation]
By (Vershynin, 2018)
[Equation]
there exists a universal constant C > 0 such that for any t > 0 ,
P (cid:110) (cid:107) Z (cid:107) 2 > C
[Equation]
n + √ d + t (cid:17)(cid:111) <
[Equation]
Deﬁne K = C
[Equation]
n + √ d (cid:17) .
Then we have
[Equation]
Recall Γ( z ) = (cid:82) ∞ 0 x
z − 1 e − x dx .
Setting t = C √ u in the equation below yields
[Equation]
Then we can bound the following integral
[Equation]
where the ﬁrst inequality is because of Equation (20).
We are in a position to bound E (cid:107) Z (cid:107) p 2 :
[Equation]
where the ﬁrst inequality is because
[Equation]
Lemma 5.
The following equation for the bias term B λ,d,n (deﬁned in Equation (3) ) holds
[Equation]
[Equation] [Equation]
Moreover, we have | B λ,d,n | (cid:46) (cid:107) θ ∗ (cid:107)
and lim λ
→ 0 + B λ,d,n = B 0 ,d,n .
For all sufﬁciently large n and d such that n/d → γ ∈ (0 , 1) , we have 0 ≤ ddλ B λ,d,n (cid:46) (cid:107) θ ∗ (cid:107) 22 .
Therefore, { B λ,d,n } is uniformly bounded and uniformly equicontinuous with respect to λ ∈ (0 , ∞ ) .
Proof.
Introduce the shorthand notation M = Λ 1 / 2 ZZ
(cid:62) Λ 1 / 2 ∈ R d × d , A
= I d + 1 nλ M ∈ R
[Equation]
× d ,
N = nλI n + Z (cid:62) Λ Z ∈ R n × n , and Q =
I d − Λ 1 / 2 ZN − 1 Z (cid:62) Λ 1 / 2 ∈ R d × d .
Because
X (cid:62) ( nλI
[Equation]
XX
[Equation]
[Equation] [Equation]
, we have
[Equation]
Using the Sherman–Morrison-Woodbury formula yields
[Equation]
It follows that
[Equation]
Therefore, we deduce
[Equation]
Because (cid:13)(cid:13) Λ 1 / 2 (cid:13)(cid:13) 2 (cid:46) 1 and (cid:13)(cid:13)(cid:13)(cid:0) I d + 1 nλ Λ 1 / 2 ZZ (cid:62) Λ 1 / 2 (cid:1) − 1 (cid:13)(cid:13)(cid:13) 2 ≤ 1 , we have (cid:107)
[Equation]
22 (cid:46) (cid:107)
[Equation]
(cid:48) (cid:107)
[Equation]
(cid:107) θ
[Equation]
(cid:107) 22 .
Therefore | B λ,d,n | (cid:46) (cid:107) θ ∗ (cid:107) 22 .
Moreover, by the dominated convergence theorem,
[Equation]
We compute the derivative of A − 1 :
[Equation]
The matrix Mn = Λ 1 /
[Equation]
Λ 1 / 2 n ∈ R d × d is positive semideﬁnite and its d − n smallest eigenvalues are zeros.
Its non-zero eigenvalues are the same as the non-zero eigenvalues of Z (cid:62) Λ Z n .
Because all eigenvalues of Z (cid:62) Λ Z n are positive almost surely, the spectrum of M n consists of d − n zeros and the spectrum of Z (cid:62) Λ Z n .
We study the range of the spectrum of Z (cid:62) Λ Z n .
Because λ − Z (cid:62) Zn (cid:52) Z (cid:62) Λ Z n (cid:52) λ
+ Z (cid:62)
Zn , we deduce λ min (cid:18) Z (cid:62)
Λ Z n (cid:19) ≥ λ
[Equation]
λ min (cid:18) Z (cid:62) Z n (cid:19) → λ − (cid:16) 1 − (cid:112) 1 /γ (cid:17)(24) λ max (cid:18) Z (cid:62) Λ Z n (cid:19) ≤ λ + λ max (cid:18) Z (cid:62) Z n (cid:19) → λ + (cid:16)
+ (cid:112) 1 /γ (cid:17) 2 .
Deﬁne L 1 = λ − λ min (cid:16) Z (cid:62) Zn (cid:17) and L 2 = λ + λ max (cid:16) Z (cid:62) Zn (cid:17) .
We get lim n,d → + ∞ n/d → γ< 1 L 1 = λ − (cid:16) 1 − (cid:112) 1 /γ (cid:17) 2 , lim n,d → + ∞ n/d → γ< 1 L 2 = λ
+ (cid:16)
+ (cid:112) 1 /γ (cid:17) 2 and spec (cid:18) Z (cid:62) Λ Z n (cid:19) ⊆
[ L 1 , L 2 ] .
We bound (cid:13)(cid:13) MA − 3 (cid:13)(cid:13)(cid:13)(cid:13) MA − 3 (cid:13)(cid:13) 2 = n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
max
[Equation]
max
[Equation]
We compute ddλ (cid:107) Λ 1 / 2 A − 1 θ (cid:48) (cid:107) 22 : d dλ
[Equation] [Equation]
A
[Equation]
θ
[Equation]
=nλ 2 θ (cid:48)(cid:62) (cid:0)
A
[Equation]
=nλ
[Equation]
(cid:62) (cid:0) Λ
[Equation]
MA − 1 Λ (cid:1) (cid:0)
[Equation]
θ (cid:48) (cid:1)
Next, we bound (cid:12) (cid:12) ddλ (cid:107) Λ 1 / 2 A − 1 θ (cid:48) (cid:107) 22 (cid:12) (cid:12) : (cid:12)(cid:12)(cid:12)(cid:12)
d dλB λ,d,n (cid:12)(cid:12)(cid:12)(cid:12) ≤nλ 2 (cid:13)(cid:13)
MA − 2 Λ A − 1 + A − 1 Λ MA − 2 (cid:13)(cid:13) 2 (cid:107) θ (cid:48)
[Equation]
where the last inequality is because (cid:13)(cid:13)
A Λ A − 1 (cid:13)(cid:13)
= (cid:107) Λ (cid:107) 2 ≤ λ + (cid:46) 1 .
Deﬁne f ( s ) =
s (1+ s/λ ) 3 .
Because f (cid:48) ( s ) =
λ 3 ( λ − 2 s ) ( λ + s ) 4 , the function f is increasing on [0 , λ/ 2] and decreasing on [ λ/ 2 , + ∞ ) .
If λ ≤ 2 L 1 , we have max s ∈ [ L 1 ,L 2 ]
[Equation]
/λ )
[Equation]
It follows that
[Equation]
If λ ≥ 2 L 2 , we get λmax s ∈ [ L
[Equation]
L 2 ] s
[Equation]
=
[Equation]
L
[Equation]
If 2 L 1 < λ < 2 L 2 , we obtain λmax s ∈
[ L
[Equation]
L 2 ] s
[Equation]
λ
[Equation]
λ
[Equation]
1 L 1 .
In all three cases, we show that 1 λ 2 max s ∈ [ L 1 ,L 2 ] s (1+ s/λ ) 3 (cid:46) 1 L 1 .
It follows that (cid:12)(cid:12)(cid:12)(cid:12)
d dλ
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
By Lemma 3, there exists a universal constant n 0 such that for all n > n 0 , one has
[Equation]
Thus we conclude that
[Equation]
We can exchange differentiation and expectation and get
[Equation]
and
[Equation]
Lemma 6.
The following equation for the variance term holds
[Equation]
Moreover, for all sufﬁciently large n and d such that n/d → γ (cid:54) = 1 , we have lim λ → 0 + V λ,d,n =
V 0 ,d,n , | V λ,d,n | (cid:46) 1 and (cid:12)(cid:12) ddλ V λ,d,n (cid:12)(cid:12) (cid:46) 1 .
Therefore, { V λ,d,n } is uniformly bounded and uniformly equicontinuous with respect to λ ∈ (0 , ∞ ) .
Proof.
As in the proof of Lemma 5, deﬁne M = Λ 1 / 2 ZZ (cid:62) Λ 1 / 2 ∈ R d × d and N = nλI n + Z (cid:62) Λ Z ∈
R n × n .
Recalling Σ
= P Λ P (cid:62) and X =
Z (cid:62) Λ 1 / 2 P (cid:62) , we have
[Equation]
Recalling Equation (23) yields
[Equation]
Deﬁne R = Λ 1 / 2 Z ∈ R d × n .
We get
[Equation]
R .
Notice that if 0 < a < b , then aI d + RR (cid:62) (cid:52) bI d + RR (cid:62) .
We deduce (cid:0)
[Equation]
−
[Equation]
aI d + RR
[Equation]
=
[Equation]
I
[Equation] [Equation]
− a ) RR (cid:62) (cid:60) 0 .
Thus (cid:0) bI d + RR (cid:62) (cid:1) 2 (cid:60) (cid:0) aI d + RR (cid:62) (cid:1) 2 , which implies (cid:0) bI
d + RR (cid:62) (cid:1) − 2 (cid:52) (cid:0) aI d + RR (cid:62) (cid:1) − 2 .
We get
[Equation]
, tr (cid:16)
[Equation]
(cid:1)
[Equation]
Let λ 0 ( · ) denote the smallest non-zero eigenvalue of a positive semideﬁnite matrix.
We bound the
Frobenius norm
[Equation]
It follows that
[Equation]
If n/d → γ < 1 , the matrix Z (cid:62) Z is full-rank almost surely.
Then, using the formula for the mean of inverse Wishart distribution, we have E tr (cid:0) Z (cid:62) Z (cid:1) + = tr E (cid:0) Z (cid:62) Z (cid:1) − 1 = tr (cid:16) I n d − n − 1 (cid:17) (cid:16) 1 .
If n/d → γ > 1 , the matrix ZZ (cid:62) is full-rank almost surely.
Similarly, we have E tr (cid:0) ZZ (cid:62) (cid:1) + = tr E (cid:0) ZZ (cid:62) (cid:1) − 1 (cid:16) 1 .
By the dominated convergence theorem, we have lim λ → 0 + V λ,d,n = V 0 ,d,n .
Moreover, V λ,d,n (cid:46) E (cid:13)(cid:13) Λ ZN − 1 (cid:13)(cid:13) 2 F (cid:46) 1 .
Next we bound ddλ V (ˆ θ ) .
Because dN − 1 dλ = − N − 1 dNdλ N − 1 = − nN − 2 , we deduce d
[Equation]
≤
[Equation]
On the other hand, we have
[Equation]
+
[Equation]
Because the number of non-zero eigenvalues of Z (cid:62) Λ Z equals rank (cid:0) Z (cid:62) Λ Z (cid:1) =
n ∧ d (cid:16) n , we get (cid:12)(cid:12)(cid:12)(cid:12)
d
[Equation]
[Table]
[Equation]
[Table]
[Equation]
If γ < 1 , the matrix Z (cid:62) Λ Z n is full-rank almost surely.
By Equation (24) and Equation (25) in the proof of Lemma 5, there exists universal positive constants C 1 and C 2 such that spec (cid:16) Z (cid:62) Λ Z n (cid:17) ⊆
[ C 1 , C 2 ] for all sufﬁciently large n and d such that n/d → γ < 1 .
If γ > 1 , the non-zero eigenvalues of Z (cid:62) Λ Z n and Mn are the same.
The matrix Mn is full-rank almost surely.
Thus spec
[Equation]
Z
[Equation] [Equation]
[Equation]
= spec (cid:0) Mn (cid:1) .
Because y (cid:62) Λ − 1 y (cid:46) y
(cid:62) y , λ min (cid:18) M n
[Equation]
=
[Equation]
x
[Equation]
Similarly, we get λ
[Equation]
Therefore, there exists universal positive constants C 1 and C 2 such that spec
[Equation]
Thus in both cases, we have shown that there exists universal positive constants C 1 and C 2 such that spec
[Equation]
Deﬁne L 1 = C 1 λ min (cid:16) Z (cid:62) Zn (cid:17) and L 2 = C 2 λ max (cid:16) Z (cid:62) Zn (cid:17) .
As a result, we get
[Equation]
d
[Equation]
Deﬁne f ( s ) =
s ( λ + s ) 3 .
Because f (cid:48) ( s )
=
λ − 2 s ( λ + s ) 4 , the function f is increasing on [0 , λ/ 2] and decreasing on [ λ/ 2 , + ∞ ) .
If λ ≥ 2 L 2 or λ ≤ 2 L 1 , we get
[Equation]
s ∈ [ L 1 ,L 2 ]
[Equation]
If 2 L 1 < λ < 2 L 2 , we get
[Equation]
As a result, for all sufﬁciently large n , we have
[Equation]
where the ﬁnal inequality is because of Lemma 3.
We can exchange the expectation and differentiation and obtain d
[Equation]
and
[Equation]
[Table]
C L EMMAS ON S TIELTJES T RANSFORM
Deﬁnition 3 (Stieltjes transform) .
The Stieltjes transform of a distribution with cumulative distribution function F is deﬁned by
[Equation]
Lemma 7 (Theorem 4.3 (Bai & Silverstein, 2010)) .
Suppose that the entries of X n ∈ C n × p are complex random variables that are independent for each n and identically distributed for all n and satisfy E (cid:104) | x 11 − E x 11 | 2 (cid:105)
= 1 .
Also, assume that T n = diag( τ 1 , . .
. , τ p ) , τ i is real, and the empirical distribution function of { τ 1 , . . .
, τ p } converges almost surely to a probability distribution function H as n → ∞ .
The entries of both X n and T n may depend on n , which is suppressed for brevity.
Set B n =
A n + 1 n X n T n X ∗ n , where X ∗ n is the conjugate transpose of X n , A n is Hermitian, n × n satisfying F A n → F A almost surely, where F A is a distribution function (possibly defective) on the real line.
Assume also that X n , T n , and A n are independent.
When p = p ( n ) with p/n → y > 0 as n → ∞ , then, almost surely, F B n , the empirical spectral distribution of the eigenvalues of B n , converges vaguely, as n → ∞ , to a (nonrandom) distribution function F , where for any z ∈ C + = { z ∈ C | (cid:61) z > 0 } , its Stieltjes transform s = s ( z ) is the unique solution in C + to the equation
[Equation]
where s A is the Stieltjes transform of F A .
Lemma 8.
If the functions f α , g α : I → R satisfy f α ( x ) −
g α ( x ) → 0 uniformly as α → + ∞ , then lim α → + ∞ (inf x
∈
I f ( x ) − inf x ∈ I g ( x ))
= 0 .
Proof.
Because f α ( x ) − g α ( x ) → 0 uniformly as α → + ∞ , we have for ∀ (cid:15) > 0 , there exists N ( (cid:15) ) such that for ∀ α > N ( (cid:15) ) and ∀ x ∈
I , it holds that |
f α ( x ) − g α ( x ) | < (cid:15) .
Therefore, we get
[Equation]
Thus we obtain
[Equation]
which in turn implies
[Equation]
It follows that | inf x ∈ I
f α ( x ) − inf x ∈
I g α
( x ) | ≤ (cid:15) .
In other words, we proved
[Equation]
Lemma 9.
Deﬁne N = λnI n + Z (cid:62) Λ Z .
Then we have
[Equation]
Proof.
Proof of Equation (26).
We apply Lemma 7 with A n = 0 n × n , X n = Z (cid:62) ∈ R n × d ,
T n = Λ , and B n = 1 n Z (cid:62) Λ Z .
The distribution function of 0 n × n converges to 1 t ≤ 0 and its Stieltjes transform is s A ( z )
= (cid:82)
[Equation]
−
[Equation]
d
[Equation]
≤ 0 = − 1 z .
The empirical distribution function of { λ 1 , . .
, λ 1 (cid:124)
[Equation] [Equation]
[Equation]
m (cid:124) (cid:123)(cid:122) (cid:125)
d m } is H n,d i ( t ) =
(cid:80) i ∈ [ m ] d i d 1 t ≤ λ i .
Recall d i /n
→ z i .
Thus d i /d
→
z i /K , where d/n → y = (cid:80) j ∈ [ m ] z j .
The empirical distribution function converges to
H ( t ) =
(cid:80) i ∈ [ m ] z i y 1 t ≤ λ i .
Then the empirical spectral distribution of the eigenvalues of 1 n Z (cid:62) Λ Z converges vaguely to a nonrandom distribution function F and its Stieltjes transform is
[Equation]
(this is because of (Bai & Silverstein, 2010, Theorem B.9)).
By Lemma 7, s ( z ) is the unique solution in C + to the equation
[Equation]
which gives
[Equation]
We want to prove Equation (26) ﬁrst.
The lefthand side of Equation (26) equals
[Equation]
Because the matrix 1 n Z (cid:62) Λ Z is positive semideﬁnite and thereby all of its eigenvalues are nonnegative, its limiting spectral distribution is supported on [0 , ∞ ) .
The Stieltjes transform s ( z ) of the limiting spectral distribution can be continuously extended to ( −∞ , 0) .
Therefore, for ∀ λ > 0 , s ( − λ ) is the unique solution to the following equation
[Equation]
We will verify that
[Equation]
satisﬁes Equation (28).
Take a minimizer ρ ∗ of Equation (5).
Using the envelope theorem yields
[Equation]
Plugging the righthand side of Equation (29) into Equation (28)
[Equation]
we get
[Equation]
Rewriting the above equation yields
[Equation]
It sufﬁces to show that each summand on the lefthand side equals its counterpart on the righthand side
[Equation]
We need to show
[Equation]
which is equivalent to Equation (6) and therefore holds.
Hence we have proved Equation (26).
Proof of Equation (27).
We use α to denote the indices n, d i .
Deﬁne
[Equation]
First, we want to show that lim λ 0 → + ∞ ( h ( λ 0 ) − log λ 0 )
= 0 .
Deﬁne
[Equation]
The Hessian matrix of q ( p ) is diag (cid:16) z 1 ρ 21 , . .
z m ρ 2 m (cid:17) , which is positive deﬁnite since z i , ρ i > 0 .
Therefore, q ( p ) is convex and the minimum of q ( ρ ) on R m + is attained at ρ
= z , where z = ( z 1 , . . .
, z m ) (cid:62) .
The minimum is inf ρ ∈ R m + q ( ρ ) =
q ( z ) = 0 .
Because lim (cid:107) ρ (cid:107) 2 → + ∞ l λ 0 ( ρ ) = + ∞ , there exists a universal constant K 1 > (cid:107) z (cid:107) 2 > 0 such that l λ 0 ( ρ ) > l λ 0 ( z ) for all (cid:107) ρ (cid:107)
>
K 1 .
Deﬁne E = (cid:8) ρ ∈ R m + | (cid:107) ρ (cid:107)
≤ K 1 (cid:9) .
We have z ∈ E , inf ρ ∈ E l λ 0 ( ρ ) = inf ρ ∈ R m
+ l λ 0 ( ρ ) , and inf ρ ∈ E q ( ρ ) = inf ρ ∈ R m + q ( ρ ) = 0 .
Therefore, we get
[Equation]
On E , there exists a universal constant K 2 > 0 such that (cid:80) j ∈ [ m ] λ j ρ j < K 2 .
Thus on E , we deduce
[Equation]
The right-hand side log (cid:16) 1 + K 2 λ 0 (cid:17) → 0 as λ 0 → + ∞ .
Thus lim λ 0 → + ∞ ( l λ 0 ( ρ ) − q ( ρ )) =
uniformly for ρ ∈ E .
By Lemma 8, we get
[Equation]
Recalling Equation (30) yields
[Equation]
Deﬁne f α ( λ ) =
1 n log det Nn .
Second, we want to show lim α f α ( λ )
=
h ( λ ) , where lim α means lim n,d i → + ∞ d i /n
=
z i .
We have f α ( λ ) − f α ( λ 0 ) =
(cid:82) λ λ 0 f (cid:48)
α ( x ) dx for ∀ λ, λ 0 > 0 .
It follows that
[Equation]
Taking lim sup α on both sides gives lim sup
[Equation]
Recall f (cid:48)
α ( λ ) =
tr N − 1 and lim α
f
(cid:48)
α ( λ ) =
h (cid:48) ( λ ) (this is exactly Equation (26)).
Because (cid:12)(cid:12) tr N − 1 (cid:12)(cid:12) =
tr N − 1 ≤ 1 λ and (cid:82) λ λ 0 1 x dx < + ∞ , by the dominated convergence theorem, we have
[Equation]
It follows that
[Equation]
Since
[Equation]
and the matrix 1 nλ 0 Z (cid:62) Λ Z is positive semideﬁnite, we have
[Equation]
We have
[Equation]
Then taking lim sup α , we get
[Equation]
where the last inequality is because lim sup α λ
[Equation]
1 by Lemma 1.
Using Equation (32), Equation (33) and Equation (34) gives
[Equation]
Then taking lim λ 0 → + ∞ and recalling Equation (31) yields
[Equation]
Therefore, we conclude lim
α f α ( λ ) =
h ( λ ) .
Lemma 10.
Deﬁne N = λnI n + Z (cid:62) Λ Z = λnI n + (cid:80) i ∈ [ m ] λ i
Z i Z (cid:62)
i .
The following equation holds
[Equation]
Proof.
Proof of Equation (35).
We use α to denote the indices n, d i and use lim α to denote lim n,
[Equation]
i .
Deﬁne
[Equation]
and
[Equation]
We have
[Equation]
By Lemma
[Equation]
there exists a universal constant C > 0 such that for all sufﬁciently large n ,
[Equation]
Therefore
[Equation]
we get
[Equation]
≤ log
[Equation]
λ
[Equation]
C ) .
By the dominated convergence theorem and Lemma 9 (speciﬁcally
[Equation]
Equation (27)), we obtain
[Equation]
Because
[Equation]
and E (cid:2) 1 λn 2 tr (cid:0) Z (cid:62)
i Z i (cid:1)(cid:3) <
+ ∞ , we can interchange the differentiation and the expectation and get
[Equation]
Thus we deduce
[Equation]
∂
∂λ i E
[Equation]
By Lemma 4, E tr (cid:0) Z (cid:62) i Z i (cid:1) (cid:16) n 2 and therefore E (cid:2) 1 λn 2 tr (cid:0) Z (cid:62)
i Z i
[Equation]
(cid:46) 1 λ .
The function sequence { f (cid:48) α } is uniformly bounded.
Then we want to show that { f (cid:48) α } is uniformly equicontinuous by showing that { f (cid:48)(cid:48) α } is uniformly bounded.
Because
[Equation]
and E (cid:20) 1 nλ 2 tr (cid:16) Z (cid:62)
i Z i n (cid:17) 2 (cid:21) <
+ ∞ , we can interchange the differentiation and the expectation and get
[Equation]
Therefore, we deduce
[Equation]
Again, by Lemma 4, tr (cid:16) Z (cid:62) i Z i n (cid:17) 2 (cid:16) n .
It follows that 1 nλ 2 E tr (cid:16) Z (cid:62) i Z i n (cid:17) 2 (cid:46) 1 λ 2 .
Therefore { f (cid:48) α } is uniformly equicontinuous.
We want to show lim α f (cid:48)
α ( λ i ) =
h (cid:48)
( λ i ) by contradiction.
If it is not true, there exists (cid:15) > 0 and a subsequence { f (cid:48)
α k } such that (cid:12)(cid:12) f (cid:48)
α k ( λ i ) − h (cid:48)
( λ i ) (cid:12)(cid:12) ≥ (cid:15) .
Let E =
[ a, b ] (cid:51)
λ i ( b > a > 0 ) be a closed interval that contains λ i .
The subsequence { f (cid:48)
α k } is uniformly bounded and uniformly equicontinuous.
By the Arzela-Ascoli theorem, there exists a subsequence (cid:110) f (cid:48)
α kj (cid:111) that converges uniformly on λ i ∈ E .
Recall lim
α f α ( λ i ) =
h ( λ i ) (Equation (37)).
Thus lim
j f α kj ( λ i )
=
h ( λ i ) .
By (Rudin, 1976, Theorem 7.17), for λ i ∈ E , we have
[Equation]
This is a contradiction.
Hence, we have shown that lim
α f (cid:48) α ( λ i ) =
h (cid:48)
( λ i ) , which is exactly Equation (35) (recall f (cid:48)
α ( λ i ) =
∂∂λ i E (cid:2) 1 n log det Nn (cid:3) =
E (cid:104)
∂∂λ i 1 n log det Nn (cid:105) in Equation (38)).
Proof of Equation (36).
Deﬁne g α
( λ j ) =
∂∂λ i E (cid:2) 1 n log det Nn (cid:3) =
E (cid:104)
∂∂λ i 1 n log det Nn (cid:105) .
Then g (cid:48)
α ( λ j ) =
[Equation]
=
[Equation]
j E (cid:104) ∂∂λ
[Equation]
log det Nn (cid:105) .
We have
[Equation] [Equation]
where the last inequality is because (cid:13)(cid:13)
N − 1 (cid:13)(cid:13) 2 F ≤ (cid:13)(cid:13)
λn I n (cid:13)(cid:13) 2 F = 1 λ 2 n .
If i (cid:54) =
j , by Lemma 4, we have
[Equation]
If i = j , by Lemma 4, we have
[Equation]
As a result, we get
[Equation]
Thus we can interexchange ∂∂λ j and expectation, and get g (cid:48)
α ( λ j ) =
E (cid:104) ∂
∂λ j ∂λ i 1 n log det Nn (cid:105) .
Because | g (cid:48) α ( λ j ) | ≤ E (cid:12)(cid:12)(cid:12) ∂ 2 ∂λ j ∂λ i 1 n log det Nn (cid:12)(cid:12)(cid:12) (cid:46) 1 λ 2 , the function sequence { g (cid:48)
α } is uniformly bounded for λ j .
Deﬁne L = Z (cid:62) j N − 1 Z i and W =
Z (cid:62) j N − 1 Z j .
We have
[Equation]
where the ﬁrst inequality is because W (cid:52) 1 λn Z (cid:62) j Z j and the third inequality is because N − 1 (cid:52) 1 λn I n and then (cid:13)(cid:13)
N − 1 (cid:13)(cid:13) 2 F ≤ (cid:13)(cid:13)
λn I n (cid:13)(cid:13)
[Equation]
F ≤ 1 λ 2 n .
By Lemma 4, we have E (cid:107) Z j (cid:107)
(cid:46) n 2 and E (cid:107)
Z i (cid:107)
(cid:46) n .
If i (cid:54) =
j , then Z j and Z i are independent, and we deduce
[Equation]
If i = j , we have
[Equation]
As a result, we deduce E (cid:104) ∂
3 ∂λ
j ∂λ i 1 n log det Nn (cid:105) =
∂∂λ j E (cid:104) ∂
∂λ j ∂λ i 1 n log det Nn (cid:105) =
g (cid:48)(cid:48) α
( λ j ) .
Moreover, we have
[Equation]
Therefore { g (cid:48) α } is uniformly equicontinuous.
Deﬁne
[Equation]
i (cid:18)
[Equation]
(cid:19)(cid:19) .
We want to show by contradiction that lim α g (cid:48)
α ( λ j ) =
w (cid:48)
( λ j ) .
Assume that it is not true.
Then there exists (cid:15) > 0 and a subsequence (cid:8) g (cid:48)
α k (cid:9) such that
(cid:12)(cid:12) g (cid:48)
α k ( λ j ) − w (cid:48)
( λ j ) (cid:12)(cid:12) > (cid:15) .
Since (cid:8) g (cid:48)
α k (cid:9) is uniformly bounded and uniformly equicontinuous, by the Arzela-Ascoli theorem, there is a subsequence (cid:110) g (cid:48)
α kr (cid:111) that converges uniformly on a closed interval E containing λ j .
Equation (35) shows that lim
α g α ( λ j ) =
w ( λ j ) .
It follows that lim r g α kr ( λ j ) =
w ( λ j ) .
By (Rudin, 1976, Theorem 7.17), for λ i ∈ E , we have
[Equation]
which is a contradiction.
Therefore, we have shown that lim α g (cid:48) α ( λ j ) = w (cid:48) ( λ j ) , which is exactly
Equation (36).
D P ROOF OF T HEOREMD. P ROOF OF I TEM
[Equation]
The function g ( ρ ) is continuously differentiable on R m + .
The boundary of R + m is ∂ R m + =
[Equation] [Equation]
i
[Equation]
Because lim R m +
[Equation]
g ( ρ ) =
+ ∞ , there exists a minimizer ρ ∗ ∈ R m + of g ( ρ ) .
Taking the derivative with respect to ρ i gives
[Equation]
Setting it to zero gives Equation (6)
[Equation]
D. P ROOF OF I TEM
Recall Equation (6)
[Equation]
Rewriting the above equation gives
[Equation]
Rewriting it in the linear algebraic form yields
[Equation]
Applying ∂∂
λ to both sides and using the implicit function theorem, we get
[Equation]
Arranging the above equation yields
[Equation]
Deﬁne a = λ + λ (cid:62) ρ ∗ , A = diag ( λ ) + (cid:0) λ + λ (cid:62) ρ ∗ (cid:1) I m = diag
( λ ) + aI m and B = diag ( λ ) + (cid:0) λ
+ λ (cid:62)
[Equation]
I m −
[Equation]
z − ρ
[Equation]
λ
[Equation]
=
[Equation]
( z − ρ ∗ ) λ (cid:62) .
The matrix determinant lemma gives
[Equation]
Recall Equation (6) again and we have
[Equation]
We have
[Equation]
It follows that (cid:80)
[Equation]
a < 1 .
Then we compute λ (cid:62) A − 1 ( z − ρ ∗ ) :
[Equation]
Thus we get 1 − λ (cid:62) A − 1 ( z − ρ ∗ ) > 0 .
Therefore, det B (cid:54) = 0 and the matrix B is invertible.
D. P ROOF OF I TEM
Lemma 11.
Deﬁne N = λnI n + Z (cid:62) Λ Z , γ = (cid:80) i ∈ [ m ] z i , r = ( r 1 , . . .
, r m ) , λ = ( λ 1 , . . . , λ m ) , and ϑ ( r
[Equation]
[Equation] [Equation]
[Equation] [Equation]
− λr
[Equation]
For any K t ≥ 2 λ and K u ≥ 2 λ + ( 2+ √ γ ) λ , we have lim n
[Equation]
If r ∗ is a solution to the optimization problem in Equation (39)
[Equation]
then
[Equation]
Moreover
[Equation]
we have
[Equation]
Proof.
Let g ∼ N (0 , I n ) be a multivariate standard normal random vector.
We have
[Equation]
[Table]
We view
[Equation]
as the primal optimization (PO) problem in the convex Gaussian min-max theorem (CGMT) (Thrampoulidis et al., 2015).
The KKT conditions for Equation (42) give
[Equation]
Solving the above equations gives
[Equation]
With probability at least 1 − 4 exp( − cn ) ( c > 0 is a universal constant), we have (cid:107) g (cid:107) 2 ≤ 2 √ n and (cid:107) Z (cid:107) ≤ √ d + 2 √ n ≤ (cid:0) 2 + √ γ (cid:1) √ n .
Therefore, we get
[Equation]
Write u =   u 1 .
u m   , where u
i ∈ R d i .
For all K t ≥ 2 λ , K u ≥ 2 λ
+ ( 2+ √ γ
) λ , the optimal solutions t ∗ and u ∗ to Equation (42) satisfy √ n (cid:107) t ∗ (cid:107) 2 ≤ K t and (cid:107)
u
i (cid:107) 2 ≤ K u for all i ∈ [ m ] with probability at least 1 − 4 exp( − cn ) .
Deﬁne S t
= { t ∈ R n | √ n (cid:107) t (cid:107) 2 ≤ K t } and
S u
= (cid:8) u ∈ R
d
| (cid:107) u
i (cid:107) ≤ K u , ∀ i ∈ [ m ] (cid:9) .
We use α to denote the indices n, d i and use lim α to denote lim n,d i → + ∞ d i /n
=
z i .
Deﬁne event
[Equation]
Then with probability at least 1 − 4 exp( − cn ) , we have t ∗ ∈ S t and u ∗ ∈ S u .
Therefore the event
E α occurs with probability at least 1 − 4 exp( − cn ) , which yields
P { E cα } ≤ 4 exp( − cn ) .
Since (cid:80) n ≥ 1 4 exp( − cn ) < + ∞ , by Borel-Cantelli lemma, we have
[Equation]
Then with probability 1 , all but ﬁnitely many E α occur.
Then almost surely there exists n 0 such that for all n > n 0 , E α occurs.
The auxiliary optimization (AO) problem is inf
[Equation]
u
[Equation]
=
[Equation]
λ
[Equation]
r
[Equation]
+ 1 2 λr
[Equation]

 , where
Taking n, d i → + ∞ with d i /n
→ z i constant, the strong law of large numbers gives
[Equation]
Deﬁne
[Equation]
It is a stochastic process on ( r t , r ) ∈ [0 , K t ] × [0 , K u ] m .
We have
[Equation]
almost surely.
Since √ 1 + x 2 is convex and increasing and the function (cid:107) r (cid:107) 2 is convex, thus (cid:113)
+ (cid:107) r (cid:107) 22 is convex in r and then − (cid:113) 1 + (cid:80)
[Equation]
∈ [ m ]
[Equation]
is concave in r .
Because − 12 (cid:80) i ∈ [ m ] 1 λ
i r 2 i is concave in r and r t (cid:80) i ∈ [ m ] (cid:107) g
[Equation]
i
[Equation]
2 √ n r i is linear in r , we deduce that X α ( r t , r ) is concave in r .
By (Liese & Miescke, 2008, Lemma 7.75), sup r ∈ [0 ,K u ] m | X α ( r t , r ) − X ( r t , r ) | → 0 almost surely.
Then for ∀ (cid:15) > 0 , there exists n 0 ( (cid:15) ) , d 0 ,i ( (cid:15) ) , δ 0 ,i ( (cid:15) )
such that for all n > n 0 ( (cid:15) ) , d i > d 0 ,i ( (cid:15) ) , | d i /n
− z i | < δ 0 ,i ( (cid:15) ) and for all r ∈ [0 , K u ] m , we have
[Equation]
Thus we obtain
[Equation] [Equation]
which in turn implies
[Equation]
sup r ∈ [0 ,K u ] m X ( r t , r ) + (cid:15) .
It follows that (cid:12)(cid:12)(cid:12) sup r ∈ [0 ,K u ] m X α ( r t , r ) − sup r ∈ [0 ,K u ] m X ( r t , r ) (cid:12)(cid:12)(cid:12) ≤ (cid:15) .
In other words, we showed | Y α ( r t ) − Y ( r t ) | →almost surely, where Y ( r t ) :
= sup r ∈ [0 ,K u ] m X α ( r t , r ) and Y ( r t ) :
= sup r ∈ [0 ,K u ] m X ( r t , r ) .
Because X α ( r t , r ) is convex in r t , then Y ( r t ) =
sup r ∈ [0 ,K u ] m X α ( r t , r ) is convex in r t .
By (Liese & Miescke, 2008, Lemma 7.75) again, sup r t ∈ [0 ,K t ] | Y α ( r t ) − Y ( r t ) | → 0 almost surely.
A similar argument shows that
[Equation]
almost surely.
Therefore
[Equation]
we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Recall
[Equation]
We have A α ∩ E α ⊆ B α .
Equation (43) gives lim α P { C α } = 0 for any τ > 0 because almost sure convergence implies convergence in probability.
By the convex Gaussian min-max theorem (Thrampoulidis et al., 2015), we have
[Equation]
It follows that
[Equation]
Taking lim sup α on both sides, because lim sup α P { B α } ≤ 2 lim sup α P { C α } = 0 , we get
[Equation]
where the second inequality is because of the reverse Fatou’s lemma.
Thus
[Equation]
Therefore
[Equation]
we deduce g
[Equation]
Because (cid:12)(cid:12) g (cid:62) N − 1 g (cid:12)(cid:12) ≤ 1 λn (cid:107) g
(cid:107) 22 and E
1 λn (cid:107) g (cid:107) 22 = 1 λ < ∞ , by the dominated convergence theorem for convergence in probability (Cohn, 2013, Proposition 3.1.6)
, we get
[Equation] [Equation]
Note that 2 r t (cid:113) 1 + (cid:80) i ∈
[ m ] r 2 i is convex in r , − 2 r t (cid:80) i ∈ [ m ]
√ z
i r i is linear in r , and (cid:80) i ∈
[ m ] 1 λ i r
i is strongly convex in r .
Thus ϑ is strongly convex in r .
Note that 2 r t (cid:113) 1 + (cid:80) i ∈
[ m ] r 2 i − 2 r t (cid:80) i ∈
[ m ] √ z i r i is linear in r t and that − λr 2 t is strongly concave in r t .
Thus ϑ is strongly concave in r t .
Then ϑ has a unique saddle point ( r ∗ t , r ∗ ) on [0 , K t ] × [0 , K u ] m that satisﬁes
[Equation]
where the ﬁrst equality is due to Sion’s minimax theorem.
Since (cid:12)(cid:12) tr N − 1 (cid:12)(cid:12) ≤ 1 λ , using the dominated convergence theorem and combining Equation (45) and
Equation (46) yields
[Equation]
By the uniqueness of the limit, the right-hand side max 0 ≤ r t ≤ K t min 0 ≤ r i ≤ K u ϑ ( r t , r , λ ) and min 0 ≤ r i ≤ K u max 0 ≤ r t ≤ K t ϑ ( r t , r , λ ) do not depend on K t and K u as long as K t ≥ 2 λ andK u ≥ 2 λ +
[Equation]
2+ √ γ ) λ .
Thus we have
[Equation]
If r ∗ t = 0 , then ϑ (0 , r ∗ ) = min r ∈ [0 ,K u ] m (cid:80) i ∈
[ m ] 1 λ i r 2 i = 0 .
Thus r ∗ must be zero.
However, ϑ (cid:0) 12 λ , 0 (cid:1) = 34 λ > ϑ (0 , r ∗ ) .
Therefore r ∗ t > 0 .
We compute the partial derivative
[Equation]
If r ∗ i = 0 , we have
[Equation]
Therefore, one can increase r ∗ i and make max r t ∈ [0 ,K t ] min r ∈ [0 ,K u ] m ϑ ( r t , r ) smaller, which results in a contradiction.
Thus r ∗ i > 0 .
Thus the minimax value is attained when r t , r i > 0 for all i ∈ [ m ] .
To obtain the optimality condition, we compute the partial derivatives
[Equation]
Setting them to zero gives the optimality condition for r ∗ t , r ∗ 1 , . .
, r ∗ m and yields Equation (40) and
Equation (41).
Using the envelope theorem, we get
[Equation]
Lemma 12.
Deﬁne N = λnI n + Z (cid:62) Λ Z .
The following equation holds lim
[Equation]
z i
[Equation]
where r ∗ is a solution to sup r t > 0 inf r 1 ,...,r m >
ϑ ( r t , r 1 , . . .
, r m , λ 1 , . . .
, λ m ) and ϑ ( r t , r 1 , . .
, r m , λ 1 , . .
[Equation]
Proof.
Since
[Equation]
and E 1 ( λn ) 2 tr (cid:0) Z
[Equation]
i Z i (cid:1) (cid:16) 1 λ 2 (by Lemma 4), using the dominated convergence theorem gives
[Equation]
We use α to denote the indices n, d i and use lim α to denote lim n,d i → + ∞ d
[Equation]
n = z i .
Deﬁne
[Equation]
E
[Equation]
ρ i − z i (cid:16) log ρ
i z i
+ 1 (cid:17)(cid:17)(cid:105) , and h ( λ i ) =
sup r t > 0 inf r 1 ,...,r m > 0 ϑ ( r t , r 1 , . . .
, r m , λ 1 , . . .
, λ m ) .
Because (cid:12) (cid:12) tr (cid:0) N − 1 (cid:1)(cid:12) (cid:12) ≤ tr (cid:0) 1 λn I n (cid:1) ≤ 1 λ and lim α tr (cid:0) N − 1 (cid:1) =
g ( λ i ) (by Lemma 9), we have Lemma 9 lim α f α
( λ i ) =
lim α E tr (cid:0) N − 1 (cid:1) =
g ( λ i ) .
Lemma 11 shows lim α f α ( λ i )
=
h ( λ i ) .
Therefore lim
α f α ( λ i ) =
g ( λ i ) =
h ( λ i ) .
Because of Equation (47), we have f (cid:48)
α ( λ i ) =
∂∂λ i E tr (cid:0) N − 1 (cid:1) and
[Equation]
and therefore { f (cid:48)
α } is uniformly bounded for λ i .
Because
[Equation]
and E 1 ( λn ) 3 tr (cid:0) Z i Z (cid:62) i (cid:1) 2 (cid:16) 1 λ 3 (by Lemma 4)
[Equation]
using the dominated convergence theorem yields
[Equation]
Moreover
[Equation]
we have
[Equation]
Thus { f (cid:48) α } is uniformly equicontinuous for λ i .
We want to show that lim
α f (cid:48)
α ( λ i ) =
g (cid:48)
( λ i ) by contradiction.
Assume that it is not true.
Then there exists (cid:15) > 0 and a subsequence (cid:8) f (cid:48)
α k (cid:9) such that (cid:12)(cid:12) f (cid:48)
α k ( λ i ) − g (cid:48)
( λ i ) (cid:12)(cid:12) > (cid:15) .
Since (cid:8) f (cid:48)
α k (cid:9) is uniformly bounded and uniformly equicontinuous for λ
i ∈ E ( E is any closed ﬁnite interval containing λ i ), by the Arzela-Ascoli theorem, there exists a subsequence (cid:110) f (cid:48)
α k r (cid:111) that converges uniformly on E .
Since lim α f α kr ( λ i ) =
g ( λ i ) , by (Rudin, 1976, Thoerem 7.17), we have
[Equation]
which yields a contradiction.
Therefore, we have lim
α f (cid:48)
α ( λ i ) =
g (cid:48)
( λ i ) .
Recall g ( λ i ) =
h ( λ i ) for any λ i > 0 .
Then by the ﬁnal part of Lemma 11, we have lim
α f (cid:48)
α ( λ i ) =
g (cid:48)
( λ i ) =
h (cid:48)
( λ i ) =
− r ∗ 2 i λ 2 i .
D.
B IAS
Lemma 13.
Suppose that U ∼ (cid:76) i ∈ [ m ] Unif ( O ( d i )) and V are two independent d × d random matrices such that V d = UV U (cid:62) , where d = (cid:80) mi =1 d i .
Let θ ∈ R
d be a ﬁxed vector.
Write θ =   θ 1 .
θ m 
 , where θ i ∈ R d i .
Let φ ∼ (cid:77) i ∈ [ m ]
Unif (cid:0) S d i − 1 ( (cid:107) θ i (cid:107) 2 ) (cid:1) be a random vector independent of V and let Λ = diag ( λ 1 I d 1 , . .
λ m I d m ) ∈ R d × d .
Then we have
[Equation] [Equation]
(cid:21) .
Proof.
Recall U Λ U (cid:62) =
Λ and noticing U (cid:62) θ
d = φ , we get
E
[Equation] [Equation]
=
[Equation]
=
[Equation]
Lemma 14.
Deﬁne
[Equation]
and
S = Λ
[Equation]
/ 2 Z (cid:0) nλI n + Z (cid:62) Λ Z (cid:1) − 1 Z (cid:62) Λ 1 / 2 .
Then we have
[Equation]
Proof.
Recall Equation (22) in Lemma
[Equation]
Let U ∼ (cid:76) i ∈ [ m ] Unif ( O ( d i )) be a random matrix independent of Z .
Because UZ d = Z , we have
I
[Equation]
I d + 1 nλ Λ 1 / 2 ZZ (cid:62) Λ 1 / 2 (cid:19) U (cid:62) .
Deﬁne ˜ θ ∼ (cid:76) i ∈ [ m ] Unif( S d i − 1 ( (cid:107) θ (cid:48)
i (cid:107) 2 )) .
Lemma 13 gives
[Equation]
Notice that (cid:13)(cid:13)(cid:13)
˜
θ (cid:13)(cid:13)(cid:13) 2 Λ = (cid:107) θ (cid:48) (cid:107) 2Λ and ˜Θ
= E (cid:104) ˜
θ ˜ θ (cid:62) (cid:105) .
Because ˜Θ commutes with Λ , we have tr (cid:16) S Λ˜Θ (cid:17) = tr (cid:16) S ˜ΘΛ (cid:17) = tr (cid:16) Λ S ˜Θ (cid:17) .
In light of these, we deduce
[Equation]
Lemma 14 expresses the bias B λ,d,n as the sum of three terms.
Computing (cid:107) θ (cid:48) (cid:107) 2Λ
Note that (cid:107) θ (cid:48) (cid:107) 2Λ
= θ (cid:48)(cid:62) Λ θ (cid:48)
= (cid:80) i ∈
[ m ] λ i (cid:107) θ (cid:48)
i (cid:107) 22 .
Therefore, lim n,d i → + ∞ d i /n
→ z i (cid:107) Π i θ ∗ (cid:107)
→ η i (cid:107) θ (cid:48) (cid:107) 2Λ
=
q (cid:62) ( λ (cid:12) z ) .
Computing E tr (cid:16) Λ S ˜Θ (cid:17) Deﬁne N = λnI n + Z (cid:62) Λ Z = λnI n + (cid:80) i ∈ [ m ] λ i Z i Z (cid:62) i .
We have
[Equation]
where the second inequality is because ˜Θ commutes with Λ 3 / 2 and the ﬁnal equality is because of
Equation (38).
Taking lim n,d i → + ∞ d i /n
→ z i
[Equation]
2 → η i and using Lemma 10 gives
[Equation]
Using the envelope theorem yields
[Equation]
where the ﬁnal equality is because of Equation (6) in Item 1.
Therefore
[Equation]
we deduce
[Equation]
Computing E tr (cid:16) S Λ S ˜Θ (cid:17)
We have
[Equation]
where the third equality is because ˜Θ commutes with Λ 1 / 2 .
Taking lim n,d i
[Equation]
+ ∞ d
[Equation] [Equation]
and using
Lemma 10 gives
[Equation]
Write λ
= ( λ 1 , . . .
, λ m ) (cid:62) and
z = ( z 1 , . . .
z m ) (cid:62) .
Let ρ ∗ ∈ R
m be a minimizer of Equation (5) and J = ∂ρ ∗ ∂ λ ∈ R m × m be the Jacobian matrix J ij =
∂ρ ∗
i ∂λ j .
Recall Item
[Equation]
Using the envelope theorem
[Equation]
we have
[Equation]
Recall Equation (6) yields
[Equation]
Differentiating the above equation with respect to λ j gives
[Equation]
It follows that
[Equation]
Putting all three terms together, we have
[Equation]
Since { B λ,d,n } is uniformly bounded and uniformly equicontinuous for λ ∈ (0 , 1] by Lemma 5, { B λ,d,n } can be extended continuously to [0 , 1] and the family of extended functions is still uniformly bounded and uniformly equicontinuous for λ ∈ [0 , 1] .
By the Arzela-Ascoli theorem, { B λ,d,n } converges uniformly to the limit.
By the Moore-Osgood theorem, we can exchange the two limits
[Equation]
and lim λ → 0 + and get
[Equation]
D. V ARIANCE
Deﬁne N = nλI n + Z (cid:62) Λ Z .
Recalling Lemma 6 gives
[Equation]
Using Lemma
[Equation]
we get
[Equation]
Using the envelope theorem
[Equation]
we deduce
[Equation]
Then we take ∂∂λ i and obtain
[Equation]
As a result,
[Equation]
By Lemma
[Equation]
the variance is given by
[Equation]
where r ∗ solves
[Equation]
Since { V λ,d,n } is uniformly bounded and uniformly equicontinuous with respect to λ ∈ (0 , 1] by
Lemma 5, { V λ,d,n } can be extended continuously to [0 , 1] and the family of extended functions is still uniformly bounded and uniformly equicontinuous.
By the Arzela-Ascoli theorem, { V λ,d,n } converges uniformly to the limit.
By the Moore-Osgood theorem, we can exchange the two limits lim n,d i →∞ d i /n
→ z i and lim λ
→ 0 + and get lim n,d
[Equation]
E P ROOF OF T HEOREM
We use Theorem 1 to prove Theorem 2.
As in Theorem 1, let r ∗ solve min r i ≥ 0 max r t ≥
ϑ ( r t , r , λ ) , where ϑ is deﬁned in Equation (7).
Note that ϑ is a quadratic function of r t .
Deﬁne A = (cid:113)(cid:80) i ∈
[ m ] r 2 i + 1 , B = (cid:80) i ∈ [ m ] √ z i r i , A ∗ =
(cid:113)(cid:80) i ∈
[ m ] r ∗ 2 i + 1 , and B ∗ = (cid:80) i ∈ [ m ]
√ z i r ∗ i .
Then r ∗ t =
A − Bλ and we get min r i ≥ 0 max
[Equation]
Taking the partial derivative with respect to r i gives
[Equation]
Setting it to zero gives the optimality condition for r ∗ i :
[Equation]
It follows that
[Equation]
Some algebraic manipulation in the above equation yields
[Equation]
Deﬁne z
=
( z 1 , . . .
, z m ) .
Then (cid:107) z (cid:107)
=
(cid:80) i ∈ [ m ] z i .
By Cauchy–Schwarz inequality, if d/n →
[Equation]
i ∈ [ m ] z i <
[Equation]
Thus there does not exist r such that A = B .
If d/n → (cid:80) i ∈ [ m ] z i > 1 , then
A =
B is feasible for r .
For example, set
[Equation]
We have
[Equation]
If (cid:107) z (cid:107) 1 > 1 , since A = B is feasible, then
[Equation]
If (cid:107) z (cid:107) 1 < 1 , then A − B always holds.
To be precise, we have
A − B ≥ (cid:18)(cid:113) (cid:107) r (cid:107) 22 + 1 − (cid:107) r (cid:107) 2 (cid:19) ∨ (cid:18)(cid:18) 1 − (cid:113) (cid:107) z (cid:107) 1 (cid:19) (cid:107) r (cid:107) 2 (cid:19) .
If (cid:107) r (cid:107) 2 > 1 , then (cid:0) 1 − (cid:112) (cid:107) z (cid:107) 1 (cid:1) (cid:107) r (cid:107) 2 > 1 − (cid:112) (cid:107) z (cid:107) 1 .
If (cid:107) r (cid:107) 2 ≤ 1 , then (cid:113) (cid:107) r (cid:107) 22 + 1 −(cid:107) r (cid:107) 2 ≥ √ 2 − 1 .
Thus there exists a universal constant C
= (cid:0) 1 − (cid:112) (cid:107) z (cid:107) 1 (cid:1) ∨ (cid:0) √ 2 − 1 (cid:1) >
such that
A − B ≥ C 0 .
Recall Equation (48).
We have
[Equation]
Taking lim λ → 0 + , since A ∗ − B ∗ ≥ C 0 does not go to zero, we have r ∗
i
A ∗ − √ z i =
[Equation]
i ∈ [ m ] .
Then we get r ∗ 2 i 1 + (cid:80) j ∈ [ m ] r ∗ 2 j =
z i , i ∈ [ m ] .
Summing all i ∈ [ m ] yields (cid:107)
[Equation]
1 +
[Equation]
] r ∗ 2 i .
Therefore, we have
[Equation]
F P ROOF OF T HEOREM
Deﬁne A ∗ =
(cid:113)(cid:80) i ∈
[ m ] r ∗ 2 i + 1 and B ∗ = (cid:80) i ∈ [ m ]
√ z
i r ∗ i .
Equation (15) in Theorem 2 yields
[Equation]
Using the constraint A ∗ =
B ∗ , we get
[Equation]
Deﬁne q = r ∗ 1 r ∗ 2 .
We have the following equation
[Equation]
Solving the above equation yields
[Equation]
Here we discard the negative root.
Let x = r ∗ 21 + r ∗ 22 =
r ∗ 22 (cid:0) 1 + q 2 (cid:1) . ??
yields
[Equation]
Solving x from the above equation gives
[Equation]
Therefore,
[Equation]
OF T HEOREM
Instead of considering the θ ∗ speciﬁed in Equation (18), we ﬁrst consider a Bayesian setting where θ ∗ ∼ N (cid:0) 0 , 1 d I d (cid:1) .
Later, we will show that the setup in Equation (18) is asymptotically (as d i → ∞ ) equivalent to this Bayesian setting.
The precise meaning of equivalence will also be presented later.
Our strategy can be divided into two steps.
The ﬁrst step is to show that the Bayes risk of the
Bayes estimator is monotonically decreasing in the sample size n .
The second step is to translate the sample-wise monotonicity of the Bayes estimator to the excess risk of the optimally regularized estimator ˆ θ λ,d,n in the setup of Equation (18).
Recall that since we are interested in sample-wise monotonicity, we add a subscript n to X and y (they are deﬁned by Equation (1) in Section 1.1) to emphasize that they consist of n data items.
In this Bayesian setting, the likelihood function of θ ∗ is
L ( θ ∗ | X n , y n )
[Equation]
(cid:89)
[Equation]
The density of the prior of θ ∗ is proportional to exp (cid:16) − d 2 (cid:107) θ ∗ (cid:107) 22 (cid:17) .
Therefore, the posterior density of θ ∗ is given by
[Equation]
As a result, the posterior distribution of θ ∗ is Gaussian.
The Bayes estimator is
[Equation]
Taking the derivative with respect to θ gives
[Equation]
Setting the above equation to zero yields Σ (cid:16) ˆ θ
[Equation]
= 0 and therefore
[Equation]
The ﬁnal equality is because the posterior mean of a Gaussian distribution equals its mode.
Deﬁne the Bayes risk
[Equation]
Write X = R d
and Y = R .
Deﬁne
[Equation]
We have
[Equation]
where the third equality is because the conditional expectation minimizes the (cid:96) 2 loss.
Next, we want to show that R n +1 ≤ R n ,
i
[Equation]
e., the Bayes risk of the Bayes estimator is monotonically decreasing in the sample size n .
[Equation]
= R n .
Then we want to show that R n equals the Bayes risk of the optimally regularized estimator ˆ θ λ,n,d :
[Equation]
Since R n = inf ˆ θ : X n ×Y n → R d E θ ∗ ,X n , y n (cid:13)(cid:13)(cid:13) ˆ θ ( X n , y n ) − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ , we get
[Equation]
On the other hand, recalling ˆ θ
[Equation]
we deduce
[Equation]
Therefore we deduce R n = inf λ ≥
E θ ∗ ∼N (
[Equation]
, 1 d I d ) ,X n , y n (cid:13)(cid:13)(cid:13) ˆ θ λ,n,d − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ .
As a result, we establish sample-wise monotonicity of the Bayes risk of optimal regularized ˆ θ λ,n,d :
[Equation]
In what follows, we show that if θ ∗ is given by Equation (18), the excess risk of ˆ θ λ,n,d is asymptotically equal to its Bayes risk when θ ∗ ∼ N (0 , 1 d I d ) :
[Equation]
We abuse the notation in the above equation.
The θ ∗ in E X n , y n (cid:13)(cid:13)(cid:13) ˆ θ λ,n − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ satisﬁes Equation (18), while the θ ∗ in E θ ∗ ∼N (0 , 1 d I d ) E X n , y n (cid:13)(cid:13)(cid:13)
ˆ θ λ,n,d − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ follows a normal distribution N (0 , 1 d I d ) .
By
Lemma 5 and Lemma 6, if Σ = P Λ P (cid:62) and θ (cid:48) =
P (cid:62) θ ∗ are as deﬁned in Table 1 (where P is an orthogonal matrix and Λ
= diag
( λ 1 I d 1 , . . .
λ m I d m ) ∈
R d × d is a diagonal matrix), for ﬁxed θ ∗ we have
[Equation]
where every entry of Z ∈ R d × n follows i.i.d.
N (0 , 1) .
If θ ∗ ∼ N (0 , 1 d I d ) , we have θ (cid:48) ∼ N (0 , 1 d I d ) .
Since the variance term σ 2 E X n , y n
[Equation]
Λ Z
[Equation]
λnI n + Z
[Equation]
Λ Z (cid:1) −
[Equation]
22 (cid:105) does not depend on θ ∗ , the two variance terms cancel out and we get
[Equation]
For U ∼ (cid:76) i ∈ [ m ] Unif ( O ( d i )) , we have
[Equation]
By Lemma 13, for θ ∗ (and thereby θ (cid:48) ) speciﬁed in Equation (18), we get
E X n , y n
[Equation]
, where φ
[Equation]
In the Bayesian setting, if θ (cid:48) ∼ N (0 , 1 d I d ) , then U (cid:62) θ (cid:48) ∼ N (0 , 1 d I d ) .
We have
E θ (cid:48)
∼N (0 , 1 d I d ) ,X n , y n (cid:13)(cid:13)(cid:13) ˆ θ λ,n − θ ∗ (cid:13)(cid:13)(cid:13)
[Equation]
 
[Equation]
where ψ = U (cid:62) θ (cid:48) ∼ N (0 , 1 d I d ) .
Next, we want to couple φ and ψ .
Let s i i.i.d.
∼ Unif (cid:0) S d i − 1 (1) (cid:1) , h
i i.i.d.
∼ χ 2 ( d i ) , and deﬁne
[Equation]
=
[Equation]
We have (cid:107) φ (cid:107) 2 =
and
[Equation]
By the strong law of large numbers, lim d i → + ∞ h
i /d i
= 1 almost surely.
Thus we get lim d i → + ∞ ,d i /d
→ ν i (cid:107) ψ (cid:107)
= (cid:112)(cid:80) mi =
ν i and lim d i → + ∞ ,d i /d
→ ν i (cid:107) φ − ψ (cid:107) 2 = 0 almost surely (recall that we will let d i → + ∞ and d i /d → ν i for some constant ν i > 0 . ).
Because (cid:13)(cid:13) Λ 1 / 2 (cid:13)(cid:13) 2 (cid:46) 1 and (cid:13)(cid:13)(cid:13)(cid:0)
I d + 1 nλ Λ 1 / 2 ZZ (cid:62) Λ 1 / 2 (cid:1) − 1 (cid:13)(cid:13)(cid:13) 2 ≤ (cid:107) I
d (cid:107) 2 = 1 , we bound the norm of
Q (cid:44) Λ 1 / 2 (cid:0) I
d + 1 nλ Λ 1 / 2 ZZ (cid:62) Λ 1 / 2 (cid:1) − 1 as follows
[Equation]
It follows that
[Equation]
where the last inequality is because (cid:107) φ (cid:107)
[Equation]
+ (cid:107) ψ (cid:107)
(cid:46) 1 for all sufﬁciently large d i .
We know that lim d i → + ∞ ,d i /d
→ ν i (cid:107) φ − ψ (cid:107) 2 = 0 almost surely.
To apply Lebesgue’s dominated convergence theorem, we need to ﬁnd a dominating integrable random variable.
In fact, 1 + (cid:107) ψ (cid:107) 2 dominates (cid:107) φ − ψ (cid:107) 2 : (cid:107)
[Equation]
It is integrable because E (cid:107) ψ (cid:107) 2 = E (cid:20)(cid:113) χ 2 ( d ) d (cid:21) ≤ (cid:113) E
[ χ 2 ( d )]
d = 1 .
Application of Lebesgue’s dominated convergence theorem yields
[Equation]
Therefore, we conclude that
[Equation]
and this convergence is uniform in n and λ ∈ (0 , ∞ ) .
It follows that
[Equation]
and this convergence is uniform in λ ∈ (0 , ∞ )
[Equation]
By Lemma 8 (the proof is similar when we replace α → + ∞ by n, d i → ∞ , n/d i → γ i ), we have
[Equation]
Deﬁne f α ( λ ) =
E θ ∗ ∼N (0 , 1 d I d ) E X n , y n (cid:13)(cid:13) (cid:13) ˆ θ λ,n,d − θ ∗ (cid:13)(cid:13) (cid:13) 2 Σ .
We use α to denote the indices n, d i .
By
Lemma 5 and Lemma 6, we have
[Equation]
Therefore { f α ( λ ) } is uniformly bounded for λ > 0 .
Since (cid:12)(cid:12)(cid:12)(cid:12) ddλ E X n , y n (cid:13)(cid:13)(cid:13)
ˆ θ λ,n − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ (cid:12)(cid:12)(cid:12)(cid:12) (cid:46) (cid:107) θ ∗ (cid:107)
and
E θ ∗ ∼N (0 , 1 d I d ) (cid:107) θ ∗ (cid:107)
= 1 , we have
[Equation]
d dλ
[Equation]
As a result, { f α ( λ ) } is uniformly equicontinuous for λ > 0 , and in particular λ ∈ (0 , M ] for any
M > 0 .
Therefore { f α ( λ ) } can be extended continuously to [0 , M ] and the family of extended functions is still uniformly bounded and uniformly equicontinuous.
Recall that if θ ∗ ∼ N (0 , 1 d I d ) , we have θ (cid:48) ∼ N (0 ,
1 d I d ) .
As in Equation (19), write θ (cid:48) in a row-partitioned form
[Equation]
where θ (cid:48) i ∈
R d i .
Then (cid:107) Π
i θ ∗ (cid:107)
= (cid:107) θ (cid:48)
i (cid:107) 2 ∼ (cid:113) χ 2 ( d i ) d
= (cid:113) χ
2 ( d i )
d i · d i d → √ ν i as n, d i → + ∞ and n/d i → γ i , where ν
i = (cid:16) γ i (cid:80) j ∈ [ m ] 1 γ j (cid:17) − 1 .
By Theorem 1, { f α ( λ ) } converges pointwise, say, to h ( λ, γ 1 , . .
γ m ) .
By the Arzela-Ascoli theorem, lim α f α ( λ )
=
h ( λ, γ 1 , . .
γ m ) uniformly on λ ∈ [0 , M ] .
Therefore, as n, d i → ∞ and n/d i → γ i , by Lemma 8, we have inf
[Equation]
θ ∗ (cid:13)(cid:13)(cid:13) Σ → inf λ ∈ [0 ,M ] h ( λ, γ 1 , . . .
, γ m ) .
Recalling ˆ θ Bayes ( X n, y n ) =
arg min θ (cid:16) d (cid:107) θ (cid:107)
+ (cid:107) X n θ − y n (cid:107) 22 σ(cid:17)
=
ˆ θ σ 2 dn ,n and
R n = E θ ∗ ∼N (0 , 1 d I d ) ,X n , y n (cid:20)(cid:13)(cid:13)(cid:13)
ˆ θ Bayes ( X n , y n ) − θ ∗ (cid:13)(cid:13)(cid:13) 2 Σ (cid:21) =
inf
λ ≥
E θ ∗ ∼N (0 , 1 d I d ) ,X n , y n (cid:13)(cid:13)(cid:13)
ˆ θ λ,n,d − θ ∗ (cid:13)(cid:13)(cid:13) Σ .
For all M > C M := 2 σ 2 (cid:80) i ∈ [ m ] 1 γ i ≥ σ 2 d n (recall d n → (cid:80) i ∈
[ m ] 1 γ i ), we have inf λ ∈ [0 ,M ] E θ ∗ ∼N (0 , 1 d I d ) ,X n , y n (cid:13)(cid:13)(cid:13)
ˆ θ λ,n,d − θ ∗ (cid:13)(cid:13)(cid:13) Σ = inf
λ ≥
E θ ∗ ∼N (0 , 1 d I d ) E X n , y n (cid:13)(cid:13)(cid:13)
[Table]
The uniqueness of limits implies that inf λ ∈ [0 ,M ] h ( λ, γ 1 , . .
, γ m ) is independent of M as long as M > σ 2 .
As a result, if M > C M , we have inf λ ∈ [0 ,M ] h ( λ, γ 1 , . . .
, γ m ) =
inf λ ≥ 0 h ( λ, γ 1 , . .
γ m ) , which yields
[Equation]
Equation (50) implies inf λ ≥ 0 h ( λ, γ 1 , . .
, γ m ) is decreasing in every γ i .
Combining Equation (51) and Equation (52) gives
[Equation]