P ROVABLE M ORE D ATA H URT IN H IGH D IMENSIONAL L EAST S QUARES E STIMATOR
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper investigates the ﬁnite-sample prediction risk of the high-dimensional least squares estimator.
We derive the central limit theorem for the prediction risk when both the sample size and the number of features tend to inﬁnity.
Furthermore, the ﬁnite-sample distribution and the conﬁdence interval of the prediction risk are provided.
Our theoretical results demonstrate the sample-wise nonmonotonicity of the prediction risk and conﬁrm “more data hurt” phenomenon.
I NTRODUCTION
More data hurt refers to the phenomenon that training on more data can hurt the prediction performance of the learned model, especially for some deep learning tasks.
Loog et al.
(2019) shows that various standard learners can lead to sample-wise non-monotonicity.
Nakkiran et al.
(2019) experimentally conﬁrms the sample-wise non-monotonicity of the test accuracy on deep neural networks.
This challenges the conventional understanding in large sample properties: if an estimator is consistent, more data makes the estimator more stable and improves its ﬁnite-sample performance.
Nakkiran (2019) considers adding one single data point to a linear regression task and analyzes its marginal effect to the test risk.
Derezi ´ nski et al.
(2019) gives an exact non-asymptotic risk of the high-dimensional least squares estimator and observes the sample-wise non-monotonicity on mean square error.
For adversarially robust models, Min et al.
(2020) proves that more data may increase the gap between the generalization error of adversarially-trained models and standard models.
Chen et al.
(2020) shows that more training data causes the generalization error to increase in the strong adversary regime.
In this work, we derive the ﬁnite-sample distribution of the prediction risk under linear models and prove the “more data hurt” phenomenon from an asymptotic point of view.
Intuitively, the “more data hurt” stems from the “double descent” risk curve: as the model complexity increases, the prediction risk of the learned model ﬁrst decreases and then increases, and then decreases again.
The double descent phenomenon can be precisely quantiﬁed for certain simple models (Hastie et al.
; Mei & Montanari (2019); Ba et al.
; Belkin et al.
Bartlett et al.
Xing et al.
Among these works, Hastie et al.
and Mei & Montanari (2019) use the tools from random matrix theory and explicitly prove the double descent curve of the asymptotic risk of linear regression and random features regression in high dimensional setup.
Ba et al.
(2019) gives the asymptotic risk of two-layer neural networks when either the ﬁrst or the second layer is trained using a gradient ﬂow.
The second decline of the prediction risk in the double descent curve is highly related to the more data hurt phenomenon.
In the over-parameterized regime when the model complexity is ﬁxed while the sample size increases, the degree of over-parameterization decreases and becomes close to the interpolation boundary (for example p/n = 1 in Hastie et al.
(2019)), in which a high prediction risk is achieved.
However, the existing asymptotic results, which focus on the ﬁrst-order limit of the prediction risk, cannot fully describe the more data hurt phenomenon.
In fact, the “double descent” curve is a function of the limiting ratio lim p/n , which may not be able to characterize the empirical prediction risk in ﬁnite sample situations.
There will be a non-negligible discrepancy between the empirical prediction risk and its limit, especially when the sample size or dimension is small.
Finegrained second-order results are thus needed to fully characterize such discrepancy and further, a conﬁdence band for the prediction risk can be constructed to evaluate its ﬁnite sample performance.
We take Figure 1 as an example to illustrate this.
According to the ﬁrst-order limit, given a ﬁxed dimension p = 100 , the prediction risks at sample size n = 90 and n = 98 are about 10 . 20 and
More data hurt seems true.
However, the 95% conﬁdence interval of the prediction risks with sample size 98 is [4 . 91 , 142 . 12] , which contains the risk for n = 90 .
Then more data hurt is not statistically signiﬁcant.
Hence, in this work, we characterize the second-order ﬂuctuations of the prediction risk and make attempts to ﬁll this gap.
We employ the linear regression task in Hastie et al.
(2019) and Nakkiran (2019), and introduce new tools from the random matrix theory, e.g. the central limit theorems for linear spectral statistics in Bai & Silverstein (2004); Bai et al.
(2007), to derive the central limit theorem of the prediction risk.
Consider a linear regression task with n data points and p features, the setup of the more data hurt is similar to that in the classical asymptotic analysis in Van der Vaart (2000).
According to the classical asymptotic analysis with p ﬁxed and n → ∞ , the least square estimator is unbiased and √ n -consistent to the ground truth.
This implies that the more data will not hurt and even improve the prediction performance.
However, the story is very different in the over-parameterized regime.
The prediction risk doesn’t decrease monotonously with n when p > n .
More data does hurt in the over-parameterized case.
In the following, we will justify this phenomenon by developing the second-order asymptotic results as both n and p tend to inﬁnity.
We assume p/n → c , and denote 0 < n 1 < n 2 < + ∞ , c 1 = p/n 1 and c 2 = p/n 2 .
Then the direct comparison of the prediction risk between sample sizes n 1 and n 2 can be decomposed into three parts: (i) the gap between the ﬁnitesample risk under n = n 1 and the asymptotic risk with c = c 1 ; (ii) the gap between the ﬁnite-sample risk under n = n 2 and the asymptotic risk with c = c 2 ; (iii) the comparison between two asymptotic risks under c = c 1 and c = c 2 .
Theorem 1 and 2 of Hastie et al.
(2019) give answers to the task (iii).
For (i) and (ii), we develop in this paper the convergence rate and the limiting distribution of the prediction risk as n, p → + ∞ , p/n → c .
Furthermore, the conﬁdence interval of the ﬁnite-sample risk can be obtained as well.
The main goal of this paper is to study the second order asymptotic behavior of two different types of conditional prediction risk in the linear regression model.
One is R X , β ( ˆ β , β ) given both the training data and regression while the other is R X ( ˆ β β ) given the training data only
We summarize our main results as follows: (1) The regression coefﬁcient is set to be either random or nonrandom to cover more cases.
Different convergence rates and limiting distributions of both prediction risk are derived under various scenarios.
(2) In particular, the ﬁnite-sample distribution of the conditional prediction risk given both the training data and regression coefﬁcient is derived and the sample-wise double descent is characterized in Theorem 4.2 and Theorem 4.5 (see Figure 1).
Under certain assumptions, the more data hurt phenomenon can be conﬁrmed by comparing the conﬁdence intervals built via the central limit theorems.
(3) Our results incorporate non-Gaussian observations.
For Gaussian data, the limiting mean and variance in the central limit theorems have simpler forms, see Section 4.2 and 4.3 for more details.
The rest of this paper is organized as follows.
Section 3 introduces the model settings and two different prediction risk.
Section 4 presents the main results on CLTs for the two types of risk with discussion.
Section 5 conducts simulation experiments to verify the main results.
All the technical proofs and lemmas are relegated to the appendix in the supplementary ﬁle.
R ELATED WORK
Double Descent
The double descent curve describes how generalization ability changes as model capacity increases.
It subsumes the classical bias-variance trade-off, a U-shape curve, and further show that the test error exhibits a second drop when the model capacity exceeds the interpolation threshold (Belkin et al.
; Geiger et al.
; Spigler et al.
; Advani & Saxe (2017)).
The double descent phenomenon has been quantiﬁed for certain models, including two-layer neural networks via non-asymptotic bounds or asymptotic risk (Belkin et al.
; Muthukumar et al.
; Hastie et al.
; Mei & Montanari (2019); Ba et al.
As our results are based on the linear regression, we mainly focus on the literature of linear models.
Muthukumar et al.
(2020) and Bartlett et al.
(2020) derive the generalization bounds for over-parametrized linear models and show the beneﬁts of the interpolation.
Hastie et al.
(2019) gives the ﬁrst-order limit of the generalization error for linear regressions as n, p → + ∞ .
Derezi´nski et al.
(2019) provides an exact non-asymptotic expression for double descent of the high-dimensional least square estimator.
Wu & Xu (2020) extends the ﬁrst-order limit of the prediction error of the generalized weighted ridge estimator to more general case with anisotropic features and signals.
Montanari et al.
, Deng et al.
and Kini & Thrampoulidis (2020) investigate the sharp asymptotic of binary classiﬁcation tasks with the max-margin solution and the maximum likelihood solution.
Emami et al.
(2020) and Gerbelot et al.
(2020a) consider the double descent in generalized linear models.
Furthermore, the double descent phenomenon is also observed on linear tasks with various problems and assumptions, e.g. LeJeune et al.
(2020); Gerbelot et al.
(2020b)
; Javanmard et al.
; Dar & Baraniuk (2020); Xu & Hsu (2019); Dar et al.
Xing et al.
(2019) sharply quantiﬁes the beneﬁt of interpolation in the nearest neighbors algorithm.
Mei & Montanari (2019) derives the limiting risk on the random features model and shows that minimum generalization error is achieved by highly overparametrized interpolators.
Ba et al.
(2019) gives the limiting risk of the regression problem under two-layer neural networks.
However, the existing asymptotic results focus on the ﬁrst-order limit of the prediction risk and do not indicate the convergence rate.
There are very few second-order results in the literature, Shen & Bellec (2020) establishes the asymptotic normality for the derivatives of 2-layers neural network, but not the exact limiting distribution of the risk.
In this work, we are the ﬁrst to develop results on second-order ﬂuctuations of the prediction risk in linear regressions and provide its corresponding conﬁdence intervals.
The more data hurt phenomenon is further justiﬁed from the asymptotic point of view.
Random Matrix Theory The primary tool for analyzing the second-order ﬂuctuations of prediction risk comes from random matrix theory.
In particular, Bai & Silverstein (2004) reﬁnes the central limit theorem for linear spectral statistics of large dimensional sample covariance matrix with general population and the population is not necessary to be Gaussian.
Similar central limit theorems are also developed for other random matrix ensembles, see Sinai & Soshnikov (1998); Bai & Yao (2005); Zheng (2012).
Other than the central limit theorem for linear spectral statistics, Bai et al.
and Pan & Zhou (2008) study the asymptotic ﬂuctuation of eigenvectors of sample covariance matrices.
Bai & Yao (2008) considers the ﬂuctuation of quadratic forms.
All these technical tools and results are adopted and fully utilized in this paper, especially those related to Stieltjes transform, which are closely connected to the prediction risk studied in this paper.
P RELIMINARIES 3. P ROBLEM , DATA AND ESTIMATOR
Suppose that the training data { ( x i , y i ) ∈ R p × R , i = 1 , 2 , . .
. , n } is generated independently from the model (ground truth or teacher model):
[Equation]
Here, P x is a distribution on R p such that E ( x i ) = 0 , Cov( x i ) =
Σ , and P (cid:15) is a distribution on R
such that E ( (cid:15) i ) =
0 , Var( (cid:15) i )
=
σ 2 .
In particular, the coordinates of x i are not necessarily independent, that is, Σ is not restricted to be diagonal.
To proceed further, we denote
[Equation]
The minimum (cid:96) 2 norm (min-norm) least squares estimator, of y on X , is deﬁned by
[Equation]
where ( X T X )
+ denotes the Moore-Penrose pseudoinverse of X T X .
3. B IAS , VARIANCE AND RISK
Similar to Hastie et al.
(2019), we deﬁne two different types of out-of-sample prediction risk.
The ﬁrst one is given by
[Equation]
where x 0 ∼ P x is a test point and is independent of the training data, and the notation (cid:107) β (cid:107) 2 Σ stands for β T Σ β .
Here β is assumed to be a random vector independent of x 0 .
In this deﬁnition, the expectation E stands for the conditional expectation for x 0 , ˆ β and β when X is given.
According to the bias-variance decomposition, we have R X ( ˆ β , β ) :
= B X ( ˆ β , β ) + V X ( ˆ β , β ) , where
[Equation]
V X ( ˆ β
[Equation]
β ) =
[Equation]
{ Cov
[Equation]
ˆ β
[Equation]
X )
[Equation] [Equation]
Plugging the model (1) into the min-norm estimator (2), the bias and variance terms can be rewritten as
[Equation]
where ˆ Σ = X T X /n is the (uncentered) sample covariance matrix of X , and
Π = I p − ˆ Σ + ˆ Σ is the projection onto the null space of X .
The second type of out-of-sample prediction risk is deﬁned as
[Equation]
where
B X ,
[Equation]
In this deﬁnition, the parameter β is assumed to be given.
The expectation E is the conditional expectation for x 0 and ˆ β when X and β are given.
This is consistent with the commonly-used testing procedure, in which a trained model is evaluated by the average loss on unseen testing data.
M AIN R ESULTS
Before stating our main results, we brieﬂy highlight the challenges we faced in proving the more data hurt phenomenon.
First, the ﬁnite-sample behaviors of the prediction risk is required.
Hastie et al.
(2019) gives the ﬁrst-order limits of both R X , β
( ˆ β , β ) and R X ( ˆ β , β ) as n, p → + ∞ and p/n → c ∈ (0 , + ∞ ) .
However, to prove the more data hurt phenomenon, we should ﬁx p and investigate the ﬁnite-sample risk with sample size n varies.
This implies that only knowing the ﬁrstorder limit is not enough, the convergence rate is also needed.
To solve this problem, we have derived the central limit theorems for both R X , β ( ˆ β , β ) and R X ( ˆ β , β ) , respectively, which characterize the second-order ﬂuctuations of the risk.
Then we can ﬁgure out the ﬁnite-sample behavior of the risk by computing the gap between the risk and its limit.
The conﬁdence intervals of the risk can be further obtained.
Second, the parameter β also contributes randomness to the ﬁnite-sample risk, which further inﬂuences the convergence rate.
To analyze the contribution of β , we need to make use of the technical tools and asymptotic results for eigenvectors and quadratic forms developed in
Bai et al.
and Bai & Yao (2008).
Another interesting ﬁnding is that, in the over-parameterized regime such that p > n , the two types of out-of-sample prediction risk R X , β ( ˆ β , β ) and R X ( ˆ β , β ) enjoy different convergence rates.
4. A SSUMPTIONS AND MORE NOTATIONS
As follows are some notations used in this paper.
The p × p identity matrix is denoted by I p .
For a symmetric matrix A ∈ R p × p , we deﬁne its empirical spectral distribution as
[Equation]
where 1 {·} is the indicator function and λ i ( A ) , i = 1 , 2 , . . .
p are the eigenvalues of A .
The notation d −→ stands for the convergence in distribution.
Z α/ 2 is the α/ 2 upper quantile of the standard normal distribution, λ max ( A ) and λ min ( A ) denote the largest and smallest eigenvalues of A , respectively.
Here we list all the assumptions for X and β needed under different scenarios:
(A) x j ∼ P x is of the form x j = Σ 1 / 2 z j , where z j is a p -length random vector with i.i.d.
entries that have zero mean, unit variance, and a ﬁnite 4 -th order moment E ( z 4 ij ) =
ν 4 , i = 1 , · · · , p , j = 1 , · · · , n. (B1) Σ is a deterministic positive deﬁnite matrix, such that 0 < c 0 ≤ λ min ( Σ ) ≤ λ max ( Σ ) ≤ c 1 , for all n , p and some constants c 0 , c 1 .
As p → ∞ , we assume that the empirical spectral distribution F Σ converges weakly to a probability measure H. (B2) Σ is an identity matrix, Σ
= I p .
(C1) β is a nonrandom constant vector, and (cid:107) β (cid:107)
=
β T β =
r 2 .
(C2)
β ∼ P β is independent of X and follows multivariate Gaussian distribution N p ( 0 , r 2 p I p ) .
Throughout this paper, we consider the limiting distributions and the convergence rates of the outof- sample prediction risk when n, p → ∞ such that p/n = c n → c ∈ (0 , ∞ ) .
If c > 1 , the sample size n is smaller than the number of parameters p , we call this case “over-parametrized” .
Otherwise when c < 1 , we call it “under-parameterized” .
4. U NDER - PARAMETRIZED ASYMPTOTICS
In this section, we focus on the risk of the min-norm estimator (2) in the under-parametrized regime.
According to Theorem 1 of Hastie et al.
(2019), both B X , β ( ˆ β , β ) and B X ( ˆ β , β ) converge to σ 2 c/ (1 − c ) almost surely.
The following Theorem 4.1 and 4.2 show that both B X ( ˆ β , β ) and
B X , β ( ˆ β , β ) converge to σ 2 c/ (1 − c ) at the rate 1 /p.
Furthermore, the limiting distributions are derived by making use of the CLT for linear spectral statistics of large-dimensional sample covariance matrices.
Theorem 4.1.
Suppose that the training data is generated from the model (1) , and the assumptions (A) and (B1) hold.
Then the ﬁrst type of out-of-sample prediction risk R X ( ˆ β , β ) of the min-norm estimator (2) satisﬁes that, as n, p → ∞ such that p/n = c n → c < 1 ,
[Equation]
where
[Equation]
Conclusively,
[Equation]
where 1 − α is the conﬁdence level and
[Equation]
Under the assumptions of Theorem 4.1, we know that Π
= I
p − ˆ Σ + ˆ Σ = 0 and
[Equation]
Thus R X ( ˆ β , β ) equals to R X , β
( ˆ β , β ) and the two risk share the same asymptotic limit.
Theorem 4.2.
Under the assumptions of Theorem 4.1, the second type of out-of-sample prediction risk R X , β ( ˆ β , β ) of the min-norm estimator (2) satisﬁes that, as n, p → ∞ such that p/n = c n → c < 1 ,
[Equation]
and
[Equation]
where µ c , σ 2 c , L α,c and U α,c are the same as those in Theorem 4.1.
4. O VER - PARAMETRIZED ASYMPTOTICS
In this section, we consider the min-norm estimator (2) in the over-parametrized case c > 1 .
The bias term, either B X ( ˆ β , β ) or B X , β ( ˆ β , β ) , is generally nonzero.
According to Lemma 2 in Hastie et al.
(2019), both B X ( ˆ β , β ) and B X , β ( ˆ β , β ) converge to r 2 (1 − 1 /c ) as n, p → + ∞ and p/n → c > 1 .
This implies that the bias term can inﬂuence the asymptotic behavior of the prediction risk, including the convergence rate.
Hence to derive the CLT of the out-of-sample prediction risk, we need to consider both the bias and variance terms in (4).
In the following, we investigate the asymptotic properties of the two prediction risk R X ( ˆ β , β ) and
R X , β ( ˆ β , β ) under various combinations of the assumptions (A1), (B2) for X and scenarios (C1), (C2) for both random and nonrandom β .
We start with the case when β is a constant vector.
Theorem 4.3.
Suppose that the training data is generated from the model (1), and the assumptions (A) , (B2) and (C1) hold.
Then the ﬁrst type of out-of-sample prediction risk R X ( ˆ β , β ) of the minnorm estimator (2) satisﬁes that, as n, p → ∞ such that p/n = c n → c > 1 ,
[Equation]
where µ c, 1 = 0 and σ 2 c, 1 = 2( c − 1) c 2 r 4 .
A more practical version is to replace µ c, 1 and σ 2 c, 1 with ˜ µ c, 1 = 1 √ p (cid:110) [Equation] and
[Equation]
Conclusively,
[Equation]
where 1 − α is the conﬁdence level and
[Equation]
Remark 4.1.
Under assumption (C1), B X ( ˆ β , β ) =
B X , β ( ˆ β , β ) and R X ( ˆ β , β ) =
R X , β ( ˆ β , β ) .
Thus Theorem 4.3 still holds if we replace R X ( ˆ β , β ) with R X , β
( ˆ β , β ) .
Remark 4.2.
Under Assumption (B2), the eigenvector of ˆ Σ is asymptotically Haar distributed.
Therefore, the bias term B X ( ˆ β , β ) is only related to the length of β .
However, in the anisotropic settings with general Σ , the eigenvector of the ˆ Σ is no longer asymptotically Haar distributed.
The limiting behavior of B X ( ˆ β , β ) heavily relies on the interaction between β and the eigenvectors of ˆ Σ . Therefore, we conjecture that there is no universal convergence rate for the bias term B X ( ˆ β , β ) that can cover arbitrary non-random β and anisotropic Σ in the over-parametrized case, not to mention the prediction risk R X ( ˆ β , β ) .
A small simulation experiment is conducted in Appendix G to conﬁrm our conjecture on this point.
Next we consider the case when β is a random vector that follows assumption (C2).
Theorem 4.4.
Suppose that the training data is generated from the model (1) , and the assumptions (A) , (B2) and (C2) hold.
Then as n, p → ∞ such that p/n = c n → c > 1 , the ﬁrst type of out-of-sample prediction risk R X ( ˆ β , β ) of the min-norm estimator (2) satisﬁes,
[Equation]
where
[Equation]
Hence we have
[Equation]
where
L
[Equation]
As for R X , β ( ˆ β , β ) , we have the following theorem.
Theorem 4.5.
Suppose that the training data is generated from the model (1) , and the assumptions (A) , (B2) and (C2) hold.
Then, as n, p → ∞ such that p/n = c n → c > 1 , the second type of out-of-sample prediction risk R X , β
( ˆ β , β ) of the min-norm estimator (2) satisﬁes,
[Equation]
where µ c, 3 = 0 and σ 2 c, 3 = 2(1 − 1 c ) r 4 .
A more practical version is to replace µ c, 3 and σ 2 c, 3 with ˜ µ c, 3 = 1 √ p [Equation] and
[Equation]
The corresponding (1 − α ) -conﬁdence interval is given by
[Equation] [Equation]
with
L α,c
[Equation]
Remark 4.3.
Note that besides the leading constants in ( µ c, 3 , σ c, 3 ) , the version (˜ µ c, 3 , ˜ σ c, 3 ) also contains smaller order terms, including terms of order O (1 / √ p ) in ˜ µ c, 3 and terms of order O (1 /p ) in ˜ σ c, 3 .
These smaller order terms will vanish when p and n grow very large, but for ﬁnite sample situations, these smaller order terms will provide a ﬁner approximation for the ﬁnite sample distribution of R X , β ( ˆ β , β ) .
As shown in the following experiments, these terms have indeed made non-negligible contributions to ﬁtting the empirical distribution of R X , β
( ˆ β , β ) , which sheds new lights for practitioners.
Remark 4.4.
If we compare the results in Theorem 4.3 and 4.5, we will ﬁnd out that R X ( ˆ β , β ) with constant β and R X , β ( ˆ β , β ) with random β share the same ﬁrst-order limit and second-order error rate O ( p − 1 / 2 ) .
This is quite intuitive because both risk treat β as a constant.
Their differences are reﬂected in their limiting variances.
Nevertheless, it’s very interesting to observe from Theorem 4.
that, R X ( ˆ β , β ) with random β under the over-parametrized case has a smaller second-order error rate O ( p − 1 ) .
It enjoys the same rate as the under-parametrized case in Theorem 4.1.
A possible explanation would be that averaging over the randomness in β can partially offset the curse of dimensionality so that R X ( ˆ β , β ) achieves the same error rate for all p, n combinations.
Remark 4.5.
It’s worth mentioning that the only assumption regarding data distribution is assumption (A), where only ﬁnite fourth order moment is required.
Non-Gaussianity allows our theoretical results more widely applied.
4. D ISCUSSION
In this section, we ﬁrst make a short conclusion of what we have done theoretically in this paper and further discuss some possible directions of extension.
We have systematically investigated the second-order ﬂuctuations of two types of prediction risk,
R X ( ˆ β , β ) and R X , β ( ˆ β , β ) , for the high dimensional least square estimators ˆ β .
Theorem 4.1 and 4.4 are for R X ( ˆ β , β ) while Theorem 4.2 and 4.5 are for R X , β
( ˆ β , β ) .
Both ﬁxed effect and random effect of the regression coefﬁcients β are discussed following the settings in Hastie et al.
R X ( ˆ β , β ) and R X , β
( ˆ β , β ) are the same when β is nonrandom as established in Theorem 4.3.
Asymptotic results are categorized into the under-parametrized case ( p < n ) and the over-parametrized case ( p > n ).
Although the ﬁrst-order limits of the prediction risk in high dimensional linear models have already been well studied in recent years, including general extensions to anisotropic features and signals in
Wu & Xu (2020), the “double descent” risk curve is just a function of the limiting ratio lim p/n .
There is still a non-negligible discrepancy between the ﬁnite-sample prediction risk and its ﬁrst-order limit on the “double descent” curve.
How large is this discrepancy?
How fast does the risk converge to its limit?
Our CLTs provide answers to such questions and give a ﬁne-grained characterization of the second-order ﬂuctuations of the prediction risks.
Not only explicit forms of the leading constants in the limiting means and variances are shown in the main theorems, smaller order terms are also derived to improve the empirical performance for practitioners.
It is also important to recognize the limitations of our results.
First, the present paper only concerns linear regression tasks since the linear regression task is simple but important as well.
Some recent works linearize neural networks at the initialization and employ Neural Tangent Kernels (Jacot et al.
to approximate the training procedure of a strongly over-parametrized neural network by solving a linear regression task, e.g. Du et al.
Arora et al.
; Lee et al.
Though the setting considered in this paper is simple and limited, the problem has not however been fully understood so far in the literature.
Therefore, we are among the ﬁrst to take the task and develop the second-order ﬂuctuation results for the prediction risk.
Second, we assume general covariance Σ and non-Gaussianity for the under-parametrized case, which ﬁts the most updated and realistic settings in the literature, however we only investigate the isotropic settings while still allow for non-Gaussianity under over-parametrization.
We haven’t extended it to the more general anisotropic settings yet.
The reasons are two-fold.
On the one hand, according to Wu & Xu (2020), the ﬁrst-order limits depend on the Stieltjes transforms of the unknown spectral distribution of Σ .
Since Σ is unknown, we cannot obtain any explicit characterization of the ﬁrst-order limits, not to mention the second-order ﬂuctuations.
The CLTs would only be written as certain complicated implicit functions of Σ and would be too abstract to evaluate practically.
More restrictions would be imposed on Σ to guarantee the second-order convergence.
On the other hand, from the technical perspective, the techniques required for anisotropic over-parametrized cases are very different from the isotropic cases due to difference in the bias-variance decomposition in (4).
The tools in random matrix theory have not been fully developed yet for anisotropic cases.
Since we have considered various scenarios in this paper, including random and nonrandom signals β for both conditional and unconditional risks, it will take great efforts and continuous work to extend all of them to the most general settings, which would lead to many subsequent works in the ﬁeld of machine learning and random matrix theory literature.
E XPERIMENTS
In this section, we carry out simulation experiments to examine the central limit theorems and the corresponding conﬁdence intervals in Theorem 4.2 and Theorem 4.5.
We generate data points from the linear model (1) and directly compute the prediction risk via the bias-variance decomposition in (4).
The generative distribution P x is taken to be the standard normal distribution.
The noise distribution P (cid:15) is taken to be N (0 , 1) .
In the following, we present the gap between the ﬁnite-sample distribution of the prediction risk and the corresponding limiting distribution to check the central limit theorems and use the cover rate to measure the effectiveness of the conﬁdence intervals.
More simulation results, including cases with non-Gaussian distributions for P x and P (cid:15) are relegated to the Appendix due to space limitations.
Example 1 .
This example examines the results in Theorem 4.2.
We deﬁne a statistic
[Equation]
According to Theorem 4.2, T n weakly converges to the standard normal distribution as n, p → ∞ .
In this example, c = 1 / 2 and p = 50 , 100 , 200 .
The ﬁnite-sample distribution of T n is presented by the histogram of T n in Figure 2 with 1000 repetitions, where the solid blue curve stands for standard normal density function.
It can be seen that the ﬁnite-sample distribution of T n is very consistent with the standard normal distribution, especially when n, p become larger.
When α = 0 .
05 , the empirical cover rates of the 95% -conﬁdence interval are 94 .
5% and 95 .
3% for p = 50 ,and 200 , respectively.
All these experiments verify the correctness of our theoretical results.
Example 2 .
This example veriﬁes the results in Theorem 4.5.
Here we deﬁne two statistics:
[Equation]
According to Theorem 4.5, both T n, 0 and T n, 1 weakly converge to the standard normal distribution as n, p → + ∞ .
Compared to T n, 0 , T n, 1 provides a better approximation for the ﬁnite sample distribution of R X , β ( ˆ β , β ) because it contains smaller order terms in the asymptotic mean and variance.
We take c = 3 / 2 and p = 150 , 300 , 450 .
Similarly the ﬁnite-sample distributions of T n, 0 and T n,are presented by the histogram of T n, 0 and T n, 1 with 1000 repetitions.
The comparison between these two statistics is shown in Figure 3.
It can also be seen that the ﬁnite sample distributions of
T n, 0 and T n, 1 both match the standard normal distribution quite well, especially T n, 1 with more precise characterization.
The empirical cover rates of the 95% -conﬁdence interval (11) are 93 .
7% and 94 .
4% for p = 150 , 300 and 600 respectively, which further shows the validity of our theoretical results.
A P ROOF OF THEOREM 4.1 AND THEOREM 4.
Let X =
ZΣ 1 / 2 .
According to the Bai-Yin theorem (Bai & Yin (2008)), the smallest eigenvalue of
Z T Z /n is almost surely larger than (1 − √ c ) 2 / 2 for sufﬁciently large n.
Thus
[Equation]
which implies that the matrix X T X /n is almost surely invertible for large n. By Section 3.2, Π = 0 ,
B X ( ˆ β , β ) =
B X , β ( ˆ β , β ) = 0 and V X ( ˆ β , β ) =
V X , β ( ˆ β , β ) .
Thus the CLT of R X ( ˆ β , β ) is same to that of R X , β ( ˆ β , β ) .
For simplicity, we focus on R X ( ˆ β , β ) in the following.
Notice that
[Equation]
where F Z is the spectral measure of Z T Z /n.
According to Theorem 1 of Hastie et al.
(2019), as n, p → ∞ such that p/n = c n → c ∈ (0 , ∞ ) , F Z ( x ) weakly converges to the standard MarcenkoPastur law F c ( x ) and
[Equation]
Here the standard Marcenko-Pastur law F c ( x ) has a density function
[Equation]
where a = (1 −√ c ) 2 , b
= (1+ √ c ) 2 and p c ( x ) has a point mass 1 − 1 c at the origin if c > 1 .
Hence
[Equation]
c n ( s ) (cid:1) .
According to Theorem 1.1 of Bai & Silverstein (2004),
[Equation]
where
[Equation]
Here the contours in (13) and (14) are closed and taken in the positive direction in the complex plane, enclosing the support of F Z , i.e. [(1 − √ c ) 2 , (1 + √ c ) 2 ] .
The Stieltjes transform m ( z ) satisﬁes the equation
[Equation]
To further simplify the integrations in µ c and σ c , let z = 1 + √ c
( rξ + 1 rξ )
+ c and perform change of variables, then we have
[Equation]
and when ξ moves along the unit circle | ξ | = 1 on the complex plane, z will orbit around the center point 1 + c along an ellipse which enclosing the support of F Z .
Thus
[Equation]
As for σ 2 c
[Equation]
note that
[Equation]
therefore
[Equation]
Meanwhile,
[Equation]
hence
[Equation]
and
[Equation]
Let
[Equation]
According to (12)
[Equation]
we have
[Equation]
where
[Equation]
B P ROOF OF THEOREM 4.
Notice that
[Equation]
Since β is a constant vector, we can make use of the results in Theorem 3 in Bai et al.
(2007) and
Theorem 1.3 in Pan & Zhou (2008) regarding eigenvectors.
Their works investigate the sample covariance matrix A p =
T 1 / 2 p
[Equation]
/n , where T p is
an p × p nonnegative deﬁnite Hermitian matrix with a square root T 1 / 2 p and X p is an n × p matrix with i.i.d.
entries ( x ij ) n × p .
Let U p Λ p U T p denote the spectral decomposition of A p where Λ p = diag ( λ 1 , · · · , λ p ) and U p is a unitary matrix consisting of the orthonormal eigenvectors of A p .
Assume that x p is an arbitrary nonrandom unit vector and y
=
( y 1 , y 2 , · · · , y p ) T =
U T p x p , two empirical distribution functions based on eigenvectors and eigenvalues are deﬁned as
[Equation]
Then for a bounded continuous function g ( x ) , we have
[Equation]
The results in Bai et al.
and Pan & Zhou (2008) show that
Lemma B.1.
(Theorem 3 Bai et al.
and Theorem 1.3 Pan & Zhou (2008))
Suppose that
(1) x ij ’s are i.i
d. satisfying E ( x ij ) = 0 , E ( | x ij | 2 ) = 1 and E ( | x ij | 4 ) < ∞ ; (2) x p ∈ C p , (cid:107) x p (cid:107)
= 1 , lim n,p →∞ p/n = c ∈ (0 , ∞ ) ; (3) T p is nonrandom Hermitian non-negative deﬁnite with with its spectral norm bounded in p , with H p =
F T p d −→ H a proper distribution function and x T p ( T p −
z
I p ) − 1 x p → m F H ( z ) , where
m F H ( z ) denotes the Stieltjes
transform of H ( t ) ; (4) g 1 , · · · , g k are analytic functions on an open region of the complex plain which contains the real interval (cid:104) lim
[Equation]
p
[Equation]
(5) as n, p → ∞ ,
[Equation] [Equation]
Deﬁne G p ( x ) = √ n ( F A p 1 ( x ) − F A p ( x )) , then the random vectors
[Equation]
forms a tight sequence and converges weakly to a Gaussian vector x g 1 , · · · , x g k with mean zero and covariance function
[Equation]
The contours C 1 , C 2 are disjoint, both contained in the analytic region for the functions ( g 1 , · · · , g k ) and enclose the support of F c n ,H p for all large p .
(6) If H ( x ) satisﬁes
[Equation]
then the covariance function can be further simpliﬁed to
[Equation]
Recall that B X ( ˆ β , β ) =
lim
z → 0 + z β T ( ˆ Σ + z I p ) − 1 β .
Let g ( x ) =
1 / ( x + z ) and x p = β /r .
Then we have
[Equation]
where F c n ( x ) is the standard Marcenko-Pastur law.
It is not difﬁcult to check that under Assumptions (A1), (B1) and (C1), all the conditions (1)-(6) in Lemma B.1 are satisﬁed.
To proceed further, denote a = (1 − √ c ) 2 , b
= (1+ √ c ) 2 .
If c is replaced by c n , a and b are denoted by a n and b n respectively.
By some algebraic calculations, we have
[Equation]
and
[Equation]
Therefore, lim
[Equation]
and lim
[Equation]
Furthermore, as n, p → ∞ , p/n = c n → c > 1 ,
[Equation]
This can be rewritten as
[Equation]
Next we deal with the variance term V X ( ˆ β , β ) .
According to the Assumption (B1) , the variance term is
[Equation]
where s i , i = 1 , . .
. , n are the nonzero eigenvalues of X T X /n.
Let { t i , i = 1 , . . .
n } denote the non-zero eigenvalues of XX T /p , then we have
[Equation]
By interchanging the role of p and n , from the result in Theorem 4.1, as n, p → ∞ , p/n = c n → c > 1 , we have, n
[Equation]
where c (cid:48) n = n/p = 1 /c n , c (cid:48)
= 1 /c .
This result can be rewritten as n (cid:88)
[Equation]
Hence the CLT of V X ( ˆ β , β ) is given by
[Equation]
Notice that Cov (cid:16) B X ( ˆ β , β ) , V X ( ˆ β , β ) (cid:17) = 0 .
According to the consistency rate and the limiting distribution of B X ( ˆ β , β ) and V X ( ˆ β , β ) , we know that the bias B X ( ˆ β , β ) is the leading term of
R X ( ˆ β , β ) .
This implies that
[Equation]
where σ 2 c, 1 = 2( c − 1) r 4 /c 2 .
A practical version of this CLT is given by
[Equation]
where
[Equation]
C P ROOF OF THEOREM 4.
First we consider the bias term B X ( ˆ β , β ) .
By Assumption (A1), (B1), and (C2),
[Equation]
Alternatively, we can rewrite the bias as
[Equation]
Deﬁne that f n ( z ) =
z r 2 p
Tr( ˆ Σ + z I p ) − 1 .
Notice that | f n ( z ) | and | f (cid:48) n ( z ) | are bounded above.
By the
Arzela-Ascoli theorem
, we deduce that f n ( z ) converges uniformly to its limit.
Under Assumption (C2) , by the Moore-Osgood theorem, almost surely,
[Equation]
In fact,
[Equation]
where m n ( z ) is the Stieltjes transform of empirical spectral distribution of ˆ Σ =
X T X /n.
According to Theorem 2.1 in Zheng et al.
(2015) and Lemma 1.1 in Bai & Silverstein (2004), the truncated version of p ( m n ( z )
− m ( z )) converges weakly to a two-dimensional Gaussian process M ( · ) satisfying
[Equation]
and
[Equation]
where m = m ( z ) represents the Stieltjes transform of limiting spectral distribution of companion matrix XX T /n satisfying the equation
[Equation]
When p > n , we can actually solve m ( z ) equation and obtain that
[Equation]
Therefore, by some algebraic calculations, we have
[Equation]
Moreover,
[Equation]
By substituting of the explicit form of m ( z )
[Equation]
we can easily derive that
[Equation]
which means that the second-order limit of B X ( ˆ β , β ) is still r 2 (1 − 1 /c ) .
All in all, B X ( ˆ β , β ) is identical with a constant r 2 (1 − 1 /c ) in distribution
[Equation]
On the other hand, by Assumption (B1),
[Equation]
where s i , i = 1 , . .
. , n are the nonzero eigenvalues of X T X /n.
Similar to the proof of Theorem 4.3, the CLT of V X ( ˆ β , β ) is given by
[Equation]
Combining the results of B X ( ˆ β , β ) and V X ( ˆ β , β ) , we have
[Equation]
where
[Equation]
D P ROOF OF THEOREM 4.
Note that under Assumption (B1) and (C2) , B X , β ( ˆ β , β ) =
β T Π β = β T ( I p − ˆ Σ + ˆ Σ ) β .
If we directly consider β T ( I p − ˆ Σ + ˆ Σ ) β , we can make use of the asymptotic results for quadratic forms
Theorem 7.2 in Bai & Yao (2008) stated as follows.
Lemma D.1.
(Theorem 7.2 in Bai & Yao (2008)) Let { A n = [ a ij ( n )] } be a sequence of n × n real symmetric matrices, { x i } i ∈ N be a sequence of i.i.d.
K dimensional real random vectors, with
E ( x i ) = 0 , E ( x i x T i ) =
( γ ij ) K × K and E [ (cid:107) x i (cid:107) 4 ] < ∞ .
Denote x i =
( x (cid:96)i ) K × 1 ,
X ( (cid:96)
[Equation]
assume the following limits exist
[Equation]
Then the K -dimensional random vectors
[Equation]
converge weakly to a zero-mean Gaussian vector with covariance matrix D
= D 1 + D 2 where
[Equation]
According to the results in Lemma D.1, let A n = Π
= I p − ˆ Σ + ˆ Σ , then we have, as p → ∞ ,
[Equation]
where
[Equation]
and
[Equation]
Since in the proof of Theorem 4.4, we have already shown that
[Equation]
In particular, if β follows multivariate Gaussian distribution, i
[Equation]
e. β ∼ N p (0 , r 2 p I p ) , then as p → ∞ ,
[Equation]
Moreover, V X , β ( ˆ β , β ) =
V X ( ˆ β , β ) , we have already proved in Theorem 4.4 that
[Equation]
( c − 1)(cid:17) .
Note that Cov( B X , β ( ˆ β , β ) , V X , β ( ˆ β , β ))
= 0 .
According to the consistency rate of B X , β ( ˆ β , β ) and V X , β ( ˆ β , β ) , we know that the bias B X ( ˆ β , β ) is the leading term of R X , β ( ˆ β , β ) .
This implies that
[Equation]
where σ 2 c, 3 = 2 r 4 (1 − 1 /c )
A practical version of this CLT is given by
[Equation]
where
[Equation]
E M ORE EXPERIMENTS E. M ORE RESULTS OF E XAMPLE
This example checks Theorem 4.2.
We deﬁne a statistic
[Equation]
According to Theorem 4.2, T n weakly converges to the standard normal distribution as n, p → ∞ .
In this example, c = 1 / 2 and p = 50 , 100 , 200 .
To make sure the assumption (A) holds, the generative distribution P x is taken to be the standard normal distribution, the centered gamma with shape 4 .and scale 0 .
5 , and the normalized Student-t distribution with 6 . 0 degree of freedom.
The ﬁnitesample distribution of T n is estimated by the histogram of T n under 1000 repetitions.
The results are presented in Figure 4.
One can ﬁnd that the ﬁnite-sample distribution of T n tends to the standard normal distribution as n, p → + ∞ .
When α = 0 .
05 , the empirical cover rates of the 95% -conﬁdence interval are reported in Figure 5.
[Figure]
E. M ORE RESULTS OF E XAMPLE
The Example 2 checks Theorem 4.5.
Here we consider the standardized statistics:
[Equation]
According to the central limit theorem (10) and its practical version, both T n, 0 and T n, 1 weakly converge to the standard normal distribution as n, p → + ∞ .
We take c = 2 and p = 100 , 200 , 400 .
The ﬁnite-sample distributions of T n, 0 and T n, 1 are estimated by the histogram of T n, 0 and T n,under 1000 repetitions.
The results are presented in Figure 6 and Figure 7.
When α = 0 .
05 , the empirical cover rates of the 95% -conﬁdence interval (11) are reported in Figure 8.
E. E XAMPLE
This example checks Theorem 4.3.
To proceed further, we denote two statistics:
[Equation]
According to the central limit theorem (8) and its practical version, both T n, 2 and T n, 3 weakly converge to the standard normal distribution as n, p → + ∞ .
We take c = 2 and p = 100 , 200 , 400 .
The ﬁnite-sample distributions of T n, 2 and T n, 3 are estimated by the histogram of T n, 2 and T n,under 1000 repetitions.
The results are presented at Figure 9 and Figure 10.
One can see that the ﬁnite-sample distributions of T n, 2 and T n, 3 are close to the standard normal distribution, and the ﬁnite-sample performance of T n, 3 is better than that of T n, 2 .
When α = 0 .
05 , the empirical cover rates of the 95% -conﬁdence interval (9) are reported in Figure 11. [Figure] A [Figure]
G A N ANISOTROPIC EXAMPLE FOR R EMARK 4.
In the over-parameterized case, the bias term B X ( ˆ β , β ) = β T Π Σ Π β is non-zero while the variance term V X ( ˆ β , β ) remains the same as under-parameterized case. Therefore in this section, we conduct a small simulation to examine the ﬂuctuation of the bias B X for both isotropic and anisotropic Σ in the over-parameterized case with non-random β satisfying Assumption (C1).
In particular, in the following we set r = 1 .
We consider both localized and delocalized β such that
Localized case: β 1 = (1 , 0 , · · · , 0) ; 2.
Delocalized case: β 2 = 1 √ p (1 , · · · , 1) ;
and both the isotropic and anisotropic Σ
Identity case: Σ 1 = I p ; 4.
Compound symmetric case: Σ 2 = 0 .
5 I p + 0 .
5 1 p 1 T p .
Then we ﬁx p/n = 2 and let p vary from 10 to 300, we present in Figure 13 the empirical variance of √ p ∗ B X and p ∗ B X under various combinations of Σ and β with 1000 replications.
From the plot on the top left panel in Figure 13, we can see that the variance of √ p ∗ B X for both β 1 and β 2 remain constant as p grows, which indicates that the convergence rate of B X is 1 / √ p under the isotropic case regardless of localized or delocalized β .
As for the anisotropic case on the top right corner, the variance of √ p ∗ B X stabilizes for β 1 , while decays for β 2 , which indicates that convergence rate of B X under ( Σ 2 , β 2 ) and ( Σ 2 , β 1 ) are different.
This simulation result further conﬁrms our conjecture that in the over-parameterized case, there is no universal CLT for the prediction risk
R X
( ˆ β , β ) under the anisotropic setting for non-random β .