M ULTIPLE D ESCENT : D ESIGN Y OUR O WN G ENERALIZATION C URVE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized.
We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled.
Our results highlight the fact that both classical U-shaped generalization curve and the recently observed double descent curve are not intrinsic properties of the model family.
Instead, their emergence is due to the interaction between the properties of the data and the inductive biases of learning algorithms.
I NTRODUCTION
The main goal of machine learning methods is to provide an accurate out-of-sample prediction, known as generalization.
For a ﬁxed family of models, a common way to select a model from this family is through empirical risk minimization, i.e., algorithmically selecting models that minimize the risk on the training dataset.
Given a variably parameterized family of models, the statistical learning theory aims to identify the dependence between model complexity and model performance.
The empirical risk usually decreases monotonically as the model complexity increases, and achieves its minimum when the model is rich enough to interpolate the training data, resulting in zero (or nearzero) training error.
In contrast, the behaviour of the test error as a function of model complexity is far more complicated.
Indeed, in this paper we show how to construct a model family for which the generalization curve can be fully controlled (away from the interpolation threshold) in both under-parameterized and over-parameterized regimes.
Classical statistical learning theory supports a U-shaped curve of generalization versus model complexity (Geman et al., 1992; Hastie et al., 2009).
Under such a framework, the best model is found at the bottom of the U-shaped curve, which corresponds to appropriately balancing under-ﬁtting and over-ﬁtting the training data.
From the view of the bias-variance trade-off, a higher model complexity increases the variance while decreasing the bias.
A good choice of model complexity achieves a relatively low bias while still keeping the variance under control.
On the other hand, a model that interpolates the training data is deemed to over-ﬁt and tends to worsen the generalization performance due to the soaring variance.
Although classical statistical theory suggests a pattern of behavior for the generalization curve up to the interpolation threshold, it does not describe what happens beyond the interpolation threshold, commonly referred to as the over-parameterized regime.
This is the exact regime where many modern machine learning models, especially deep neural networks, achieved remarkable success.
Indeed, neural networks generalize well even when the models are so complex that they have the potential to interpolate all the training data points (Zhang et al., 2017; Belkin et al., 2018b; Ghorbani et al., 2019; Hastie et al., 2019).
Modern practitioners commonly deploy deep neural networks with hundreds of millions or even billions of parameters.
It has become widely accepted that large models achieve performance superior to small models that may be suggested by the classical U-shaped generalization curve (Bengio et al., 2003; Krizhevsky et al., 2012; Szegedy et al., 2015;
He et al., 2016; Huang et al., 2019).
This indicates that the test error decreases again once model complexity grows beyond the interpolation threshold, resulting in the so called double-descent phenomenon described in (Belkin et al., 2018a), which has been broadly supported by empirical evidence (Neyshabur et al., 2015; Neal et al., 2018;
Geiger et al., 2019; 2020) and conﬁrmed empirically on modern neural architectures by Nakkiran et al.
On the theoretical side, this phenomenon has been recently addressed by several works on various model settings.
In particular, Belkin et al.
(2019a) proved the existence of double-descent phenomenon for linear regression with random feature selection and analyzed the random Fourier feature model (Rahimi and Recht, 2008).
Mei and Montanari (2019) also studied the Fourier model and computed the asymptotic test error which captures the double-descent phenomenon.
Bartlett et al.
(2020); Tsigler and Bartlett (2020) analyzed and gave explicit conditions for “benign overﬁtting” in linear and ridge regression, respectively.
In a recent work, Caron and Chretien (2020) provided a ﬁnite sample analysis of the nonlinear function estimation and showed that the parameter learned through empirical risk minimization converges to the true parameter with high probability as the model complexity tends to inﬁnity, implying the existence of double descent.
Among all the aforementioned efforts, one particularly interesting question is whether one can observe more than two descents in the generalization curve.
In a recent work, d’Ascoli et al.
(2020) empirically showed a sample-wise triple-descent phenomenon under the random Fourier feature model.
Similar triple-descent was also observed for linear regression (Nakkiran et al., 2020).
More rigorously, Liang et al.
(2020) presented an upper bound on the risk of the minimum-norm interpolation versus the data dimension in Reproducing Kernel Hilbert Spaces (RKHS), which exhibits multiple descent.
However, a multiple-descent upper bound without a properly matching lower bound does not imply the existence of a multiple-descent generalization curve.
In this work, we study the multiple descent phenomenon by addressing the following questions:
• Can the existence of a multiple descent generalization curve be rigorously proven?
• Can an arbitrary number of descents occur?
• Can the generalization curve and the locations of descents be designed?
In this paper, we show that the answer to all three of these questions is yes.
Further related work is presented in Appendix A.
Our Contribution.
We consider the linear regression model and analyze how the risk changes as the dimension of the data grows.
In the linear regression setting, the data dimension is equal to the dimension of the parameter space, which reﬂects the model complexity.
We rigorously show that the multiple descent generalization curve exists under this setting.
To our best knowledge, this is the ﬁrst work proving a multiple descent phenomenon for any learning model.
Our analysis considers both the underparameterized and overparameterized regimes.
In the overparameterized regime, we show that one can control where a descent or an ascent occurs in the generalization curve.
This is realized through our algorithmic construction of a feature-revealing process.
To be more speciﬁc, we assume that the data is in R D , where D can be arbitrarily large or even essentially inﬁnite.
We view each dimension of the data as a feature.
We consider a linear regression problem restricted on the ﬁrst d features, where d < D .
New features are revealed by increasing the dimension of the data.
We then show that by specifying the distribution of the newly revealed feature to be either a standard Gaussian or a Gaussian mixture, one can determine where an ascent or a descent occurs.
In order to create an ascent when a new feature is revealed, it is sufﬁcient that the feature follows a Gaussian mixture distribution.
In order to have a descent, it is sufﬁcient that the new feature follows a standard Gaussian distribution.
Therefore, in the overparameterized regime, we can fully control the occurrence of a descent and an ascent. As a comparison, in the underparameterized regime, the generalization loss always increases regardless of the feature distribution.
We also consider a dimension-normalized version of the generalization loss under which we show the generalization curve can have multiple descent in the underparameterized regime.
Generally speaking, we show that we are able to design the generalization curve.
On the one hand, we show theoretically that the generalization curve is malleable and can be constructed in an arbitrary fashion.
On the other hand, we rarely observe complex generalization curves in practice, besides carefully curated constructions.
Putting these facts together, we arrive at the conclusion that realistic generalization curves arise from speciﬁc interactions between properties of typical data and the inductive biases of algorithms.
We should highlight that the nature of these interactions is far from being understood and should be an area of further investigations.
P RELIMINARIES AND P ROBLEM F ORMULATION
Notation.
For x ∈ R D and d ≤ D , we let x [1 : d ] ∈ R d a d -dimensional vector with x [1 : d ] = x i for all 1 ≤ i ≤ d . a matrix A ∈ R n × d , we
Moore- pseudoinverse by
A + ∈ R d × n .
We use the big O notation O and write variables in the subscript of O if the implicit constant depends on them.
For example, O n,d,σ (1) is a constant that only depends on n , d , and σ .
If f ( σ ) and g ( σ ) are functions of σ , write f ( σ ) ∼ g ( σ ) if lim f ( σ ) g ( σ ) = 1 .
It will be given in the context how we take the limit.
Distributions.
Let N ( µ, σ 2 ) ( µ, σ ∈ R ) and N ( µ, Σ) ( µ ∈ R n , Σ ∈ R n × n ) denote the univariate and multivariate Gaussian distributions, respectively, where µ ∈ R n and Σ ∈ R n × n is a positive semi-deﬁnite matrix.
We deﬁne a family of trimodal Gaussian mixture distributions as follows
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] For an illustration, please see Fig.
[Figure]
Smaller σ leads to higher concentration around each modes.
Let χ 2 ( k, λ ) denote the noncentral chi-squared distribution with k degrees of freedom and the noncentrality parameter λ .
For example, if X i ∼ N ( µ i , 1) (for i = 1 , 2 , . . . , k ) are independent Gaussian random variables, we have (cid:80) ki
=1 X 2 i ∼ χ 2 ( k, λ ) ,
where λ = (cid:80) ki =
1 µ 2 i .
We also denote by χ 2 ( k ) the (central) chi-squared distribution with k degrees and the F -distribution by F ( d 1 , d 2 ) where d 1 and d 2 are the degrees of freedom.
Problem Setup.
Let x 1 , . . .
, x n ∈ R D be column vectors that represent the training data of size n and let x test ∈ R D be a column vector that represents the test data.
We assume that they are all independently drawn from a distribution
[Equation]
Let us consider a linear regression problem on the ﬁrst d features, where d ≤ D for some arbitrary large D .
Here, d can be viewed as the number of features revealed.
The design matrix A equals [ x 1 [1 : d ] , . . .
, x n [1 : d ]] (cid:62) ∈ R n × d .
The true linear model is β ∗ ∈ R d .
The noise ε ∈ R n follows the multivariate standard Gaussian distribution N (0 , η 2 I n ) .
Let x = x test [1 : d ] denote the ﬁrst d features of the test data.
For the underparameterized regime where d < n , the least square solution on the training data is
A + ( Aβ ∗ + ε ) .
For the overparameterized regime where d > n , A + ( Aβ ∗ + ε ) is the minimum-norm solution.
In both regimes we consider the solution ˆ β (cid:44) A + ( Aβ ∗ + ε ) .
The excess generalization loss on the test data is then given by
[Equation]
where y
= x (cid:62) β ∗ + ε test and ε test ∼ N (0 , η 2 ) .
We call the term E (cid:2) ( x (cid:62) ( A + A − I ) β ∗ ) 2 (cid:3) the bias and call the term η 2 E (cid:13)(cid:13)
( A (cid:62) )
+ x (cid:13)(cid:13) 2 the variance .
In this paper, we assume β ∗ = 0 and the noise level η = 1 .
In this settings, we get
[Equation]
Remark 1.
In the underparametrized regime, if D is a continous distribution (our construction presented later satisﬁes this condition), the matrix A has independent column almost surely.
In this case, we have A + A
= I and therefore the bias E (cid:2) ( x (cid:62) ( A + A − I ) β ∗ ) 2 (cid:3) vanishes irrespective of the true linear model β ∗ .
In other words, in the underparametrized regime, L d equals η 2 E (cid:107)
( A (cid:62) )
+ x (cid:107)for all β ∗ .
We would like to study the change in the loss caused by the growth in the number of features revealed.
Note that the product ( A + ) (cid:62) x sums over d dimensions.
Once we reveal a new feature, which is equivalent to adding a new row b (cid:62) to A (cid:62) and a new component y to x , the product (cid:20) A (cid:62) b (cid:62) (cid:21) + (cid:20) x y (cid:21) sums over d + 1 dimensions.
As a result, to compare quantities of different dimensions, we need to normalize the generalization loss by the dimension.
We deﬁne the dimension-normalized generalization loss L (cid:48) d as follows
[Equation]
[Table]
Local Maximum and Multiple Descent.
We say that a local maximum occurs at a dimension d ≥if L (cid:48) d − 1 < L (cid:48)
d and L (cid:48) d > L (cid:48) d +1 .
Intuitively, a local maximum occurs if there is an increasing stage of the generalization loss, followed by a decreasing stage, as the dimension d grows.
Additionally, we deﬁne L {+(cid:48) 
+} (cid:44) −∞ .
If the generalization loss exhibits a single descent, based on our deﬁnition, a unique local maximum occurs at d = 1 .
For a double-descent generalization curve, a local maximum occurs at two different dimensions.
In general, if we observe a local maximum at K different dimensions we call it a K -descent.
U NDERPARAMETERIZED R EGIME
First, we present our main theorem for the underparametrized regime below, whose proof is deferred to the end of Section 3.
It states that the un-normalized generalization loss L d is always nondecreasing as d grows.
Moreover, it is possible to have an arbitrarily large ascent, i.e., L d +1 − L d >
C for any C > 0 .
Theorem 1 (Proof in Appendix B.1) .
If d < n , we have L d +1 ≥ L d irrespective of the data distribution.
Moreover, for any C > 0 , there exists a distribution D such that L d +1 − L d > C . For the dimension-normalized generalization loss L (cid:48) d , there can be both ascents and descents. And it is possible to specify where the local peaks in the generalization curve occur.
Theorem 2 (Underparameterized regime) .
Let D + 2 < √ 2 n .
For any 1 < d 1 < d 2 < · · · < d K <
D where d j +1 − d j ≥ 2 , there exists a distribution D such that a local maximum of the L (cid:48) d curve occurs at d j .
Note that the assumption d j +1 − d j ≥ 2 is necessary because two local maxima may not be adjacent.
We present an example in Fig.
Remark 2 ( D can be a product distribution) .
As will be clear later in the proof of Theorem 2, the distribution D can be made as simple as a product distribution D
= D 1 × · · · × D D
such that x
i,j iid ∼ D j for all 1 ≤ i ≤ n , where D j is either sampled from N (0 , 1) or a Gaussian mixture N mix σ j for some σ j > 0 .
As a consequence, by permuting the order of D i ’s in the product distribution, we can change the order of revealing the features.
Remark 3 (Kernel regression on Gaussian data) .
In light of Remark 2, D can be chosen to be a product distribution that consists of only N (0 , 1) and N mix σ j .
Note that one can simulate N mix σ, [Figure] with N (0 , 1) through the inverse transform sampling.
To see this, let F N (0 , 1) and F N mix σ, 1 be the cdf of N (0 , 1) and N mix σ, 1 , respectively.
If X ∼ N (0 , 1) , we have F N (0 , 1) ( X ) ∼ Unif((0 , 1)) and therefore ϕ σ ( X ) (cid:44) F − 1 N mix σ, 1 ( F N (0 , 1) ( X )) ∼ N mix σ, 1 .
In fact, we can use a multivariate Gaussian
D (cid:48)
=
N (0 , I D × D ) and a sequence of non-linear kernels k [1: d ] ( x, y ) (cid:44) (cid:104) φ [1: d ] ( x ) , φ [1: d ] ( y ) (cid:105) , where the feature map is φ [1: d ] ( x ) (cid:44) [ φ 1 ( x 1 ) , φ 2 ( x 2 ) , . .
. , φ d ( x d )] (cid:62) ∈ R d .
Here is a simple rule for deﬁning φ j : If D j = N (0 , 1) , we set φ j to the identity function.
If D j = N mix σ j , we set φ j to ϕ σ j .
Thus, the problem becomes a kernel regression problem on the standard Gaussian data.
Getting back Theorem 2, let us discuss how we will construct such a distribution D inductively.
We ﬁx d .
Again, denote the ﬁrst d features of x test by x (cid:44) x test [1 : d ] .
Let us consider adding an additional component to the training data x 1 [1 : d ] , . . .
, x n [1 : d ] and test data x so that we increment the dimension d by 1.
Let b i ∈ R denote the additional component that we add to the vector x i (so that the new vector is [ x i [1 : d ] (cid:62) , b i ] (cid:62) .
Similarly, let y ∈ R denote the additional component that we add to the vector x .
We form the column vector b =
[ b 1 , . . .
, b n ] (cid:62) ∈ R n that collects all additional components that we add to the training data.
We consider the change in (dimension-normalized) generalization loss as follows [Equation] (cid:21) [Equation]
xy (cid:21)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Note that the components b 1 , . . .
, b n , y are i.i.d.
Lemma 3 relates the pseudo-inverse of [ A, b ] (cid:62) to that of A (cid:62) .
Lemma 3 (Proof in Appendix B.2) .
Let A ∈ R n × d and 0 (cid:54) =
b ∈ R n × 1 , where n ≥ d + 1 .
Additionally, let P = AA + and Q = bb + = bb (cid:62) (cid:107) b (cid:107) 2 , and deﬁne z (cid:44) b (cid:62) ( I − P ) b (cid:107) b (cid:107).
If z (cid:54) = 0 and the columnwise partitioned matrix [ A, b ] has linearly independent columns, we have
[Equation]
In our construction of D , the components D j are all continuous distributions.
The matrix I − P is an orthogonal projection matrix and therefore rank( I − P ) = n − d .
As a result, it holds almost surely that b (cid:54) = 0 , z (cid:54) = 0 , and [ A, b ] has linearly independent columns.
Thus the assumptions of Lemmaare satisﬁed almost surely.
In the sequel, we assume that these assumptions are always fulﬁlled.
Lemma 4 (Proof in Appendix B.3) .
Assume d , n > d + 2 and P are ﬁxed, where P ∈ R n × n is an orthogonal projection matrix whose rank is d .
Deﬁne z (cid:44) b (cid:62) ( I − P ) b (cid:107) b (cid:107), where b =
[ b 1 , . . .
, b n ] (cid:62) ∈ R n .
If y, b 1 , · · · , b n iid ∼ N mix σ, 1 , we have E [1 /z ]
=
O n,d,σ (1) and E
[ y 2 /b (cid:62) ( I − P ) b ]
=
O n,d,σ (1) .
Theorem 5 provides an upper bound for the following quantity
[Equation]
− [Equation] ( A
[Equation]
2 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
A, x   if b 1 , . . .
, b n , y are i.i d. according to N (0 , 1) or N mix σ, 1 .
This quantiry is similar to the difference between the dimension-normalized generalization loss L (cid:48) d +1 − L (cid:48) d but with expectation only over b and y .
Theorem 5 (Proof in Appendix B.4) .
Conditioned on A and x , the following statements hold
: (a) If d + 2 < √ 2 n and b 1 , . . .
, b n , y iid ∼ N (0 , 1) , we have E b,y  
[Equation]
[Equation]
(b) If d + 2 < n and b 1 , . . .
, b n , y iid ∼ N mix σ, 1 , we have E b,y
[Equation]
1 d +(cid:20) A (cid:62) b (cid:62) [Equation] (cid:20) xy (cid:21)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13) 1 d (
[Equation]
[Equation]
where O n,d,σ (1) is a universal constant that only depends on n , d , and σ .
Corollary 6.
Assume d + 2 < √ 2 n .
If either b 1 , . . .
, b n , y iid ∼ N (0 , 1) or b 1 , . . .
, b n , y iid ∼ N mix σ, 1 , and by taking expectation over all random variables, we have
[Equation]
We will use Theorem 5 in two different ways.
The ﬁrst way is presented in Corollary 6.
We would like to show inductively (on d ) that L (cid:48) d is ﬁnite for every d .
Provided that we are able to guarantee ﬁnite L (cid:48) 1 , Corollary 6 implies that L (cid:48) d is ﬁnite for every d if the components are always sampled from
N (0 , 1) or N mix σ, 1 .
Alternatively, we can use Theorem 5 to create a descent, i.e., make L (cid:48)
d +1 < L (cid:48) d .
In light of (2), to make the left-hand side negative, we need
[Equation]
−
[Equation] d + 2) [Equation] ) < 0 , which is equivalent to
[Equation]
One we take expectation over A and x , we need the above equation to hold in expectation in order to create a descent, i.e.,
[Equation]
Provided that L (cid:48)
d can be made sufﬁciently large, letting L (cid:48)
D satisfy the above inequality and then adding an additional N (0 , 1) entry will lead to L (cid:48) d +1 < L (cid:48) d . Making a large L (cid:48) d , in turn, can be achieved by adding an entry sampled from N mix σ, 1 when the data dimension increases from d − 1 to d in the previous step.
Indeed, Theorem 7 shows that adding a N mix σ, 1 feature can increase the loss by arbitrary amount.
Theorem 7 (Proof in Appendix B.5) .
For any C > 0 and E (cid:13)(cid:13) ( A + ) (cid:62) x
(cid:13)(cid:13) 2 < + ∞ , there exists a σ > 0 such that if b 1 , . . .
, b n , y iid ∼ N mix σ, 1 , we have
[Equation]
We are now ready to prove Theorem 2.
Proof of Theorem 2.
We construct D inductively.
Let D 1 = N (0 , 1) .
When d = 1 , we have
A = [ x 1 [1 : d ] , . . . , x n [1 : d ]] (cid:62) = [ x 1 , 1 , . . .
x n, 1 ] (cid:62) ∈ R n , which is a column vector.
Therefore, A + = A (cid:62) (cid:107) A (cid:107) 2 .
As a result, we get
[Equation]
where x ∼ N (0 , 1) and (cid:107) A (cid:107) 2 ∼ χ 2 ( n ) .
Since we will set D j (for j ≥ 2 ) to either N (0 , 1) or N mix σ, 1 , by Corollary 6, we have L [-j +
-](cid:48)
j +1 =
O n,j,σ j +1 ( L (cid:48) j ) .
By induction, we obtain that L (cid:48) j is ﬁnite for all 1 ≤ j ≤ D .
We deﬁne d 0 (cid:44) 0 .
Assume that we have determined distributions D 1 , . . .
, D d j +1 , where 0 ≤ j <
K .
We set D d j +2 , . . .
, D d j +1 − 1 to N (0 , 1) .
For D d j +1 , by Theorem 7, we pick σ d j +1 such that if
D d j +1 =
N mix σ dj
+1 , we have
[Equation]
Next, we set D
d
j +
+
=
N (0 , 1) .
Taking the expectation of (2) in Theorem 5 over all random variables, we have
[Equation]
where the last inequality is due to (3).
So far we have constructed a local maximum at d j +1 .
By induction, we conclude that a local maximum occurs at every d j .
Remark 4.
From Remark 2 and the proof of Theorem 2 it is clear that D
= D 1 × · · · × D D is a product distribution.
The construction in the proof also shows that the generalization curve is actually determined by the speciﬁc choice of the D i ’s. Note that permuting the order of D i ’s is equivalent to changing the order by which the features are being revealed (i.e., permuting the entries of the data x i ’s).
Therefore, given the same data points x 1 , · · · , x n ∈ R D , we can create many different generalization curves simply by changing the order of the feature-revealing process.
O VERPARAMETERIZED R EGIME
In this section, we study the multiple decent phenomenon in the overparameterized regime.
Note that as stated in Section 2, we consider the minimum-norm solution here.
As stated in the following theorem, we require d ≥ n +8 , which means d starts at roughly the same order as n .
In other words, the result covers almost the entire spectrum of the overparameterized regime.
Theorem 8 (Overparameterized regime) .
Let n < D − 9 .
Given any sequence ∆ n +8 , ∆ n +9 , . . .
, ∆ D − 1 where ∆ d ∈ {↑ , ↓} , there exists a distribution D such that for every n + 8 ≤ d ≤ D − 1 , we have [Equation] < [Equation]
if
[Equation]
n+ n+ n+ n+ n+ n+ n+14 Dimension of data ( o r ‘ ( In Theorem 8, the sequence ∆ n +8 , ∆ n +9 , · · · , ∆ D − 1 is just used to specify the increasing/decreasing behavior of the L (cid:48) d sequence for d > n + 8 .
Compared to Theorem 2 for the underparameterized regime, where one is able to fully control the ascents but only partially control the descents, Theorem 8 indicates that one is able to fully control both ascents and descents in the overparameterized regime by placing an ascent/descent wherever one desires.
Fig.
3 illustrates an example.
Lemma 9 gives the pseudo-inverse of A when d > n .
Lemma 9 (Proof in Appendix C.1) .
Let A ∈ R n × d and b ∈ R n × 1 , where n ≤ d .
Assume that matrix A and the columnwise partitioned matrix B (cid:44)
[ A, b ] have linearly independent rows.
Let
G (cid:44) ( AA (cid:62) ) − 1 ∈ R n × n and u (cid:44) [Equation] Gb ∈ R 1 × n .
We have [Equation]
Lemma 10 establishes ﬁnite expectation for several random variables.
These ﬁnite expectation results are necessary for Theorem 11 and Theorem 12 to hold.
Technically, they are the dominating random variables needed in Lebesgue’s dominated convergence theorem.
Lemma 10 indicates that to guarantee these ﬁnite expectations, it sufﬁces to set the ﬁrst n + 8 distributions to the standard normal distribution and then set D n +8 , . . .
, D D to either a Gaussian or a Gaussian mixture distribution.
In fact, in Theorem 11 and Theorem 12, we always add a Gaussian distribution or a Gaussian mixture.
Lemma 10 (Proof in Appendix C.2) .
Let D
= D 1 × · · · × D D be a product distribution where
(a) D d = N (0 , 1) if d = 1 , . . . , n + 8 ; and (b) D d is either N (0 , σ 2 d ) or N mix σ d ,µ d for d > n + 8 .
Let D [1: d ] denote D 1 × · · · × D d .
Assume that every row of A ∈ R n × d and x ∈ R
d
× 1 are i.i.d.
and follow D [1: d ] .
For any d such that n + 8 ≤ d ≤ D , all of the followings hold:
[Equation]
Theorem 11 shows that in order to have L d
+1 < L d and L (cid:48) d +1 < L (cid:48)
d , it sufﬁces to add a Gaussian feature.
Theorem 11 (Appendix C.3) .
If E [ (cid:107) ( A (cid:62) A ) + x (cid:107) 2 ] > 0 and all equations in (4) hold, there exists σ > 0 such that if y, b 1 , . .
, b n iid ∼ N (0 , σ 2 ) , we have
[Equation]
Theorem 12 shows that adding a Gaussian mixture feature can make L d +1 > L d and L (cid:48) d +1 > L (cid:48) d .
Theorem 12 (Proof in Appendix C.4) .
Assume E (cid:107) ( A + ) (cid:62) x (cid:107) 2 < + ∞ .
For any C > 0 , there exist µ , σ > 0 such that if y, b 1 , . . .
, b n iid ∼ N mix σ,µ , we have
[Equation]
The proof of Theorem 8 immediately follows from Theorem 11 and Theorem 12.
Proof of Theorem 8.
We construct the product distribution D
=
(cid:81)
Dd =1 D d .
We set D d = N (0 , 1) for d = 1 , . . . , n + 8 .
For n + 8 < d ≤ D , D d is either N (0 , σ 2 d ) or N mix σ d ,µ d depending on ∆ d being either ↓ or ↑ . First we show that for each step d , the assumption E [ (cid:107) ( A (cid:62) A ) + x (cid:107) 2 ] > 0 of Theorem 11 is satisﬁed. If E [ (cid:107) ( A (cid:62) A ) + x (cid:107) 2 ] = 0 , we know that ( A (cid:62) A ) + x = 0 almost surely.
Since D is a continuous distribution, the matrix A has full row rank almost surely.
Therefore, rank(( A (cid:62) A ) + ) = rank( A (cid:62) A )
= n almost surely. Thus dim ker( A (cid:62) A ) + = d − n ≤ d − 1 almost surely, which implies x / ∈ ker( A (cid:62) A ) + . In other words, ( A (cid:62) A ) + x (cid:54) = 0 almost surely.
We reach a contradiction. Moreover, by Lemma 10, the assumption E (cid:107) ( A + ) (cid:62) x (cid:107) 2 < + ∞ of Theorem 12 is also satisﬁed. If ∆ d − 1 = ↓ , by Theorem 11, there exists σ d > 0 such that if D d = N (0 , σ 2 d ) , then L d < L d − 1 and L (cid:48)
d < L (cid:48)
d − 1 . Similarly if ∆ d − 1 = ↑ , by Theorem 12, there exists σ d and µ d such that D d = N mix σ d ,µ d guarantees L d > L d − 1 and L (cid:48) d > L (cid:48) d − 1 .
C ONCLUSION
Our work proves that the expected risk of linear regression can manifest multiple descents when the number of features increases and sample size is ﬁxed.
This is carried out through an algorithmic construction of a feature-revealing process where the newly revealed feature follows either a
Gaussian distribution or a Gaussian mixture distribution.
Notably, the construction also enables us to control local maxima in the underparameterized regime and control ascents/descents freely in the overparameterized regime.
Overall, this allows us to design the generalization curve away from the interpolation threshold.
We conjecture that the same multiple-descent generalization curve can occur in non-linear neural networks and we humbly suggest that entities with inﬁnite computational powers investigate this phenomenon.
A F URTHER R ELATED W ORK
Our work is directly related to the recent line of research in the theoretical understanding of the double descent (Belkin et al., 2019a; Hastie et al., 2019; Xu and Hsu, 2019; Mei and Montanari, 2019) and the multiple descent phenomenon (Liang et al., 2020).
Here we brieﬂy discuss some other work that is closely related to this paper.
Least Square Regression.
In this paper we focus on the least square linear regression with no regularization.
For the regularized least square regression, De Vito et al.
(2005) proposed a selection procedure for the regularization parameter.
Advani and Saxe (2017) analyzed the generalization of neural networks with mean squared error under the asymptotic regime where both the sample size and model complexity tend to inﬁnity.
Richards et al.
(2020) proved for least square regression in the asymptotic regime that as the dimension-to-sample-size ratio d/n grows, an additional peak can occur in both the variance and bias due to the covariance structure of the features.
As a comparison, in this paper the sample size is ﬁxed and the model complexity increases.
Rudi and Rosasco (2017) studied kernel ridge regression and gave an upper bound on the number of the random features to reach certain risk level.
Our result shows that there exists a natural setting where by manipulating the random features one can control the risk curve.
Over-Parameterization and Interpolation.
The double descent occurs when the model complexity reaches and increases beyond the interpolation threshold.
Most previous works focused on proving an upper bound or optimal rate for the risk.
Caponnetto and De Vito (2007) gave the optimal rate for least square ridge regression via careful selection of the regularization parameter.
Belkin et al.
(2019b) showed that the optimal rate for risk can be achieved by a model that interpolates the training data.
In a series of work on kernel regression with regularization parameter tending to zero (a.k.a.
kernel ridgeless regression), Rakhlin and Zhai (2019) showed that the risk is bounded away from zero when the data dimension is ﬁxed with respect to the sample size.
Liang and Rakhlin (2019) then considered the case when d (cid:16) n and proved a risk upper bound that can be small given favorable data and kernel assumptions.
Instead of giving a bound, our paper presents an exact computation of risk in the cases of underparameterized and overparameterized linear regression, and proves the existence of the multiple descent phenomenon.
Wyner et al.
(2017) analyzed AdaBoost and Random Forest from the perspective of interpolation.
There has also been a line of work on wide neural networks (Arora et al., 2019a;b;c; Du et al., 2019; Allen-Zhu et al., 2019; Wei et al., 2019; Cao and Gu, 2019; Advani et al., 2020; Chen and Xu, 2020; Zou et al., 2020).
Sample-wise Double Descent and Non-monotonicity.
There has also been recent development beyond the model-complexity double-descent phenomenon.
For example, regarding sample-wise non-monotonicity, Nakkiran et al.
(2019) empirically observed the epoch-wise double-descent and sample-wise non-monotonicity for neural networks.
Chen et al.
(2020) and Min et al.
(2020) identiﬁed and proved the sample-wise double descent under the adversarial training setting, and Javanmard et al.
(2020) discovered double-descent under adversarially robust linear regression.
Loog et al.
(2019) showed that empirical risk minimization can lead to sample-wise non-monotonicity in the standard linear model setting under various loss functions including the absolute loss and the squared loss, which covers the range from classiﬁcation to regression.
We also refer the reader to their discussion of the earlier work on non-monotonicity of generalization curves.
Dar et al.
(2020) demonstrated the double descent curve of the generalization errors of subspace ﬁtting problems.
B P ROOFS FOR U NDERPARAMETRIZED R EGIME B. P ROOF OF T HEOREM
Proof.
We follow the notation convention in (1):
[Equation]
[Table]
Recall d < n and the matrix B (cid:48) (cid:44) (cid:20)
A (cid:62) b (cid:62) (cid:21) is of size ( d + 1) × n .
Both matrices B (cid:48) and B (cid:44) A (cid:62) are fat matrices.
As a result, if x (cid:48) (cid:44) (cid:20) xy (cid:21) , we have
[Equation]
Since { z | B (cid:48)
z
= x (cid:48) } ⊆ { z | Bz = x } , we get (cid:107) B (cid:48)
+ x (cid:48) (cid:107) 2 ≥ (cid:107) B + x (cid:107) 2 .
Therefore, we obtain
L d +1 ≥ L d .
The second part of the theorem, which says that for any C > 0 there exists a distribution such that
L d +1 − L d > C , follows directly from Theorem 7.
B. P ROOF OF L EMMA
Proof.
By (Baksalary and Baksalary, 2007, Theorem 1), we have
[Equation]
Deﬁne r (cid:44) A (cid:62) b ∈ R d .
Since A has linearly independent columns, the Gram matrix G
= A (cid:62) A is non-singular.
The Sherman-Morrison formula gives (
[Equation]
where we use the facts r
= A (cid:62) b and AG − 1 =
( A + ) (cid:62) in the last equality.
Therefore
[Equation]
we deduce
[Equation]
Observe that
[Equation]
{+b (cid:107) 
+}
= z .
Therefore, we obtain the desired expression.
B. P ROOF OF L EMMA
Lemma 13 shows that a noncentral χ 2 distribution ﬁrst-order stochastically dominates a central χdistribution of the same degree of freedom.
It will be needed in the proof of Lemma 4.
Lemma 13.
Assume that random variables X ∼ χ ( k, λ ) and Y ∼ χ 2 ( k ) , where λ > 0 .
For any c > 0 , we have
P ( X ≥ c ) > P ( Y ≥ c ) .
In other words, the random variable X (ﬁrst-order) stochastically dominates Y .
Proof.
Let Y 1 , X 2 , . . .
, X k iid ∼ N (0 , 1) and X 1 ∼ N ( √ λ, 1) and all these random variables are jointly independent.
Then X (cid:48) (cid:44) (cid:80) ki
=1 X 2 i ∼ χ 2 ( k, λ ) and Y (cid:48)
(cid:44) Y 2 1 + (cid:80) ki
=
X 2 i ∼ χ 2 ( k ) .
It sufﬁces to show that P ( X (cid:48) ≥ c ) > P ( Y (cid:48) ≥ c ) , or equivalently, P ( |N ( µ, 1)
| ≥ c ) >
P ( |N (0 , 1) | ≥ c ) for all c > 0 and µ (cid:44) √ λ > 0 .
Denote F c ( t ) =
P ( |N ( µ, 1) | ≥ c ) and we have
[Equation]
and thus
[Equation]
>
[Equation]
This shows P ( |N ( µ, 1)
| ≥ c ) > P ( |N (0 , 1)
| ≥ c ) and we are done.
Proof of Lemma 4.
Since b i iid ∼ N mix σ, 1 , we can rewrite b = u + w where w ∼ N (0 , σ 2 I n ) and the entries of u satisfy u i iid ∼ Unif( {− 1 , 0 , 1 } ) .
Furthermore, u and w are independent.
Note that for any ﬁxed n and d , the support of u is ﬁnite and its cardinality only depends on n .
Therefore, we only need to show that conditioning on u , the expectation over w is O n,d,σ (1) .
In other words, for any ﬁxed u , we want to show E w [1 /z | u ]
=
O n,d,σ (1) and E w (cid:104)
[Equation]
Note that since y 2 /σ 2 is ﬁrst-order stochastically dominated by χ 2 (1 , 1) , we have
[Equation]
Therefore, it remains to show E w [1 /z | u ]
=
O n,d,σ (1) and E w (cid:104)
[Equation]
I
[Equation]
P ) b (cid:12)(cid:12)(cid:12)
[Equation]
O n
[Equation]
σ (
[Equation]
Note that
[Equation]
Since P is an orthogonal projection, there exists an orthogonal transformation O depending only on
P such that
[Equation]
where D d = diag([1 , .
, 0]) with d diagonal entries equal to 1 and the others equal to 0.
We denote ˜ u
= O ( u ) , which is ﬁxed (as u and O are ﬁxed), and ˜ w
=
O ( w ) ∼ N (0 , σ 2 I n ) .
It follows that
[Equation]
Observe that
[Equation]
and that these two quantities are independent.
It follows that
[Equation]
By Lemma 13, the denominator (cid:80) ni
= d +1 (˜ u
i + ˜ w i ) 2 /σ 2 ﬁrst-order stochastically dominates χ 2 ( n − d ) .
Therefore, we have
[Equation]
Putting the numerator and denominator together yields
[Equation]
Similarly
[Equation]
we have
[Equation]
B. P ROOF OF T HEOREM
Proof.
First, we rewrite the expression as follows
[Equation]
) where P, Q, z are deﬁned in Lemma 3.
Since y has mean 0 and is independent of other random variables, so that the cross term vanishes under expectation over b and y :
[Equation]
where (cid:104)· , ·(cid:105) denotes the inner product.
Therefore taking the expectation of (5) over b and y yields
E b,y
[Equation]
[Equation]
=
[Equation]
y
[Equation]
We simplify the third term.
Recall that I − P
=
I − AA + is an orthogonal projection matrix and thus idempotent
[Equation]
Thus we have
E b,y
[Equation]
We consider the ﬁrst and second terms.
We write v =
( A + ) (cid:62) x and deﬁne z
= b (cid:62) ( I − P )
b (cid:107) b (cid:107).
The sum of the ﬁrst and second terms equals
[Equation]
[Equation]
where δ = 2 d + 1 d 2 and
[Equation]
The rank of M is at most 2 .
To see this, we re-write M in the following way
[Equation]
Notice that rank( M 1 ) ≤ rank( Q ) , rank( M 2 ) ≤ rank( Q ) , and rank( Q ) = 1 .
It follows that rank( M ) ≤ rank( M 1 ) + rank( M 2 )
= 2 .
The matrix M has at least n − 2 zero eigenvalues.
We claim that M has two non-zero eigenvalues and they are 1 − 1 /z < 0 and 1 .
Since
[Equation]
and
[Equation]
thus PQ has a unique non-zero eigenvalue 1 − z .
Let u (cid:54) =
denote the corresponding eigenvector such that PQu =
(1 − z ) u .
Since u ∈ im P and P is a projection, we have Pu = u .
Therefore we can verify that
[Equation]
To show that the other non-zero eigenvalue of M is 1 , we compute the trace of M
[Equation]
where we use the fact that tr( Q ) = 1 , tr( PQ ) = 1 − z ,
[Equation]
We have shown that M has eigenvalue 1 − 1 /z and M has at most two non-zero eigenvalues.
Therefore, the other non-zero eigenvalue is tr( M ) − (1 − 1 /z )
= 1 .
We are now in a position to upper bound (12) as follows:
[Equation]
Putting all three terms of the change in the dimension-normalized generalization loss yields
[Equation]
For b 1 , . . .
, b n , y iid ∼ N (0 , 1) , we have E [ y 2 ] = 1 .
Moreover, b (cid:62) ( I − P ) b follows χ 2 ( n − d ) a distribution.
Thus 1 b (cid:62) ( I − P ) b follows an inverse-chi-squared distribution with mean 1 n − d − 2 .
Therefore the expectation E [ y 2 b (cid:62) ( I − P ) b ] = 1 n − d − 2 .
Notice that 1 /z follows a 1 + dn − d F ( d, n − d ) distribution and thus E [1 /z ] = 1 + d n − d − 2 .
As a result, we obtain
E b,y   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 d +(cid:20) A (cid:62) b (cid:62) (cid:21) + (cid:20) xy
(cid:21)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)(cid:13)
d ( A + ) (cid:62) x (cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ ( d + 1)(cid:20)(cid:18) d n − d − 2 − d (cid:19) (cid:107) v (cid:107) 2 + n − d −(cid:21) <
d − (cid:107) v (cid:107) 2 (2 n − ( d + 2) 2 ) d ( d + 1) 2 ( n − d − 2) .
For b 1 , . . .
, b n , y iid ∼ N mix σ, 1 , Lemma 4 implies that
E b,y [1 /z ] <
O n,d,σ (1) , and
[Equation]
Therefore [Equation] we conclude that
[Equation]
B. P ROOF OF T HEOREM
Proof.
We start from (11).
Taking expectation over all random variables gives
[Equation]
d
[Equation]
[Equation]
Our strategy is to choose σ so that E (cid:104)
[Equation]
is sufﬁciently large.
This is indeed possible as we immediately show.
Deﬁne independent random variables u ∼ Unif( {− 1 , 0 , 1 } ) and w ∼ N (0 , σ 2 ) .
Since y has the same distribution as u + w , we have
[Equation]
On the other hand,
[Equation]
Together we have
[Table]
[-Therefore, for any C ∈ R and E (cid:13)(cid:13) 1 d ( A + ) (cid:62) x (cid:13)(cid:13) 2 , there exists σ such that E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 d +(cid:20) A (cid:62) b (cid:62) (cid:21) + (cid:20) xy (cid:21)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)(cid:13) 
-]Since
[Table] we have [Table] 2   = + ∞ , which completes the proof.
C P ROOFS FOR O VERPARAMETRIZED R EGIME C. P ROOF OF L EMMA
Proof.
Since A and B have full row rank, ( AA (cid:62) ) − 1 and ( BB (cid:62) ) − 1 exist.
Therefore we have
[Equation]
The Sherman-Morrison formula gives
[Equation]
Hence, we deduce
[Equation]
Transposing the above equation yields to the promised equation.
C. P ROOF OF L EMMA
Proof.
Let us ﬁrst denote v
[Equation]
A
[Equation]
x and
[Equation]
First note that by Cauchy-Schwarz inequality, it sufﬁces to show there exists D such that E [ λ 4 max ( G )] < + ∞ and E (cid:107) v (cid:107) 4 < + ∞ .
We deﬁne A d ∈ R n × d to be the submatrix of A that consists of all n rows and ﬁrst d columns.
Denote
[Equation]
We will prove E [ λ 4 max ( G )] < + ∞ by induction.
The base step is d = n + 8 .
Recall D [1: n +8] = N (0 , I n +8 ) .
We ﬁrst show E [ λ max ( G n +8 )] 4 < + ∞ .
Note that since G n +8 is almost surely positive deﬁnite,
[Equation]
By our choice of D [1: n +8] , the matrix ( A n +8 A (cid:62) n +8 ) − 1 is an inverse Wishart matrix of size n × n with ( n + 8) degrees of freedom, and thus has ﬁnite fourth moment (see, for example, Theorem 4.
in (von Rosen, 1988)).
It then follows that
[Equation]
For the inductive step, assume E [ λ max ( G d )] 4 < + ∞ for some d ≥ n + 8 .
We claim that
[Equation]
or equivalently,
[Equation]
Indeed, this follows from the fact that
[Equation]
under the Loewner order, where b ∈ R n × 1 is the ( d + 1) -th column of A .
Therefore, we have
[Equation]
and by induction, we conclude that E [ λ 4 max ( G )] < + ∞ for all d ≥ n + 8 .
Now we proceed to show E (cid:107) v (cid:107) 4 < + ∞ .
We have
[Equation]
where (cid:107) · (cid:107) op denotes the (cid:96) 2 → (cid:96) 2 operator norm.
Note that
[Equation]
where the last equality uses the fact that A (cid:62) ( AA (cid:62) ) − 2 A is positive semideﬁnite.
Moreover, we deduce
[Equation]
Using the fact that A d A (cid:62) d (cid:52) A d +1 A (cid:62) d
+1 established above
[Equation]
induction gives
[Equation]
It follows that
[Equation]
− [Equation] (cid:105)(cid:17) < + ∞ , (13) where again we use that fact that inverse Wishart matrix (cid:0)
A n +8 A (cid:62) n +8 (cid:1) − 1 has ﬁnite second moment.
Next, we demonstrate E (cid:107) x (cid:107) 4 < + ∞ .
Recall that every D i is either a Gaussian or a Gaussian mixture distribution.
Therefore, every entry of x has a subgaussian tail, and thus E (cid:107) x (cid:107) 4 < + ∞ .
Together with (13) and the fact that x and A are independent, we conclude that
[Equation]
C. P ROOF OF T HEOREM
Proof.
The randomness comes from A, x, y and b .
We ﬁrst condition on A and x being ﬁxed.
Let G (cid:44) ( AA (cid:62) ) − 1 ∈ R n × n and u (cid:44)
[Equation]
We compute the left-hand side but take the expectation over only y for the moment
[Equation]
Let us ﬁrst consider the ﬁrst and third terms of the above equation:
[Equation]
Write G = V Λ V (cid:62) , where Λ = diag
( λ 1 , . . .
, λ n ) ∈ R n × n is a diagonal matrix ( λ i > 0 ) and
V ∈ R n × n is an orthogonal matrix.
Recall b ∼ N (0 , σ 2 I n ) .
Therefore w (cid:44) V (cid:62) b ∼ N (0 , σ 2 I n ) . Taking the expectation over b , we have E
[Equation]
Let R (cid:44) E w (cid:104) ww [Equation] Λ+Λ ww (cid:62) 1+ w (cid:62) Λ w (cid:105) .
We have
[Equation]
{+2 i [Equation] ni =
 λ i ν+}
2 i (cid:21) > and if i (cid:54) = j ,
[Equation]
ni
[Equation]
[Equation]
i
[Equation]
2 i (cid:21) .
Notice that for any w and j , it has the same distribution if we replace w j by − w j .
As a result,
[Equation]
Thus the matrix R is a diagonal matrix and
[Equation]
Thus we get
[Equation]
Moreover, by the monotone convergence theorem, we deduce
[Equation]
It follows that as σ → 0 + ,
[Equation]
Moreover, by (4), we have [Equation]
Next [Equation] we study the term HG 2 H/r 2 :
[Equation]
Again, by the monotone convergence theorem [Equation] we have [Equation]
It follows that as σ → 0 + , [Equation] Moreover [Equation]
by (4)
[Equation]
we have
[Equation]
We apply a similar method to the term (cid:107) Gb (cid:107) 2 r 2 . We deduce
[Equation]
It follows that
[Equation]
The monotone convergence theorem implies
[Equation]
Thus we get as σ → 0 +
E y [ y 2 ] (cid:107) Gb (cid:107)r∼ σ 4 E [tr( G 2 )] , where E [tr( G 2 )] ≤ n E [ λ 2 max (( AA (cid:62) ) − 1 )] < + ∞ . Putting all three terms together, we have as σ → 0 +
L d +1 − L d ∼ − 2 σ 2 E [ (cid:107) ( A (cid:62) A ) + x (cid:107) 2 ] . Therefore, there exists σ > 0 such that L d +1 − L d < 0 . Furthermore, we deduce
[Equation]
C. P ROOF OF T HEOREM
Proof.
Again we ﬁrst condition on A and x being ﬁxed.
Let G (cid:44) ( AA (cid:62) ) − 1 ∈ R n × n and u (cid:44) b (cid:62) G 1+ b (cid:62) Gb ∈ R 1 × n as deﬁned in Lemma 9.
We also deﬁne the following variables:
[Equation]
We compute L (cid:48) d +1 − L (cid:48)
d but take the expectation over only y for the moment
[Equation]
Our strategy is to make E [ y 2 (cid:107) Gb (cid:107) 2 r 2 ] arbitrarily large.
To this end, by the independence of y and b we have
[Equation]
By deﬁnition of N mix σ,µ , with probability 2 / 3 , y is sampled from either N ( µ, σ 2 ) or N ( − µ, σ 2 ) , which implies E [ y 2 ] ≥ 13 µ 2 . For each b i , we have [Equation]
Also note that G is positive deﬁnite.
It follows that
E
[Equation]
Altogether we have
[Equation]
Let µ = 1 /σ 2 and we have
[Equation]
where we switch the order of expectation and limit using the monotone convergence theorem.
Taking full expectation over A, x, b and y of (14) and using the assumption that E (cid:107) v (cid:107) 2 < + ∞ we have
L
[Equation]
− L
[Equation]
as σ → 0 + .
In addition, we have as σ → 0 + ,
[Equation]