A NISOTROPIC R ANDOM F EATURE R EGRESSION IN H IGH D IMENSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
In contrast to standard statistical wisdom, modern learning algorithms typically ﬁnd their best performance in the overparameterized regime in which the model has many more parameters than needed to ﬁt the training data.
A growing number of recent works have shown that random feature models can offer a detailed theoretical explanation for this unexpected behavior, but typically these analyses have utilized isotropic distributional assumptions on the underlying data generation process, thereby failing to provide a realistic characterization of real-world models that are designed to identify and harness the structure in natural data.
In this work, we examine the high-dimensional asymptotics of random feature regression in the presence of structured data, allowing for arbitrary input correlations and arbitrary alignment between the data and the weights of the target function.
We deﬁne a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment.
We also clarify several previous observations in the literature by distinguishing the behavior of the samplewise and parameter-wise learning curves, ﬁnding that sample-wise multiple descent can occur at scales dictated by the eigenstructure of the data covariance, but that parameter-wise multiple descent is limited to double descent, although strong anisotropy can induce additional signatures such as wide plateaus and steep cliffs.
Finally, these signatures are related to phase transitions in the spectrum of the feature kernel matrix, and unlike the double descent peak, persist even under optimal regularization.
I NTRODUCTION
While deep learning races ahead with its impressive list of practical successes, it has often done so by building larger models and training on ever larger datasets.
For example, recent large language models leverage billions (Brown et al., 2020; Raffel et al., 2019) or even trillions (Fedus et al., 2021) of parameters.
As the state-of-the-art pushes the computational frontier, model development, experimentation, and exploration become increasingly costly, both in terms of the computational resources and person-hours.
To facilitate the development of improved models, it is becoming ever more necessary to develop theoretical models that can help guide model development in these high-dimensional scenarios.
Indeed, our theoretical understanding is so limited that even linear regression, the most basic machine learning method of all, is still yielding surprises in high dimensions (Hastie et al., 2019).
While linear regression can capture many salient features of high-dimensional data , it is not a reasonable model for high-dimensional models .
In particular, there is no natural data-independent notion of model size: the number of parameters in the model is intimately tied (usually equal) to the dimensionality of the datapoints.
This limitation means that linear regression may not accurately capture the effect of overparameterization, which appears to be a crucial ingredient in modern deep learning models (Zhang et al., 2016).
Beyond linear regression, the next simplest model is arguably the random feature model of Rahimi et al.
(2007), which turns out to be tractable in high dimensions while also allowing for a natural data-independent lever to vary model size – the number of random features.
Random feature models have also gained independent interest through their relationship to neural kernels and Gaussian processes (Neal, 1996; Lee et al., 2017; Novak et al., 2018; Lee et al., 2019;
Jacot et al., 2018).
While recent works have examined the asymptotic performance of random feature models, most have done so in the restricted setting of isotropic covariates and target functions (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a;b).
Based on the conjectured Gaussian equivalence principle of Goldt et al. (2020b), a few recent works have begun relaxing these simplifying assumptions in an effort to describe more natural data distributions (Loureiro et al., 2021; d’Ascoli et al., 2021).
In this work, we study high-dimensional random feature regression for anisotropic data and target functions, setting the results of d’Ascoli et al. on a more rigorous footing and revealing a host of novel phenomena related to the alignment of the data and the target weights.
1. C ONTRIBUTIONS
Our primary contributions are to:
Derive theoretical predictions for the test error, bias, and variance of random feature regression for anisotropic Gaussian covariates under general linear target functions in the high-dimensional asymptotic limit (Section 3.1); 2.
Prove that overparameterization reduces the total test error, bias, and variance, even in the presence of anisotropy (Section 3.2); 3.
Deﬁne a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment (Deﬁnition 2.1, Section 3.3); 4.
Demonstrate that anisotropy can induce sample-wise multiple descent, but prove that parameter-wise multiple descent is limited to double descent (Section 3.4); 5.
Provide a theoretical explanation for steep cliffs observed in parameter-wise learning curves that persist under optimal regularization, and relate them to phase transitions in the spectrum of the feature kernel matrix (Section 3.4).
1. R ELATED WORK
Our analysis builds on a series of works that have studied the exact high-dimensional asymptotics of the test error for a growing class of model families and data distributions.
Early work in this direction examined minimum-norm interpolated least squares and ridge regression in the high-dimensional random design setting (Belkin et al., 2019; Dobriban et al., 2018; Hastie et al., 2019), ﬁnding surprising phenomena such as double descent and a negative optimal ridge constant (Kobak et al., 2020).
These analyses were generalized in (Richards et al., 2021; Wu & Xu, 2020; Hastie et al., 2019) to allow for anisotropic covariance matrices and weights that generate the targets.
This anisotropic conﬁguration was studied in further detail in (Mel & Ganguli, 2021), uncovering nuanced behavior such as sample-wise multiple descent and a sequence of phase transitions in the performance and optimal regularization.
While linear regression can provide valuable insight into generalization performance in high dimensions, it is not sufﬁciently complex to faithfully reproduce the features of non-linear models, such as the effects of overparameterization.
Random feature models, on the other hand, can capture many such features while still maintaining analytical tractability.
Recent results have developed a detailed understanding of isotropic random feature regression in the high-dimensional asymptotic limit, providing a precise characterization of peaks in the test error and the beneﬁts of overparameterization (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a).
The origin of these peaks was explained by means of a ﬁne-grained decomposition of the bias and variance in (Adlam & Pennington, 2020b; Lin & Dobriban, 2020).
While most prior work examining the high-dimensional asymptotics of random feature models have leveraged the isotropy of the covariate distribution, this assumption was recently relaxed in (Liao et al., 2020) for random Fourier feature models, in (Anonymous, 2021) in the study of covariate shift, and in a series of works based on the
Gaussian equivalence principle of Goldt et al. (2020a;b) (Gerace et al., 2020; Loureiro et al., 2021).
Though most of these works did not pursue a detailed investigation of the anisotropy itself, while ﬁnalizing this manuscript we became aware of concurrent work (d’Ascoli et al., 2021) which does focus on the interplay of the anisotropic input features and target functions, as we do here. Whereas d’Ascoli et al.
(2021) leverage the Gaussian equivalence principle and the replica method from statistical physics, we use a completely different set of techniques stemming from the literature on random matrix theory and operator-valued free probability (Pennington & Worah, 2018; 2019;
Adlam et al., 2019; Adlam & Pennington, 2020a; Louart et al., 2018; Péché et al., 2019; Far et al., 2006; Mingo & Speicher, 2017), which can be made completely rigorous. d’Ascoli et al.
(2021) investigates the role of anisotropy not just under squared loss, as we do here, but also under logistic loss, and obtains good numerical support for the Gaussian equivalence conjecture in that setting. In contrast, our narrower and deeper focus on squared loss allows us not only to uncover a host of novel phenomena, including multiple descent, steep error cliffs, and spectral phase changes, but also to rigorously prove several propositions about their existence and properties. We provide a more in-depth discussion of these and other related works in App. A. Our work relies heavily on the concept of alignment (see Def. 2.1) between the data covariance and the target vector coefﬁcients.
A completely analogous deﬁnition appears in Anonymous (2021) in the context of covariate shift. Owing to the similarity of these deﬁnitions as well as the underlying formulas, many of the techniques and analyses of Anonymous (2021) can be applied in our context, leading to a kind of mirroring of results. As the underlying implications and interpretations are otherwise quite different, we refrain from repeatedly commenting on these recurring connections.
P RELIMINARIES 2. P ROBLEM SETUP AND NOTATION
We study random feature regression (Rahimi et al., 2007) as a model for learning an unknown linear function of the data,
[Equation]
from m independent samples ( x i , y i ) 2 R n 0 ⇥
R , i = 1 , . . .
, m , where the covariates are Gaussian, x i ⇠ N (0 , ⌃ ) , and where ✏ i ⇠ N (0 ,  2 ✏ ) is additive noise (that we take to be present on the training samples only).
We focus on the high-dimensional proportional asymptotics in which the input feature dimension n 0 , the hidden layer size n 1 , and the number of samples m all tend to inﬁnity at the same rate, with  := n 0 /m and := n 0 /n 1 held constant. The overparameterization ratio  / = n 1 m is a measure of the normalized complexity of the random feature model. Interestingly, under this model in our high-dimensional setup, any nonlinear component of the signal to be learned behaves like additive noise, so the linear function in Eq. (1) does not sacriﬁce generality (Mei & Montanari, 2019; Adlam & Pennington, 2020a).
Following many works on high-dimensional regression (see e.g. Dobriban et al. , we assume the coefﬁcient vector  is random,  ⇠ N (0 , ⌃  ) , though allowing for deterministic  would be a straightforward extension and would simply involve replacing ⌃  !
> throughout.
Denoting the training dataset as X =
[ x 1 , . .
, x m ] and the test point as x , the random features are,
[Equation]
where W 2 R n 1 ⇥ n 0 is a random weight matrix with i.i.d.
standard Gaussian entries, and  ( · ) : R !
R is an activation function that is applied elementwise.
These random features deﬁne the kernel
[Equation]
which can be used to compute the model’s predictions on a point x as
[Equation]
Here we have deﬁned the training labels as Y :=
[ y ( x 1 ) , . . .
, y ( x m )] , and the regularized kernel matrix as K :=
K ( X, X ) +  I m , as well as K x := K ( X, x ) for   0 .
The training error is given by,
[Equation]
and the test error (without label noise on the test point) can be written as
[Equation]
where the outermost expectation over  has been suppressed since the quantity concentrates sharply around its mean and the inner expectation is computed over all the randomness from training, i.e. W ,
X , and ✏ .
The test error can be decomposed into its bias and variance components as
[Equation]
The bias and variance are computed with respect to all the randomness from training, i.e. W , X , ✏ , which differs from common practice in the statistics literature, but which is necessary to obtain a decomposition that is intuitive and unambiguous when the predictive function relies on randomness from multiple sources (Adlam & Pennington, 2020b; Lin & Dobriban, 2020).
As noted by (Hastie et al., 2019; Wu & Xu, 2020; Mel & Ganguli, 2021), the test error of linear regression depends on the geometry of ( ⌃ , ⌃  ) (or ( ⌃ ,  ) in the case of nonrandom  ), and we will ﬁnd the same to be true for random feature regression.
As such, we decompose the training and  covariance matrices into eigenbases as ⌃ =
P
[Equation]
[Equation] i v i > and ⌃  = P
[Equation] =1   i v  i v  i > , where the eigenvalues are in nondecreasing magnitude, i.e.  1   2  . . . 
[Equation] 0 and   1    2  . . .
   n 0 .
Following (Anonymous, 2021), we then deﬁne the overlap coefﬁcients as
[Equation]
to measure the alignment of ⌃  with the i th eigendirection of ⌃ .
Note that for deterministic  this deﬁnition simply gives the square of the component of  in the i th eigendirection, i.e. q i
= ( v > i  ) 2 .
A SSUMPTIONS
In order to deﬁne the limiting behavior of the test error in Eq.
(6), it is necessary to impose some regularity conditions on ⌃ and ⌃  .
As in (Wu & Xu, 2020; Anonymous, 2021), the limiting spectra of these matrices cannot be considered independently because their eigenspaces may align.
This alignment is most conveniently described in an eigenbasis of ⌃ .
Assumption 1.
We deﬁne the joint spectral distribution (JSD) as
[Equation]
and assume it converges in distribution to some µ , a distribution on R 2+ as n 0 ! 1 .
We refer to µ as the limiting joint spectral distribution (LJSD), and emphasize that this deﬁnes the relevant limiting properties of the covariate and weight distributions 2 .
Often we use (  , q ) for random variables sampled jointly from µ and denote the marginal of  under µ with µ data .
Since the conditional expectation E [ q |  ] is an important object in our study, we assume the following for simplicity.
Assumption 2.
µ is either absolutely continuous or a ﬁnite sum of delta masses and the expectations of  and q are ﬁnite.
Moreover, E µ [  q ] = 1 .
The normalization condition E µ [  q ] = 1 is not necessary for our analysis, but, as in (Mel & Ganguli, 2021), it enforces consistent signal-to-noise ratios across different target functions and thereby facilitates meaningful comparisons of the test error.
As we will eventually consider the high-dimensional limit, it is convenient to deﬁne the asymptotic covariance scale as s = lim n 0 !
1 1 n 0 tr( ⌃ ) =
E µ [  ] under the limiting behavior speciﬁed in
Assumption 1.
One important case is when ⌃  = I n 0 , in which case the LJSD degenerates to µ ; deﬁned by
[Equation] Following Anonymous (2021), we also enforce the following standard regularity assumptions on the activation functions to ensure the existence of the moments and derivatives we compute.
Assumption 3.
The activation function  : R !
R is assumed to be differentiable almost everywhere.
We assume there exists a universal constant C such that, |  ( x ) | , |  0 ( x ) | =
O (exp( Cx )) .
2. A SIMPLE FAMILY OF ANISOTROPIC DISTRIBUTIONS
The assumptions above allow for a wide class of possible asymptotic covariance structures and spectra. To allow for concrete examples that have intuitive interpretations, we introduce the following simple family of distributions that capture multi-scale structure in the input, which we refer to as d -scale LJSDs.
In particular, for d 2 N and real 0 < ↵  1 and ✓ , we deﬁne
[Equation]
) where C :=
[Equation]
and
[Equation]
[Equation]
1 enforce the normalization conditions
[Equation]
These simple covariance models capture the hierarchy of scales that often characterize the structure of natural datasets.
Note that by setting ↵ = 1 and s = 1 , the distribution reduces to the isotropic case with identity covariance.
For the nontrivial setting in which ↵ < 1 , the data exhibits a sequence of d distinct scales where each scale is a factor ↵ smaller than the previous one.
The exponent ✓ parameterizes the strength of the weight-data alignment in an intuitive way.
When ✓ = 0 , there is no alignment, and ⌃  is proportional to the identity.
When ✓ > 0 , there is strong alignment, as the large eigendirections of the training distribution correspond to large eigendirections of ⌃  .
When ✓ < 0 , there is anti-alignment, as the large eigendirections of the training distribution correspond to small eigendirections of ⌃  .
2. D EFINITION OF THE STRENGTH OF WEIGHT - DATA ALIGNMENT
As the preceding discussion has suggested and as we will see explicitly in the following section, the LJSD µ captures all of the information about the pair of covariance matrices ( ⌃ , ⌃  ) that is relevant for describing the asymptotic test error, bias, and variance.
As the alignment between ⌃ and ⌃  can vary in strength among the different eigendirections, the concept of alignment is inherently multi-dimensional.
Nevertheless, by requiring strong alignment along the larger eigendirections, it is possible to deﬁne the following natural partial order on the space of possible weight-data alignments
Deﬁnition 2.1.
Let µ 1 and µ 2 be LJSDs with the same marginal distribution of  .
If the asymptotic overlap coefﬁcients are such that E
[Equation] [  [Equation] is nondecreasing in  , we say that µ 1 is more strongly aligned than µ 2 and write µ 1  µ 2 .
Comparing against the case of isotropic weight distribution, µ ; , we say µ 1 is aligned when µ 1  µ ; and anti-aligned when µ 1  µ ; .
The parameter ✓ of the d -scale model in Eq.
(11) provides a quantitative measure of alignment under Deﬁnition 2.1, formalizing the intuition given in Section 2.3.
As we will see in Section 3.3, the asymptotic generalization performance of random feature regression is strictly ordered under this deﬁnition of alignment strength.
M AIN RESULTS
Our main result characterizes the high-dimensional asymptotic limits of the test error, bias, and variance of the nonlinear random feature model of Section 2.
Before stating the result, we introduce [Equation] some additional notation that captures the effect of the nonlinearity  ,
[Equation]
[Equation] where, as above, s = lim
[Equation]
µ [  ] .
The constant !
0 is a measure of the degree of nonlinearity, with !
=
corresponding to linear activation functions (see Lemma B.1).
Analogously to Anonymous (2021), we also introduce two functionals of µ , which capture all the relevant spectral information needed for describing the test loss,
[Equation]
3. F ORMULAS FOR ASYMPTOTIC BIAS , VARIANCE , AND TEST ERROR
Theorem 3.1.
Under Assumptions 1-3, as n 0 , n 1 , m !
1 with  = n 0 /m and = n 0 /n 1 held constant, the training error E ⌃ train tends toward the value of E µ train where
[Equation]
and the bias, variance, and test error B ⌃  , V ⌃  , E ⌃  tend toward B µ , V µ , E µ with
[Equation]
and V µ = E
[Equation]  B [Equation] where ⇢ , ! are deﬁned in (12) , I a,b , I  a,b are deﬁned in (13) , and x is the unique nonnegative real root of x = 1  ⌧ 1 !
+ I 1 , 1 with ⌧ 1 = p (   ) 2 +4 x  / ⇢ +   2  .
Remark 3.1.
As noted in (Adlam & Pennington, 2020a; Hastie et al., 2019) for isotropic covariates, the test error is simply related to the training error through the generalized cross-validation (GCV) formula (Golub et al., 1979).
Curiously, because ⌧ 1 is independent of  , it follows that the test error depends on  entirely through its effect on the training error; however, the bias-variance decomposition retains explicit  dependence, even conditional on the training error.
The results in Theorem 3.1 depend on a single scalar self-consistent equation for x , x = 1  ⌧ 1 !
+ I 1 , 1 , which is in fact the same as the one appearing in Anonymous (2021), and which signiﬁcantly simpliﬁes the expressions relative to those recently obtained using the replica method (d’Ascoli et al., 2021).
Owing to its simplicity, this equation [-for x and the deﬁniton of I  in (13). As a consistency check, taking  ( x ) = x and ! 
-]admits straightforward analysis that we pursue in the Appendix, yielding numerous inequalities and bounds that ultimately prove the propositions presented throughout this section.
We will on occasionally refer to the ridgeless limit of Theorem 3.1, which follows by carefully expanding the result for small  and is {+given in Corollary G.1. Finally, as a consistency check, taking  ( x ) = x and ! 
+}
in Theorem 3.1 yields an expression for the test error, bias, and variance for linear regression that agrees with the results of (Wu & Xu, 2020;
Mel & Ganguli, 2021) (see Section E).
3. T HE BENEFITS OF OVERPARAMETERIZATION
While there is abundant empirical evidence that overparameterization can improve the generalization performance of practical models (see e.g. (Zhang et al., 2016)), rigorous explanations for this behavior have been offered solely in the setting of isotropic covariates and target functions.
It is therefore natural to wonder whether the beneﬁts of overparmeterization persist in the presence of anisotropy, and indeed recent theoretical work has provided numerical evidence hinting that overparameterization is beneﬁcial (d’Ascoli et al., 2021).
The following two results prove this to be the case.
First, we show that the bias decreases (or stays constant) in response to an increase in the number of random features, regardless of any anisotropy in the covariates or target function weights.
Proposition 3.1.
In the setting of Theorem 3.1, the bias B µ is a nonincreasing function of overparameterization ratio  / .
The same is true for the variance in the overparameterized regime.
Note that our proof requires the ridgeless setting, but numerical simulations suggest that the result may hold generally (see Fig. 1).
Proposition 3.2.
In the ridgeless limit and in the overparameterized regime ( <  ), the variance V µ is a nonincreasing function of overparameterization ratio  / .
In Fig.
1, Propositions 3.1 and 3.2 are illustrated.
Moving left to right in panels (a)-(d) leads to models with more parameters.
The monotonicity of the bias across the whole range of parameterization is evident, as is the monotonicity of the variance in the overparmaeterized regime.
3. W EIGHT - DATA ALIGNMENT REDUCES THE BIAS AND TEST ERROR
A number of recent works have conﬁrmed the intuition that generalization performance should improve if the weights of the target function align with the large eigendirections of the training covariance, with formal theoretical arguments in the case of linear regression (Hastie et al., 2019; Mel & Ganguli, 2021), and with informal numerical simulations for random feature models (d’Ascoli et al., 2021).
The following result proves that this informal observation holds in full generality, not only for the total test error, but for the bias and the bias-to-variance ratio as well.
Proposition 3.3.
Let µ 1 , µ 2 be two LJSDs such that µ 1  µ 2 (see Deﬁnition 2.1).
Then B µ 1  B µ 2 , E µ 1  E µ 2 , and B µ 1 /V µ 1  B µ 2 /V µ 2 .
We illustrate Proposition 3.3 in Fig.
1 for the 2 -scale LJSD of Eq.
Following vertical lines upward in panels (a-c) or the x-axis left-to-right in (e,f) yields stronger alignment and a corresponding decrease in the bias and test error.
Panel (f) shows the alignment-dependence of the bias, variance, and test error when normalized by the training error.
As suggested by Remark 3.1, the normalized test error is independent of the alignment strength, and as intuition would suggest and as predicted by Proposition 3.3, the normalized bias decreases in response to stronger alignment.
3. A NISOTROPY INDUCES STRUCTURED LEARNING CURVES
Strong anisotropy has been observed to induce structured learning curves in the context of linear regression (Mel & Ganguli (2021)).
It is natural to wonder whether these effects generalize to the case of nonlinear random features.
Figure 2 shows that this is indeed the case for sample-wise learning curves, where   1 = m/n 0 is varied while = n 0 /n 1 is held ﬁxed.
Horizontal slices through Fig.
2 (a) exhibit sample-wise multiple descent.
Fig.
2 (b) and (c) show that these peaks are associated with phase transitions at critical values of   1 where the spectrum of the kernel matrix 1 n 1 F T F acquires a new component.
Increased nonlinearity ! attenuates this effect.
In contrast, the following proposition shows that parameter-wise multiple descent does not occur even in the presence of strong anisotropy, as long as µ is aligned .
Proposition 3.4.
If µ is aligned (see Deﬁnition 2.1), then, in the ridgeless limit, the test error has at most two interior critical points as a function of the overparameterization ratio  / .
Remark 3.2.
Since k -fold descent requires at least k critical points, we conclude that multiple ( k > 2 ) descent does not occur, even in the presence of covariate anisotropy, so long as µ is aligned.
Fig.
3(a,b) shows parameter-wise learning curves for various values of ✓ in the 2 - and 3 -scale models.
As predicted by Proposition 3.4, the curves for aligned LJSDs ( ✓  0 ) have at most two critical points (not shown).
In fact, even for anti-aligned LJSDs ( ✓ < 0 ), most curves also have at most two critical points; however, for small enough ✓ , an additional critical point emerges, and indeed the upper-most traces in Fig.
3(b) showcase a second peak in the learning curves.
Even though parameter-wise multiple descent is not possible for aligned LJSDs, strong anisotropy can nevertheless induce other signatures, including wide plateaus and steep cliffs.
To understand the origin of these effects, consider the ridgeless limit, for which the self-consistent equation for x can be written as,
[Equation]
For small ! , the left side becomes relatively insensitive to changes in x whenever   ⌧  /x ⌧  + for some wide spectral gap with boundary (   ,  + ) .
It follows that, in the underparameterized regime (  < ), the derivative @
(  / ) / @
x is small or, equivalently, @
x/ @
(  / ) is large.
This strong sensitivity of x to changes in the overparameterization ratio  / induces similarly strong changes to the error because @ E µ / @
x is bounded.
As a result, the parameter-wise learning curves exhibit sharp downward cliffs in the underparameterized regime for sufﬁciently small !
and sufﬁciently large spectral gaps  + /   .
We make this argument more precise in Appendix F.
Fig. 3(a) illustrates these cliffs in the context of the d = 2 - and 3 -scale LJSD.
Along horizontal slices, the error turns sharply downward as a function of n 1 /  / at integer multiples of n 0 /d .
As can be seen in Fig. 3(b), when the alignment is increased and  overlaps more with large  s, the ﬁrst error cliff, corresponding to the largest scales in the data, strengthens relative to others (cliff at  / = 1 /in
the 2 -scale model and  / = 1 / 3 in the 3 -scale model).
Fig. 3(c) shows that, unlike the peak at the interpolation threshold, cliffs persist under optimal regularization.
Finally, Fig. 3(d) illustrates how steep decreases in the error are associated with the appearance of a new component in the spectrum of the kernel matrix at critical values of  / .
C ONCLUSIONS
We presented an asymptotically exact calculation of the test error, bias, and variance for random feature kernel regression in the presence of anisotropic covariates and target function weights.
After deﬁning a partial order over weight-data alignments, we proved that stronger alignment decreases the bias and test error.
We also proved that the beneﬁts of overparameterization persist in the anisotropic setting, and that weight-data alignment guarantees that parameter-wise multiple descent is limited to double descent.
In contrast, we demonstrated that anisotropy can induce sample-wise multiple descent, and parameter-wise cliffs, and that their structure is dictated by the eigenstructure of the data covariance.
Future directions include extending our results to the non-asymptotic regime, accommodating feature learning and more general neural network models, and investigating the impact of anisotropy for other loss functions.