A NISOTROPIC R ANDOM F EATURE R EGRESSION IN H IGH D IMENSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
In contrast to standard statistical wisdom, modern learning algorithms typically ﬁnd their best performance in the overparameterized regime in which the model has many more parameters than needed to ﬁt the training data.
A growing number of recent works have shown that random feature models can offer a detailed theoretical explanation for this unexpected behavior, but typically these analyses have utilized isotropic distributional assumptions on the underlying data generation process, thereby failing to provide a realistic characterization of real-world models that are designed to identify and harness the structure in natural data.
In this work, we examine the high-dimensional asymptotics of random feature regression in the presence of structured data, allowing for arbitrary input correlations and arbitrary alignment between the data and the weights of the target function.
We deﬁne a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment.
We also clarify several previous observations in the literature by distinguishing the behavior of the samplewise and parameter-wise learning curves, ﬁnding that sample-wise multiple descent can occur at scales dictated by the eigenstructure of the data covariance, but that parameter-wise multiple descent is limited to double descent, although strong anisotropy can induce additional signatures such as wide plateaus and steep cliffs.
Finally, these signatures are related to phase transitions in the spectrum of the feature kernel matrix, and unlike the double descent peak, persist even under optimal regularization.
I NTRODUCTION
While deep learning races ahead with its impressive list of practical successes, it has often done so by building larger models and training on ever larger datasets.
For example, recent large language models leverage billions (Brown et al., 2020; Raffel et al., 2019) or even trillions (Fedus et al., 2021) of parameters.
As the state-of-the-art pushes the computational frontier, model development, experimentation, and exploration become increasingly costly, both in terms of the computational resources and person-hours.
In order to facilitate the development of improved models, it is becoming ever more necessary to develop theoretical models that can help guide model development in these high-dimensional scenarios.
Indeed, our theoretical understanding is so limited that even linear regression, the most basic machine learning method of all, is still yielding surprises in high dimensions (Hastie et al., 2019).
While linear regression can capture many salient features of high-dimensional data , it is not a reasonable model for high-dimensional models .
In particular, there is no natural data-independent notion of model size: the number of parameters in the model is intimately tied (usually equal) to the dimensionality of the datapoints.
This limitation means that linear regression may not accurately capture the effect of overparameterization, which appears to be a crucial ingredient in modern deep learning models (Zhang et al., 2016).
Beyond linear regression, the next simplest model is arguably the random feature model of Rahimi et al.
(2007), which turns out to be tractable in high dimensions while also allowing for a natural data-independent lever to vary model size – the number of random features.
Random feature models have also gained independent interest through their relationship to neural kernels and Gaussian processes (Neal, 1996; Lee et al., 2017; Novak et al., 2018; Lee et al., 2019;
Jacot et al., 2018).
While recent works have examined the asymptotic performance of random feature models, they have done so in restricted setting of isotropic covariates and target functions (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a;b).
Natural data tends to exhibit strong anisotropy and, as such, it is unclear to what extent the conclusions drawn from the isotropic analysis carry over to practical settings.
In this work, we study high-dimensional random feature regression for anisotropic data and anisotropic target functions, generalizing prior work in this direction that was restricted to linear regression.
1. C ONTRIBUTIONS
Our primary contributions are to:
Derive theoretical predictions for the test error, bias, and variance of random feature regression for anisotropic Gaussian covariates under general linear target functions in the high-dimensional asymptotic limit (Section 3.1); 2.
Prove that overparameterization reduces the total test error, bias, and variance, even in the presence of anisotropy (Section 3.2); 3.
Deﬁne a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment (Deﬁnition 2.1, Section 3.3); 4.
Demonstrate that anisotropy can induce sample-wise multiple descent, but prove that parameter-wise multiple descent is limited to double descent (Section 3.4); 5.
Provide a theoretical explanation for steep cliffs observed in parameter-wise learning curves that persist under optimal regularization, and relate them to phase transitions in the spectrum of the feature kernel matrix (Section 3.4).
1. R ELATED WORK
Our analysis builds on a series of works that have studied the exact high-dimensional asymptotics of the test error for a growing class of model families and data distributions.
Early work in this direction examined minimum-norm interpolated least squares and ridge regression in the high-dimensional random design setting (Belkin et al., 2019; Dobriban et al., 2018; Hastie et al., 2019), ﬁnding surprising phenomena such as double descent and a negative optimal ridge constant (Kobak et al., 2020).
These analyses were generalized in (Richards et al., 2021; Wu & Xu, 2020; Hastie et al., 2019) to allow for anisotropic covariance matrices and weights that generate the targets.
This anisotropic conﬁguration was studied in further detail in (Mel & Ganguli, 2021), uncovering nuanced behavior such as sample-wise multiple descent and a sequence of phase transitions in the performance and optimal regularization.
While linear regression can provide valuable insight into generalization performance in high dimensions, it is not sufﬁciently complex to faithfully reproduce the features of non-linear models, such as the effects of overparameterization.
Random feature models, on the other hand, can capture many such features while still maintaining analytical tractability.
Recent results have developed a detailed understanding of isotropic random feature regression in the high-dimensional asymptotic limit, providing a precise characterization of peaks in the test error and the beneﬁts of overparameterization (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a).
The origin of these peaks was explained by means of a ﬁne-grained decomposition of the bias and variance in (Adlam & Pennington, 2020b; Lin & Dobriban, 2020).
While most prior work examining the high-dimensional asymptotics of random feature models have leveraged the isotropy of the covariate distribution, this assumption was recently relaxed in (Liao et al., 2020) in the context of random
Fourier feature models, and in (Anonymous, 2021) in the study of covariate shift.
These works did not pursue a detailed investigation of the anisotropy itself, nor did they analyze the interaction with anistropic target weights as we do here.
From the technical perspective, our analytic techniques build upon these works and a series of recent results stemming from the literature on random matrix theory and operator-valued free probability (Pennington & Worah [Table] [Table] [Table]
P RELIMINARIES 2. P ROBLEM SETUP AND NOTATION
Following numerous prior works on random feature regression in high dimensions (Hastie et al., 2019; Mei & Montanari, 2019; Adlam & Pennington, 2020a; Adlam et al., 2019; Anonymous, 2021), we focus on the proportional asymptotics in which the number of samples m , input feature dimension n 0 , and hidden layer size n 1 all tend to inﬁnity at the same rate, with  := n 0 /m and := n 0 /nheld constant.
We deﬁne the overparameterization ratio as  / = n 1 m , which characterizes the normalized complexity of the random feature model.
We consider the task of learning an unknown function from m independent samples ( x i , y i )
R n 0 ⇥ R , , i = 1 , . . .
, m , where the covariates are Gaussian, x i ⇠ N (0 , ⌃ ) , and the labels are generated by a linear function,
[Equation]
where the additive noise ✏ i ⇠ N (0 ,  2 ✏ ) is present on the training points only.
We emphasize that assuming a linear target function sacriﬁces little in terms of generality – for a broad class of nonlinear target functions, there exists a linear function whose behavior is entirely equivalent in the proportional asymptotics (Mei & Montanari, 2019; Adlam & Pennington, 2020a).
In contrast to prior work on high-dimensional random feature regression, we do not assume that the coefﬁcient vector  is isotropic; instead, we allow for an arbitrary weight covariance, namely  ⇠ N (0 , ⌃  ) . .
The predictive model is deﬁned by kernel ridge regression using unstructured random feature maps (Rahimi et al., 2007) for which the random features are given by a single-layer neural network with random weights.
In particular, for a training dataset X =
[ x 1 , . .
, x m ] and a test point x , the random features are given by,
[Equation]
where W 2 R n 1 ⇥ n 0 is a random weight matrix with i.i.d.
standard Gaussian entries, and  ( · ) : R !
R is an activation function that is applied elementwise.
The induced kernel is then given by,
[Equation]
and the model’s predictions on a point x are
[Equation]
where Y :=
[ y ( x 1 ) , . . .
, y ( x m )] , K :=
K ( X, X ) +  I m , K x := K ( X, x ) , and   0 .
Owing to the implicit regularization effect of the nonlinear feature maps (Bartlett et al., 2021), in low noise settings the optimal value of  can sometimes be negative (Kobak et al., 2020; Wu & Xu, 2020).
For simplicity, we nevertheless make the standard assumption that   0 , though we emphasize our techniques readily accommodate negative values.
The training error is deﬁned as,
[Equation]
[Equation]
and the test error (without label noise on the test point) can be written as
[Equation]
[Equation]
where the outermost expectation over  has been suppressed since the quantity concentrates sharply around its mean and the inner expectation is computed over all the randomness from training, i.e. W ,
X , and ✏ .
The test error can be decomposed into its bias and variance components as
[Equation]
Following (Adlam & Pennington, 2020b), the bias and variance are also computed with respect to all the randomness from training, i.e. W , X , ✏ .
As noted by (Hastie et al., 2019; Wu & Xu, 2020; Mel & Ganguli, 2021), the test error of linear regression depends on the geometry of ( ⌃ , ⌃  ) (or ( ⌃ ,  ) in the case of nonrandom  ), and we will ﬁnd the same to be true for random feature regression.
As such, we decompose the training and  covariance matrices into eigenbases as ⌃ =
P n 0 i =1  i v i v i > and ⌃  =
P n 0 i =1   i v  i v  i > , where the eigenvalues are in nondecreasing magnitude, i.e.  1   2  . . .
  n 0 and   1    2  . . .
   n 0 .
We then deﬁne the overlap coefﬁcients as
[Equation]
to measure the alignment of ⌃  with the i th eigendirection of ⌃ .
In particular, q i is the induced norm of v i with respect to ⌃  .
Finally, for a matrix A 2 R n ⇥ n , we use ¯tr to denote the dimension-normalized trace, ¯tr( A ) =
n tr( A ) , and k A k 1 to denote its spectral norm.
We write  x to denote the Dirac delta function centered at x .
2. A SSUMPTIONS
In order to deﬁne the limiting behavior of the test error in Eq.
(6), it is necessary to impose some regularity conditions on ⌃ and ⌃  .
As in (Wu & Xu, 2020; Anonymous, 2021), it is not sufﬁcient to consider the limiting spectra of these matrices individually; because their eigenspaces can align, they must be considered jointly.
It is convenient to do this in an eigenbasis of ⌃ .
Assumption 1.
We deﬁne the joint spectral distribution (JSD) as
[Equation]
and assume it converges in distribution to some µ , a distribution on R 2+ as n 0 ! 1 .
We refer to µ as the limiting joint spectral distribution (LJSD), and emphasize that this deﬁnes the relevant limiting properties of the covariate and weight distributions 1 .
Additionally, we require that lim sup n 0 max( k ⌃ k 1 , k ⌃
[Equation]
1 )  C for a constant C .
Often we use (  , q ) for random variables sampled jointly from µ and denote the marginal of  under µ with µ data .
Since the conditional expectation E [ q |  ] is an important object in our study, we assume the following for simplicity.
Assumption 2.
µ is either absolutely continuous or a ﬁnite sum of delta masses and the expectations of  and q are ﬁnite.
Moreover, E µ [  q ] = 1 .
The normalization condition E µ [  q ] = 1 is not necessary for our analysis, but, as in (Mel & Ganguli, 2021), it enforces consistent signal-to-noise ratios across different target functions and thereby facilitates meaningful comparisons of the test error.
When the eigenspaces of ⌃ and ⌃  are aligned and q i =   i =
(  i ) for some smooth function  , the support of the LJSD degenerates.
Here, Assumption 1 is essentially equivalent to assuming the spectral distribution of ⌃ converges in distribution to some µ data , which is a standard assumption in the regression literature (Dobriban et al., 2018; Mei & Montanari, 2019).
One important case is when ⌃  = I n 0 , i.e.  :  ! 1 , in which case the LJSD degenerates to µ ; deﬁned by
[Equation]
As our analysis will eventually take place in the high-dimensional limit, we further deﬁne the asymptotic covariance scale as s = lim n 0 !
1 ¯tr( ⌃ ) =
E µ [  ] under the limiting behavior speciﬁed in
Assumption 1.
Throughout this paper, we also enforce the following standard regularity assumptions on the activation functions to ensure the existence of the moments and derivatives we compute.
Assumption 3.
The activation function  : R !
R is assumed to be differentiable almost everywhere.
We assume there exists a universal constant C such that, |  ( x ) | , |  0 ( x ) | =
O (exp( Cx )) .
2. A SIMPLE FAMILY OF ANISOTROPIC DISTRIBUTIONS
Because the above assumptions allow for such a general class of covariance structures, it will prove useful to consider our results in the context of a simple family of distributions that readily admits straightforward interpretation.
To illustrate the effects of multiple scales in the input data, we deﬁne the following d -scale LJSDs: for d 2 N and real 0 <
↵  1 and ✓ ,
[Equation]
, ✓
[Equation]
where C :=
[Equation]
the normalization conditions
[Equation]
These simple covariance models capture the hierarchy of scales that often characterize the structure of natural datasets.
Note that by setting ↵ = 1 and s = 1 , the distribution reduces to the isotropic case with identity covariance.
For the nontrivial setting in which ↵ < 1 , the data exhibits a sequence of d distinct scales where each scale is a factor ↵ smaller than the previous one. The exponent ✓ parameterizes the strength of the weight-data alignment in an intuitive way.
When ✓ = 0 , there is no alignment, and ⌃  is proportional to the identity.
When ✓ > 0 , there is strong alignment, as the large eigendirections of the training distribution correspond to large eigendirections of ⌃  .
When ✓ < 0 , there is anti-alignment, as the large eigendirections of the training distribution correspond to small eigendirections of ⌃  .
2. D EFINITION OF THE STRENGTH OF WEIGHT - DATA ALIGNMENT
As the preceding discussion has suggested and as we will see explicitly in the following section, the LJSD µ captures all of the information about the pair of covariance matrices ( ⌃ , ⌃  ) that is relevant for describing the asymptotic test error, bias, and variance.
As the alignment between ⌃ and ⌃  can vary in strength among the different eigendirections, the concept of alignment is inherently multi-dimensional.
Nevertheless, by requiring strong alignment along the larger eigendirections, it is possible to deﬁne the following natural partial order on the space of possible weight-data alignments.
Deﬁnition 2.1.
Let µ 1 and µ 2 be LJSDs with the same marginal distribution of  .
If the asymptotic overlap coefﬁcients are such that E µ 1 [  q |  ] / E µ 2 [  q |  ] is nondecreasing in  , we say that µ 1 is more strongly aligned than µ 2 and write µ 1  µ 2 .
Comparing against the case of isotropic weight distribution, µ ; , we say µ 1 is aligned when µ 1  µ ; and anti-aligned when µ 1  µ ; .
The parameter ✓ of the d -scale model in Eq.
(11) provides a quantitative measure of alignment under Deﬁnition 2.1, formalizing the intuition given in Section 2.3.
As we will see in Section 3.3, the asymptotic generalization performance of random feature regression is strictly ordered under this deﬁnition of alignment strength.
M AIN RESULTS
Our main result characterizes the high-dimensional asymptotic limits of the test error, bias, and variance of the nonlinear random feature model of Section 2.
Before stating the result, we introduce some additional notation that captures the effect of the nonlinearity  ,
[Equation]
(12) where s is deﬁned above as = lim n 0 !
1 ¯tr( ⌃ ) =
E µ [  ] .
The constant !
0 is a measure of the degree of nonlinearity, with !
=
corresponding to linear activation functions (see Lemma A.1).
We also introduce two functionals of µ , which capture all the relevant spectral information needed for describing the test loss, I
a,b : =  E µ ⇣  a (  + x  )  b ⌘ and I  a,b : =
E µ ⇣ q  a (  + x  )  b ⌘ .
3. F ORMULAS FOR ASYMPTOTIC BIAS , VARIANCE , AND TEST ERROR
Theorem 3.1.
Under Assumptions 1-3, as n 0 , n 1 , m !
1 with  = n 0 /m and = n 0 /n 1 held constant, the training error E ⌃ train tends toward the value of E µ train where
[Equation]
and the bias, variance, and test error B ⌃  , V ⌃  , E ⌃  tend toward B µ , V µ , E µ with
[Equation]
where ⇢ , ! are deﬁned in (12) , I a,b , I  a,b are deﬁned in (13) , and x is the unique nonnegative real root of x = 1  ⌧ 1 !
+ I 1 , 1 with ⌧ 1 = p (   ) 2 +4 x  / ⇢ +   2  .
Remark 3.1.
As noted in (Adlam & Pennington, 2020a; Hastie et al., 2019) for isotropic covariates, the test error is simply related to the training error through the generalized cross-validation (GCV) formula (Golub et al., 1979).
Curiously, because ⌧ 1 is independent of  , it follows that the test error depends on  entirely through its effect on the training error; however, the bias-variance decomposition retains explicit  dependence, even conditional on the training error.
Numerical predictions from Theorem 3.1 can be obtained by ﬁrst solving the self-consistent equation for x by ﬁxed-point iteration, x ! 1  ⌧ 1 !
+ I 1 , 1 , and then plugging the result into the remaining terms.
For the partial derivative in the training error in (14), ⌧ 1 , I  1 , 1 depend implicitly on  through the self-consistent equation for x and the deﬁniton of I  in (13).
As a consistency check, taking  ( x ) = x and !
in Theorem 3.1 yields an expression for the test error, bias, and variance that agrees with the results of (Wu & Xu, 2020; Mel & Ganguli, 2021) (see
Section D).
At times we will ﬁnd it convenient to consider the ridgeless limit of Theorem 3.1.
By carefully expanding x and ⌧ 1 for small  , it is straightforward to obtain the following Corollary.
Corollary 3.1.
In the setting of Theorem 3.1, as the ridge regularization constant  ! 0 , E µ =
B µ + V µ with B µ =  I  1 , 2 and V µ given by
[Equation]
where x is the unique positive real root of x = min(1 ,  / ) !
+
I 1 , 1 .
3. T HE BENEFITS OF OVERPARAMETERIZATION
While there is abundant empirical evidence that overparameterization can improve the generalization performance of practical models (see e.g. (Zhang et al., 2016)), prior theoretical work has only offered explanations for this behavior in the unrealistic setting of isotropic covariates and target functions.
It is therefore natural to wonder whether the beneﬁts of overparmeterization persist in the presence of anisotropy.
The following two results prove that indeed this is the case.
First, we show that the bias decreases (or stays constant) in response to an increase in the number of random features, regardless of any anisotropy in the covariates or target function weights.
Proposition 3.1.
In the setting of Theorem 3.1, the bias B µ is a nonincreasing function of overparameterization ratio  / .
The same is true for the variance in the overparameterized regime.
Note that our proof requires the ridgeless setting (  = 0 ), but numerical simulations suggest that the result may hold without this assumption (see Fig. 1).
Proposition 3.2.
In the setting of Corollary 3.1 and in the overparameterized regime ( <  ), the variance V µ is a nonincreasing function of overparameterization ratio  / .
In Fig.
1, Propositions 3.1 and 3.2 are illustrated.
Moving left to right in panels (a)-(d) leads to models with more parameters.
The monotonicity of the bias across the whole range of parameterization is evident, as is the monotonicity of the variance in the overparmaeterized regime.
3. W EIGHT - DATA ALIGNMENT REDUCES THE BIAS AND TEST ERROR
In the context of high-dimensional linear regression, a number of recent works formalized the intuition that generalization performance should improve if the weights of the target function align with the large eigendirections of the training covariance (Hastie et al., 2019; Mel & Ganguli, 2021).
It is natural to wonder whether this behavior is restricted to linear models of the data, or whether a similar response to alignment might persists more generally.
The following result proves this to be the case not only for the total test error, but for the bias and the bias-to-variance ratio as well.
Proposition 3.3.
Let µ 1 , µ 2 be two LJSDs such that µ 1  µ 2 (see Deﬁnition 2.1).
Then B µ 1  B µ 2 , E µ 1  E µ 2 , and B µ 1 /V µ 1  B µ 2 /V µ 2 .
Proposition 3.3 shows that Deﬁnition 2.1 provides an essentially model-independent condition for comparing the relative difﬁculty of learning two target functions.
We illustrate this monotonicity in Fig.
1 for the 2 -scale LJSD of Eq.
Following vertical lines upward in panels (a-c) or the x-axis left-to-right in (e,f) yields stronger alignment and a corresponding decrease in the bias and test error.
Panel (f) shows the alignment-dependence of the bias, variance, and test error when normalized by the training error.
As suggested by Remark 3.1, the normalized test error is independent of the alignment strength, and as intuition would suggest and as predicted by Proposition 3.3, the normalized bias decreases in response to stronger alignment.
3. A NISOTROPY INDUCES STRUCTURED LEARNING CURVES
Strong anisotropy has been observed to induce structured learning curves in the context of linear regression (Wu & Xu (2020); Mel & Ganguli (2021)).
It is natural to wonder whether these effects generalize to the case of nonlinear random features.
Figure 2 shows that this is indeed the case for sample-wise learning curves, where   1 = m/n 0 is varied while = n 0 /n 1 is held ﬁxed.
Horizontal slices through Fig.
2 (a) exhibit sample-wise multiple descent.
Fig.
2 (b) and (c) show that these peaks are associated with phase transitions at critical values of   1 where the spectrum of the kernel matrix 1 n 1 F T F acquires a new component.
Increased nonlinearity ! attenuates this effect.
In contrast, the following proposition shows that parameter-wise multiple descent does not occur even in the presence of strong anisotropy, as long as µ is aligned .
Proposition 3.4.
If µ is aligned (see Deﬁnition 2.1), then, in the setting of Corollary 3.1, the test error has at most two interior critical points as a function of the overparameterization ratio  / .
Remark 3.2.
Since k -fold descent requires at least k critical points, we conclude that multiple ( k > 2 ) descent does not occur, even in the presence of covariate anisotropy, so long as µ is aligned.
Fig.
3(a,b) shows parameter-wise learning curves for various values of ✓ in the 2 - and 3 -scale models.
As predicted by Proposition 3.4, the curves for aligned LJSDs ( ✓  0 ) have at most two critical points (not shown).
In fact, even for anti-aligned LJSDs ( ✓ < 0 ), most curves also have at most two critical points; however, for small enough ✓ , an additional critical point emerges, and indeed the upper-most traces in Fig.
3(b) showcase a second peak in the learning curves.
Even though parameter-wise multiple descent is not possible for aligned LJSDs, strong anisotropy can nevertheless induce other signatures, including wide plateaus and steep cliffs.
To understand the origin of these effects, consider the ridgeless limit, for which the self-consistent equation for x can be written as,
[Equation]
For small ! the left side becomes relatively insensitive to changes in x whenever   ⌧  /x ⌧  + for some wide spectral gap with boundary (   ,  + ) .
It follows that, in the underparameterized regime (  < ) the derivative @
(  / ) / @
x is small, or, equivalently, @
x/ @
(  / ) is large.
This strong sensitivity of x to changes in the overparameterization ratio  / induces similarly strong changes to the error because @ E µ / @
x is bounded.
As a result, the parameter-wise learning curves exhibit sharp downward cliffs in the underparameterized regime for sufﬁciently small !
and sufﬁciently large spectral gaps  + /   .
We make this argument more precise in Appendix E.
Fig. 3(a) illustrates these cliffs in the context of the d = 2 - and 3 -scale LJSD.
Along horizontal slices, the error turns sharply downward as a function of n 1 /  / at integer multiples of n 0 /d .
As can be seen in Fig. 3(b), when the alignment is increased and  overlaps more with large  s, the ﬁrst error cliff, corresponding to the largest scales in the data, strengthens relative to others (cliff at  / = 1 /in
the 2 -scale model and  / = 1 / 3 in the 3 -scale model).
Fig. 3(c) shows that, unlike the peak at the interpolation threshold, cliffs persist under optimal regularization.
Finally, Fig. 3(d) illustrates how steep decreases in the error are associated with the appearance of a new component in the spectrum of the kernel matrix at critical values of  / .
C ONCLUSIONS
We presented an asymptotically exact calculation of the test error, bias, and variance for random feature kernel regression in the presence of anisotropic covariates and target function weights.
After deﬁning a partial order over weight-data alignments, we proved that stronger alignment decreases the bias and test error.
We also proved that the beneﬁts of overparameterization persists in the anisotropic setting, and that weight-data alignment guarantees that parameter-wise multiple descent is limited to double descent.
In contrast, we demonstrated that anisotropy can induce sample-wise multiple descent, and parameter-wise cliffs, and that their structure is dictated by the eigenstructure of the data covariance.
Future directions include extending our results to the non-asymptotic regime, accommodating feature learning and more general neural network models, and investigating the impact of anisotropy for other loss functions.