A R ANDOM M ATRIX P ERSPECTIVE ON M IXTURES OF N ONLINEARITIES IN H IGH D IMENSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions.
While this paradigm has inspired signiﬁcant research on the properties of large networks, relatively little work has been devoted to the fact that these networks are often used to model large complex datasets, which may themselves contain millions or even billions of constraints.
In this work, we focus on this high-dimensional regime in which both the dataset size and the number of features tend to inﬁnity.
We analyze the performance of a simple regression model trained on the random features
F = f ( WX + B ) for a random weight matrix W and random bias vector B , obtaining an exact formula for the asymptotic training error on a noisy autoencoding task.
The role of the bias can be understood as parameterizing a distribution over activation functions, and our analysis directly generalizes to such distributions, even those not expressible with a traditional additive bias.
Intriguingly, we ﬁnd that a mixture of nonlinearities can outperform the best single nonlinearity on the noisy autoecndoing task, suggesting that mixtures of nonlinearities might be useful for approximate kernel methods or neural network architecture design.
I NTRODUCTION
It is undeniable that in recent years deep learning systems have found widespread success in their applications to a diverse and ever-expanding set of domains.
The foundational results on many tasks such as image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and machine translation (Wu et al., 2016), have begun to make their way into higher-level products that people interact with and rely upon in their daily lives.
Whether these products generate a medical diagnosis, a navigation decision, or some other important output, it is crucial to understand the inner-workings of the deep learning algorithms that generate them.
Unfortunately, our theoretical understanding of these deep learning algorithms continues to lag behind their impressive practical successes.
One main challenge in building a fuller understanding stems from the fact that deep neural networks are complex nonlinear functions that employ millions or even billions (Shazeer et al., 2017) of parameters.
Traditional wisdom would suggest that to this parameter complexity corresponds to an optimization difﬁculty.
Recent work, however, suggests that as the width of a network’s hidden layers becomes large, the loss function simpliﬁes and a theoretical analysis becomes tractable (Jacot et al., 2018; Mei et al., 2018; Chizat & Bach, 2018a; Mei et al., 2019; Rotskoff et al., 2019; Rotskoff & Vanden-Eijnden, 2018).
In some scenarios, the simpliﬁcation is such that throughout training the parameters of the model stay within an inﬁnitesimal radius of their initial (random) values, implying that much about neural network training can be understood by studying the random initialization (Jacot et al., 2018; Chizat & Bach, 2018b; Lee et al., 2019).
Another main challenge in building a rich understanding of deep learning systems stems from the fact that they are often trained on very large, complex datasets: even if the models themselves are very large, they may not be large in comparison to the number of constraints they are designed to satisfy.
Indeed, many important phenomena may become apparent only by examining the high-dimensional regime in which the dataset size and width are both large and of the same order.
In this work, we focus on the high-dimensional regime and analyze the performance of a regression model trained on the random features F = f ( WX + B ) for a random weight matrix W and random bias vector B .
We obtain an exact formula for the training error on a noisy autoencoding task in the limit that the width and dataset size both go to inﬁnity.
The result is determined by the resolvent of the kernel matrix F t F , whose properties we analyze via the resolvent method from random matrix theory.
Our analysis also provides an exact formula for the eigenvalue density of the kernel matrix, which may be of independent interest since it provides a characterization for how spectral properties of the data covariance matrix propagate through neural network layers at initialization.
1. O UR CONTRIBUTIONS
The main contribution of our work is an exact characterization of the training error of a ridgeregularized random feature regression model on a noisy autoencoder task in the high-dimensional regime.
This is one of the ﬁrst non-trivial models to be solved exactly in the joint limit of large data and large width and provides an interesting testing ground in which to analyze this regime.
Some of our additional contributions include,
• An exact characterization of the spectral density of the randm feature matrix F = f ( WX + B ) , extending prior results of Pennington & Worah (2017) to non-Gaussian data distributions and to non-zero bias distributions.
• One interpretation of the random additive bias is that it induces a distribution of activation functions parameterized by B, i.e. f ( Z ; B ) :
= f ( Z + B ) .
Our analysis trivially extends to any distribution of activation functions f ( · ; B ) parameterized by B .
• We show that there exists a non-trivial distribution over activation functions that outperforms the best possible single activation function on a noisy autoencoding task.
• Our method of proof introduces a surrogate “linearization" of F , F lin , that possesses the same spectral information as F .
F lin and its properties are likely to be of further interest and utility in analyzing neural networks in high dimensions.
1. R ELATED W ORK
Neural networks have been studied from the perspective of high-dimensional statistics in a number of recent works.
Most prior work has focused on the bias-free case.
Pennington & Worah (2017) studied the spectrum of the activation matrix f ( WX ) for iid Gaussian data and derived an analytic expression for the training error of a ridge-regularized random feature model trained on pure noise.
It is natural to consider incorporating biases by appending a constant feature 1 to X .
Unfortunately, this leads to biases that are the same order as the weights, and so the effect disappears in the large dataset limit.
Moreover, this modiﬁcation on the data violates the assumptions of Pennington &
Worah (2017).
Hastie et al.
(2019) study ridgeless interpolation in high-dimensional interpolation for linear features as well as nonlinear random features of iid Gaussian data.
Louart et al.
(2018) derived a deterministic equivalent for the resolvent of the kernel matrix F t F which allowed for a characterization of the asymptotic training and test performance of linear ridge regression of random feature models.
Other work has investigated learning dynamics and generalization in the high-dimensional regime (Liao & Couillet, 2018a; Lampinen & Ganguli, 2018; Advani & Ganguli, 2016; Advani &
Saxe, 2017) as well as the spectra of more complicated objects such as the Hessian (Pennington &
Bahri, 2017) and Fisher information matrix (Pennington & Worah, 2018).
From the mathematical perspective, random matrix theory provides natural tools (Silverstein & Bai, 1995a) for analyzing the behavior of neural networks in the high-dimensional regime.
Liao & Couillet (2018b) examined spectra for data drawn from Gaussian mixture models; see also (El Karoui, 2010) on the spectra of random kernel matrices.
P RELIMINARIES
Consider a dataset X ∈ R n 0 × m and the random feature matrix,
F = f ( WX ; B ) , generated by a single hidden-layer network with iid Gaussian weights W ∈ R n 1 × n
( W ak ∼
N (0 , σ 2 W /n 0 ) ), activation function f , and biases B = b 1 mT ∈ R n 1 × m (for b ∈ R n 1 ).
We regard the second argument of f as parametrizing (continuously or discretely) an ensemble of activation functions.
We refer to B (or b ) as the bias , in reference to the important special case f ( WX +
B ) (additive bias).
In general, however, we only assume the parameters b a are such that the measure 1 n 1 (cid:80) a δ b a converges in distribution to some limiting distribution µ B , effecting an arbitrary distribution over activation functions.
We assume that E | f ( N ; b ) | k for N ∼ N (0 , σ ) is ﬁnite for all 1 ≤ k ≤ 3 , σ > 0 , and b ∈ support ( µ B ) .
When µ B is a single Dirac mass at location b 0 , the activation function can be written as f ( WX ; B ) =
f ( WX ; b 0 ) =
g ( WX ) for some single-argument function g (for an additive bias, g ( WX )
=
f ( WX + b 0 ) ).
When this is the case, we say the model has a single activation function , as opposed to a mixture or distribution of activation functions.
The quantity of interest for the investigations below is the kernel matrix 1 n 1 F t F , and in particular its resolvent ,
[Equation]
As we review in Sec. 4, the optimal regression coefﬁcients of a linear model on the random features F is a simple function of this resolvent.
The high-dimensional regime that we study is the one in which the dataset size m , feature dimensionality n 0 , and hidden layer width n 1 all go to inﬁnity at the same rate.
In particular, as is standard in the random matrix literature, we assume that we can parameterize the limit in terms of the dataset size m in such a way that there exist two positive constants,
[Equation]
Note that the resolvent is a random matrix, but as m grows large, its normalized trace becomes a deterministic quantity.
In the limit that m → ∞ , this quantity is known as the Stieltjes transform 1 ,
[Equation]
Together with an auxiliary transform ˜ m ( z ) , deﬁned below, these deterministic quantities completely characterize the asymptotic training error of kernel ridge regression on a noisy autoencoder task in this high-dimensional regime.
The Stieltjes transform frequently arises in random matrix methods as a way to encode the spectra of matrices.
In particular, if λ i are the eigenvalues of 1 n 1 F t F and the empirical distribution of eigenvalues converges in distribution to some deterministic limiting density as m → ∞ ,
[Equation]
then (with appropriate technical assumptions), the limiting spectral density itself can be recovered from the Stieltjes transform m ( z ) via the inversion formula,
[Equation]
The Stieltjes transform then substitutes convergence in distribution for pointwise convergence for all z such that (cid:61) z > 0 .
2. M ETHODS FOR COMPUTING THE S TIELTJES TRANSFORM
We brieﬂy review two standard methods for computing the Stieltjes transform m ( z ) , the resolvent method and the moments method .
The resolvent method is an approach for computing the Stieltes transform based on the application of the Schur complement formula to the resolvent itself (or to a closely-related block matrix).
Intuitively, as the matrix size becomes large, the minors of the matrix are similar in distribution to the larger matrix, and, moreover, the Cauchy interlacing theorem guarantees that their Stieltjes transforms are
3.0      d e n s i t y
Spectrum for Gaussian biases 0.
Spectrum for Bernoulli biases SCE empirical close as well.
This allows for the derivation of a self-consistent equation (SCE) in which the Stieltjes transform appears on the left-hand side as the trace of the resolvent, and on the right-hand side as the trace of one of its minors.
The moments method is more combinatorial in nature and involves expanding the resolvent for large z and computing the traces of each term,
[Equation]
The traces themselves are expended out as
[Equation]
[Equation]
where the sum runs over matrix indices a 1 , . .
. , a k , α 1 , . . .
α k .
The essence of the moment method involves analyzing the asymptotic contribution of each term in the sum based on its combinatorial type and the details of F , and resumming the results to obtain m ( z ) .
We refer the reader to (Erdos & Yau, 2017; Tao, 2012) for more details about these methods and additional background on random matrix theory.
R ESULT FOR S TIELTJES TRANSFORM 3. M AIN THEOREM
We make the following assumptions on the data matrix X and bias vector b :
1. 1 n 0 (cid:80) a X aα X aβ =
δ αβ σ 2 X + O (cid:0) 1 / √ n 0 (cid:1) for all α and β 2.
the empirical eigenvalue distribution of 1 n 0 X t X converges in distribution to a measure µ X 3.
1 n 1 (cid:80) n 1 a =1 δ b a → µ B in distribution.
Theorem 1.
Deﬁne σ Z = σ W σ X and resolvent G ( z ) = (cid:16) 1 n 1 F t F − zI (cid:17) − 1 .
Then under the above assumptions and for all z such that (cid:61) z > 0 , the transforms 1 m
[Equation]
converge in probability to the unique solution, m ( z ) and ˜ m ( z ) , of the Eqn. (9) that map C + to C + : m ( z ) = E S ∼ µ
[Equation]
(cid:20) [Equation] z
[Equation]
and ˜
[Equation]
( z )
[Equation]
S ∼
[Equation]
(cid:20) S C
[Equation]
( z ) + SC 1 ( z ) (cid:21) (9) where, C 0 ( z ) : = − z + E B ∼ µ B (cid:20) η ( B ) − ζ ( B ) D ( B ) (cid:21) ,
C 1 ( z ) : = E B ∼ µ B (cid:20) ζ ( B ) D ( B ) (cid:21) , (10) [Figure]
[Equation]
) and where η ( B ) and ζ ( B ) are the Gaussian expectations
[Equation]
The proof is quite involved and is presented in the supplementary material.
The basic idea is to derive a multivariate Gaussian random matrix model with the same correlation structure as F , then derive a self-consistent equation (SCE) using the resolvent method for this linearized version of F , F lin .
Remark 1.
The self-consistent equations consist of two coupled equations involving the Stieltjes transform m ( z ) and an auxiliary object ˜ m ( z ) , (cf.
(Paul & Silverstein, 2009, Eq.
(2))), which we will see in Cor. 1 essentially measures the autoencoding capacity of the network.
Remark 2.
Note that the self-consistent equations contain an expectation over the limiting spectral density of the input data.
While the assumptions on the data matrix X in Thm. 1 are quite general, they may not be optimal.
See Sec. 3.3, where we show strong agreement with empirical data from
MNIST and CIFAR-10 and for a range of synthetic distributions.
This suggests that the theorem may hold for even more general data distributions.
3. A LTERNATE REPRESENTATION AND LIMITING RESULTS
When the data distribution is iid Gaussian, the expectations in Eqn. (9) can be expressed in closed form, though one must be careful to choose the correct branch of the resulting function.
For simplicity and future reference, we focus on the setting where 0 < φ ≤ ψ ≤ 1 , in which case we have the
[Figure]
coupled algebraic equations,
[Equation]
When µ B in Eqn.
(10) is trivial, i.e. a single Dirac mass, then the result should reduce to the single activation function case with F = f ( WX ) , which was studied in (Pennington & Worah, 2017).
Indeed, writing η = E B [ η ( B )] and ζ = E B [ ζ ( b )] for such a distribution, and eliminating ˜ m ( z ) from Eqns.
(13) and (14), we ﬁnd that m ( z ) satisﬁes the following quartic polynomial:
[Equation]
[Equation] which agrees with the result in (Pennington & Worah, 2017) upon identifying m ( z ) = − (1 − φ/ψ ) /z − φ/ψG ( z ) .
3. S PECTRAL DENSITY ESTIMATES
The self-consistent equations in Thm. 1 can be solved numerically by iterating Eqn. (9) until convergence, using numerical integration.
By utilizing the Stieltjes inversion formula, Eqn. (5), we can extract predictions for the limiting eigenvalue density of 1 n 1 F t F .
The results show close agreement with empirical spectral simulations from several interesting practical datasets and synthetic distributions, see Figs. 1 and 2.
R IDGE - REGULARIZED NOISY AUTOENCODER
We consider the problem of kernel ridge regression with random features given by F = f ( WX ; B ) and noisy regression targets given by Y
= AX + (cid:15) for some A ∈ R n 2 × n 0 and independent Gaussian noise (cid:15) ∈ R n 2 × m such that (cid:15) ij ∼ N (0 , σ 2 (cid:15) ) .
As is common in the literature of high-dimensional statistics, we assume an isotropic prior on A such that A t A → n 2 n 0 σ 2 A I as m, n 0 , n 2 → ∞ .
Note that when σ A = 0 , we have the pure memorization setting studied by Pennington & Worah (2017).
Corollary 1.
Let W ∗ 2 be the minimizer for regularized training loss
[Equation]
(16) with random features F = f ( WX ; B ) .
Then the asymptotic training error converges in probability as
[Equation]
In particular, we see that the derivative of the Stieltjes transform m (cid:48) ( − γ ) measures the capacity to learn noisy labels, whereas ˜ m (cid:48) ( − γ ) measures pure autoencoding capacity. See Fig.
3 for a comparison between these theoretical predictions and simulation.
Proof.
The optimal weights for the regularized loss are given by
[Equation]
resulting in training error
[Equation]
Remark 3.
As in (Pennington & Worah, 2017), there is a scaling homogeneity in the E train : an increase in the regularization constant γ can be compensated by a decrease in scale of W 2 , which, in turn, can be compensated by increasing the scale of F , which is equivalent to increasing η ( b ) and ζ ( b ) .
Owing to this homogeneity, we are free to choose a normalization of the activation function for which E b ∼ µ B [ η ( b )] = 1 .
4. N ONLINEAR MIXTURES CAN OUTPERFORM SINGLE NONLINEARITIES
The bias term in our random feature model can be viewed as one way of deﬁning a distribution over activation functions.
The choice of distribution, in general, affects the performance of the model on a given task – as quantiﬁed by the expectations in Eqn.
A key beneﬁt of our model and analytical approach is that it permits nontrivial distributions of nonlinear activation functions.
In this section, we build on our results for the training loss on noisy autoencoder tasks to examine the beneﬁts of utilizing nonlinear mixtures.
The goal here is not to identify “good" mixtures, since this will clearly be a dataset- and architecture-dependent question, nor is it to identify a large performance gap.
Instead, we merely seek to demonstrate a proof-of-principle, namely that there exist non-trivial distributions over nonlinearities that can provably outperform the best possible single nonlinearity.
For this analysis, we consider the simplest possible nontrivial distribution over activation functions: a
Bernoulli mixture of two different functions.
To each of these functions we associate two constants, η and ζ , which derive from Eqn.
(12) but have no B -dependence since each function is a single nonlinearity .
Concretely, let
[Equation]
For the two functions themselves, we utilize (i) a "pure linear" activation function with η = 1 , i.e. the identity function and (ii) a "pure nonlinear" (Hastie et al., 2019) activation function with η = 1 and ζ = 0 .
(The particular purely nonlinear function in (ii) is irrelevant, as our theory predicts and our experiments conﬁrm; see Fig. 4(a)).
To be precise, we deﬁne for p ∼ Bernoulli(p),
[Equation]
19) where g ζ =0 is any function with η = 1 and ζ = 0 (see, e.g., the functions in Fig.
(3) of (Pennington & Worah, 2017)).
The task of computing E train for f p is cumbersome but purely algebraic.
To see how to proceed, notice that the expectations in Eqn.
(10) are simple for f p :
[Equation]
[Equation]
Plugging these equations into Eqns.
(13) and (14), collecting terms and simplifying yields a set of coupled polynomial equations for m ( z ) and ˜ m ( z ) .
Taking the total derivative of these equations with respect to z yields two additional equations which can be solved to express m (cid:48) ( z ) and ˜ m (cid:48) ( z ) in terms of m ( z ) and ˜ m ( z ) .
Combining these results produces a polynomial system whose solution 2 encodes
E train through Eqn.
Fig. 4(a) shows the result of this calculation in solid lines for various values of γ , while the 1 σ error bars show empirical simulations with ﬁnite networks.
The red stars in the ﬁgure show that for many values of γ , the optimal mixture percentage is intermediate, i.e. 0 < p < 1 .
E train ✶✶✶✶ ✶ ✶ ✶ 0.
ζ γ 11 / 21 / 41 / 81 / 161 / 321 /✶✶✶✶ ✶ ✶ ✶ 0.
p
[Figure]
E train
Optimalf Optimal f p 0.001 0.010 0.100 1 10 100 0.
Δ E train 0.
γ Δ E train
[Figure]
The question is, does a non-trivial mixture actually outperform a single nonlinearity?
First, we must understand the performance of the optimal single nonlinearity.
We note that owing to the homogeneity of the the training loss in η , ζ , and γ , we can assume without loss of generality that η = 1 .
Therefore the entire effect of the nonlinearity should be encoded in the single constant ζ . In Fig. 4(a), we plot our theoretical prediction for E train in solid lines and empirical simulations for ﬁnite networks as 1 σ error bars. The activation function used for each simulation is chosen randomly, conditional on the value of ζ . So the good agreement in the left panel of Fig. 4(a) demonstrates not just the correctness of our theoretical result but also the fact that E train depends on the activation function solely through the constant ζ .
The blue stars in this ﬁgure indicate that the optimal single nonlinearity is neither purely linear ( ζ = 1 ), nor purely nonlinear ( ζ = 0 ), but rather something in between.
For this particular problem setup, the performance of the optimal single nonlinearity and the optimal
Bernoulli mixture are rather close, as indicated by the top panel of Fig. 4(b).
However, owing to our precise analytical formulation, we can evaluate the training loss to high precision and observe that there is indeed a difference in performance between the two models, as shown in the bottom panel of Fig. 4(b).
This result establishes that there are some problems for which even the best single nonlinearity is outperformed by a mixture of nonlinearities.
C ONCLUSIONS
In this work we studied the feature matrix F = f
( WX ; B ) where W is a random matrix with iid
Gaussian entries.
Under mild assumptions on X and B , we obtained an exact analytic formula, Eqn.
(9), that characterizes the Stieltjes transform of the spectral density of F .
The result allowed us to describe the exact training loss of a ridge-regularized noisy autoencoder in the high-dimensional, large-dataset limit, providing one of the ﬁrst closed-form solutions to a non-trivial model in this limit.
We found excellent agreement between the asymptotic predictions of Eqn.
(9) and a variety of ﬁnite-dimensional empirical simulations.
We also advanced the interpretation of the bias B as one particular way of parameterizing a distribution of activation functions.
Indeed, our derivations proceed completely unchanged whether this distribution is of the traditional additive form f ( · + B ) or the more general f ( · ; B ) .
By examining the latter, we showed that there are conﬁgurations in which a non-trivial distribution over activation functions provably outperforms the best possible single activation function.
This opens the door to future investigations regarding optimal methods for parameterizing distributions over activation functions for approximate kernel methods, and suggests the possibility that mixtures of nonlinearities could be a useful design consideration when constructing neural network architectures.
Supplemental Material: A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions
The main goal of the supplementary material is to prove Thm. 1.
This is achieved in three steps.
The ﬁrst step is to note that the limiting Stieltjes for the kernel matrix F t F is equivalent to a multivariate a multivariate Gaussian random matrix model with an identical correlation structure.
This was proven in great generality in Banna et al.
(2015) (see Theorem 5.).
Similar results for covariance matrices have been previously studied Bai & Zhou (2008), but since F does not have identically distributed rows (or columns) they are inadequate for our case.
Speciﬁcally, Theorem 5 from Banna et al.
(2015), implies that the two Stieltjes transforms have the same limit for all z such that (cid:61) z > 0 almost surely.
The second step is to derive the leading order correlation structure (or kernel) for F (Lem. 1), and specify a multivariate Gaussian model with the same structure (Subsec. A.1).
We refer to this Gaussian model as a linearized model , since it removes the nonlinearity f from the random part of our random matrix ensemble.
Without loss of generality, we may assume that E F aα = 0 for all a and α .
This is because E F aα only depends on a (as we demonstrate in Sec. A), so we can center F using a rank-1 matrix—a transform that has no effect on the limiting Stieltjes transform as it represents an O (1 /m ) change in the spectrum (see for more detail Bai & Zhou (2008)).
The third a ﬁnal step of the proof is a derivation of a self-consistent equation specifying the limiting
Stieltjes transform of the Gaussian model.
We do this using the resolvent method (see Erdos & Yau (2017) for an introduction).
While the application of this method to multivariate Gaussian covariance matrices is standard, we note the form of the SCE is new and that we include the derivation here for completeness.
At the end of the supplementary material, we include some additional ﬁgures.
A C ORRELATION STRUCTURE OF F
To begin we derive an asymptotic form for correlation structure of F .
For simplicity of presentation, we derive the results for f ( · , B ) ≡ f ( · + B ) , but the argument generalizes directly.
We recall that
W is an n 1 × n 0 random matrix with mean 0 and variance σ 2 w /n
iid Gaussian entries.
Deﬁne c
αβ :=
(cid:80)
k
X kα X kβ /n 0 .
The data matrix X and bias vector B are deterministic, so all expectations are over W (and can be thought of as conditional on X and B ).
Moreover, X and B satisfy the following assumptions:
• c αβ =
δ αβ σ 2 W + O (cid:0) 1 / √ n 0 (cid:1) for all α and β ; • The empirical spectral distribution of 1 n 0 X t X converges in distribution to some measure µ X ; • 1 n 1 (cid:80) n 1 a =1 δ b a → µ B in distribution, or equivalently 1 n 1 (cid:80) n 1 a =1 g ( b a ) → E B ∼ µ B g
( B ) for bounded, continuous functions g .
Observe that
Z [Equation]
[Equation]
are jointly Gaussian.
Moreover,
[Equation]
Note in particular, that Z aα and Z bβ are independent if a (cid:54)
=
b .
For convenience, we normalize c αβ and deﬁne ˜
c αβ :=
c αβ /σ 2 X
and ε α :=
˜ c
αα − 1 .
In order to specify the correlations, we employ transforms of the activation function f .
The transforms are Gaussian integrals of f at different locations and scales:
[Equation]
x ) , (23) η i,j ( x ) :=
E σ i + j X σ i + j W f ( i ) ( σ X σ W N + x ) f ( j ) ( σ X σ W N + x ) , (24) and γ i,j ( x ) :=
E σ i + j X σ i + j W f ( i ) ( σ X σ W N + x ) f ( j ) ( σ X σ W N (cid:48) + x ) =
ξ i ( x ) ξ j ( x ) .
Remark 4.
For simplicity, while writing f ( i ) we assume differentiability of f up to i th order.
However, this assumption can be relaxed, as since the derivatives are only used to take a Gaussian expectation, they can be expressed using Stein’s lemma.
For example, ξ 1 ( x ) =
E Nf ( σ X σ W N + x ) .
Thus, similar results should hold for rectiﬁed linear units for example, regardless of their lack of differentiability at 0.
Lemma 1.
To leading order the correlation structure of F is
[Equation]
Proof of Lem.
Before we prove Lem. 1, we state a simple lemma that is easy to show using Taylor expansion.
Lemma 2.
Suppose ε is a small constant and that N ∼ N (0 , 1) , then
[Equation]
[Equation] [Equation]
and
[Equation]
Recall ξ 0 ( x ) = 0 by assumption.
Using the observation in Eqn. (21), we repeatedly rewrite the expectations in F t F as we can reduce the expectation over W to an expectation over at most two correlated Gaussian random variables.
Let N and N (cid:48) denote independent, standard Gaussian random variables.
Using Lem. 2, we see
[Equation]
Again, using Lem. 2, we get
[Equation]
For the covariance calculation
[Equation]
we deﬁne ρ αβ :=
c αβ / √ c αα and recall that ρ αβ is order 1 / √ m .
Thus,
[Equation]
[Equation]
For the ﬁrst term, we get
[Equation]
For the second term, we have
[Equation]
[Equation]
Thus, to second order we have
[Equation]
A. L INEARIZED MODEL
Deﬁne
[Equation]
where: Θ 1 and Θ 2 are n 1 × m matrices that have iid Gaussian entries with mean 0 and variance 1 /n 1 ; 2) V and C are n 1 × n 1 diagonal matrices with entries
V 2 a = η 00 ( b a ) ≡ η ( b a ) and C 2 a = γ 11 ( b a ) ≡ [Equation] ( b a
[Equation]
41) for a bias vector b ; 3) Σ is a matrix square root of 1 n 0 X t X .
A simple calculation shows that the entries of the matrix F lin match the ﬁrst and second mixed moments of F given in Lem.
However, unlike F , F lin is linear in the random matrices Θ 1 and Θ 2 ; in this sense, it is a linearization of F .
B D ERIVATION OF SELF - CONSISTENT EQUATION FOR F LIN
Theorem 2.
Let m m ( z ) be the Stieltjes transform of 1 n 1 F lin t F lin , i.e. 1 m tr G ( z ) .
Then with probability 1, as n 1 → ∞ , for all z such that (cid:61) z > 0 , m n 1 ( z ) → m ( z ) , where m ( z ) is the solution of the coupled equations m
[Equation]
and
[Equation]
with
[Equation]
We want to study the eigenvalues of 1 n 1 F lin t F lin .
Note without loss of generality it is sufﬁcient to consdier diagonal Σ , since we can diagonalize Σ using some orthogonal matrices O and O (cid:48) as it is the square root of a positive deﬁnite matrix.
Moreover, these orthogonal matrices, when applied to either Θ 1 or Θ 2 do not change their distributions.
Thus, diagonalizing Σ , so that Σ αα = (cid:112) λ
Xα , where λ Xα are the eigenvalues of X t X/n 0 , results in an equivalent matrix ensemble in distribution (see Silverstein & Bai (1995b) for more detail).
With this simpliﬁcation, F lin has independent entries given by
[Equation]
To make the derivation easier, we can partly linearize the problem by studying the matrix
[Equation]
By the Schur complement formula, one easily ﬁnds
[Equation]
z
[Equation]
[Equation]
where G is the inverse of H .
Again by the Schur complement formula
[Equation]
[Equation]
and G aa
[Equation]
for α ∈ { 1 , . . .
, m } and a ∈ { m + 1 , . .
, m + n 1 } and G ( a ) is the inverse of the minor H ( a ) .
Since
G ( α ) is independent of Θ 11 α , . .
, Θ 1 n 1 α and Θ 21 α , . . .
, Θ 2 n 1 α , we see by taking the expectation over these variables that
[Equation]
Moreover, standard concentration inequalities and the Ward identity (see Erdos & Yau (2017)) show
[Equation]
with high probability.
Similar bounds are easily obtained for
[Equation]
We may also replace G ( α ) with G at the expense of another small error that can be bounded using the
Cauchy interlacing theorem: (cid:12)(cid:12)(cid:12) G ( α ) m + a,m + a − G m + a,m + a (cid:12)(cid:12)(cid:12) ≤ O (1 /m ) .
Using this control over these sums, we see
[Equation] [Equation]
55) and
[Equation] [Equation]
Finally, we invert Eqn. (56), multiply by λ
Xα ζ ( b a ) + η ( b a ) − ζ ( b a ) , and average over a to ﬁnd
[Equation]
[Equation]
where ˜ m m ( z ) = 1 m (cid:80) α λ
Xα G αα , where we Taylor expanded in the second step, and where we used our assumption on B .
We can now invert Eqn. (54) and average over α to ﬁnd
[Equation]
Similarly,
[Equation]
Note that the integral over S here is an integral over the limiting distribution of the data µ X , which we assume
[Equation]
in distribution.
In the case of iid Gaussian data, this is exactly given by the Marchenko-Pastur distribution.
C A DDITIONAL FIGURES
[Figure]
Entry ij shows x i · f j .
Although our theoretical results do not give predictions for how the singular vectors change, we found interesting behavior, with very little change to the largest singular vectors (which are nearly isolated in the spectrum), but more mixing of singular vectors in the dense part of the distribution.