Exact Solutions of a Deep Linear Network
Anonymous Author(s)
AffiliationAddress email
Abstract
This work finds the analytical expression of the global minima of a deep linearnetwork with weight decay and stochastic neurons, a fundamental model forunderstanding the landscape of neural networks.
Our result implies that zero isa special point in deep neural network architecture.
We show that weight decaystrongly interacts with the model architecture and can create bad minima at zero ina network with more than 1 hidden layer, qualitatively different from a networkwith only 1 hidden layer.
Practically, our result implies that common deep learninginitialization methods are insufficient to ease the optimization of neural networksin general.
Introduction
Applications of neural networks have achieved great success in various fields.
One central opentheoretical question is why neural networks, being nonlinear and containing many saddle pointsand local minima, can sometimes be optimized easily (Choromanska et al., 2015a) while becomingdifficult and requiring many tricks to train in some other scenarios (Glorot and Bengio, 2010; Gotmareet al., 2018).
One established approach is to study the landscape of deep linear nets (Choromanskaet al., 2015b), which are believed to approximate the landscape of a nonlinear net well.
A series ofworks proved the famous results that for a deep linear net, all local minima are global (Kawaguchi,2016; Lu and Kawaguchi, 2017; Laurent and Brecht, 2018), which is regarded to have successfullyexplained why deep neural networks are so easy to train because it implies that initialization in anyattractive basin can reach the global minimum without much effort (Kawaguchi, 2016).
However, thetheoretical problem of when and why neural networks can be hard to train is understudied.
In this work, we theoretically study a deep linear net with weight decay and stochastic neurons, whoseloss function takes the following form in general:
[Equation]
[Table]
W
[Table]
[Equation]
+
[Equation]
u ∣∣
[Equation]
∑ i =γ
i
[Equation]
∣∣ 2 F ·„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„‚„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„¶ L 2 reg.
, (1) where E x denotes the expectation over the training set, U and W ( i ) are the model parameters, D isthe depth of the network, 1 ϵ is the noise in the hidden layer (e.g., due to dropout),
d i is the widthof the i -th layer, and γ is the strength of the weight decay.
Previous works have studied specialcases of this loss function.
For example, Kawaguchi (2016) and Lu and Kawaguchi (2017) study thelandscape of L 0 when ϵ is a constant (namely, when there is no noise).
Mehta et al.
(2021) studies
L 0 with (a more complicated type of) weight decay but without stochasticity and proved that all the stationary points are isolated.
Another line of works studies L 0 when the noise is caused by dropout(Mianjy and Arora, 2019; Cavazza et al., 2018).
Our setting is more general than the previous worksin two respects.
First, apart from the mean square error (MSE) loss L 0 , an L 2 regularization term(weight decay) with arbitrary strength is included; second, the noise ϵ is arbitrary.
Thus, our setting isarguably closer to the actual deep learning practice, where the injection of noises to latent layers iscommon and the use of weight decay is virtually ubiquitous (Krogh and Hertz, 1992; Loshchilov and
Hutter, 2017).
One major limitation of our work is that we assume the label y to be 1 -dimensional,and it can be an important future problem to prove whether an exact solution exists or not when y ishigh-dimensional.
Our foremost contribution is to prove that all the global minimum of an arbitrarily deep and widelinear net takes a simple analytical form.
In other words, we identify in closed form the global minimaof Eq.
(1) up to a single scalar, whose analytical expression does not exist in general.
We then showthat it has nontrivial properties that can explain many phenomena in deep learning.
In particular, theimplications of our result include (but are not limited to):
Weight decay makes the landscape of neural nets more complicated;
• we show that bad minima 2 emerge as weight decay is applied, whereas there is no bad
minimum when there is no weight decay.
This highlights the need to escape bad local
minima in deep learning with weight decay.
Deeper nets are harder to optimize than shallower ones;
• we show that a D ≥ 2 linear net contains a bad minimum at zero, whereas a D = 1 net
does not.
This partially explains why deep networks are much harder to optimize than
shallower ones in deep learning practice.
Depending on the task, the common initialization methods (such as the Kaiming init.)
can
initialize a deep model in the basin of attraction of the bad minimum at zero;
• common initialization methods initialize the models at a radius of roughly 1 / √ widtharound the origin; however, we show that the width of the bad minimum is taskdependent and can be larger than the initialization radius for tasks with a small margin( ∣∣ E [ xy ]∣∣ ).4.
Stochastic networks in a few asymptotic limits can become deterministic;• we show that the prediction variance of a stochastic net scales towards 0 on the MSEloss as (a) the width tends to infinity, (b) the variance of the latent randomness tends toinfinity, or (c) the depth tends to infinity.
Organization :
In the next section, we discuss the related works.
In Section 3, we derive the exactsolution for a two-layer net.
Section 4 extends the result to an arbitrary depth.
In Section 5, we studyand discuss the relevance of our results to many commonly encountered problems in deep learning.
The last section concludes the work and discusses unresolved open problems.
All proofs are delayedto
Section A. Moreover, additional theoretical results on the effect of including a bias term, the zeroweight decay limit, and the zero noise limit are considered in Section C, D, and E, respectively.
Notation .
For a matrix W , we use W i ∶ to denote the i -th row vector of W .
∣∣
Z ∣∣ denotes the L 2 normif Z is a vector and the Frobenius norm if Z is a matrix.
The notation ∗ signals an optimized quantity.
Additionally, we use the superscript ∗ and subscript ∗ interchangeably, whichever leads to a simplerexpression.
For example, b 2 ∗ and ( b ∗ ) 2 denote the same quantity, while the former is “simpler."
Related Works
In many ways, linear networks have been used to help understand nonlinear networks.
For example,even at depth 0 , where the linear net is nothing but a linear regressor, linear nets are shown to berelevant for understanding the generalization behavior of modern overparametrized networks (Hastieet al., 2019).
Saxe et al.
(2013) studies the training dynamics of a depth- 1 network and uses it tounderstand the dynamics of learning of nonlinear networks.
These networks are the same as a linearregression model in terms of expressivity.
However, the loss landscape is highly complicated due to the existence of more than one layer, and linear nets are widely believed to approximate the losslandscape of a nonlinear net (Kawaguchi, 2016; Hardt and Ma, 2016; Laurent and Brecht, 2018).
Inparticular, the landscape of linear nets has been studied as early as 1989 in Baldi and Hornik (1989),which proposed the well-known conjecture that all local minima of a deep linear net are global.
Thisconjecture is first proved in Kawaguchi (2016), and extended to other loss functions and deeperdepths in Lu and Kawaguchi (2017) and Laurent and Brecht (2018).
Many relevant contemporarydeep learning problems can be understood with the deep linear models.
For example, two-layer linear
VAE models are used to understand the cause of the posterior collapse problem (Lucas et al., 2019;
Wang and Ziyin, 2022).
Deep linear nets are also used to understand the neural collapse problem incontrastive learning (Tian, 2022).
We also provide more empirical evidence in Section 5.
Two-layer Linear Net
This section finds the global minima of a two-layer linear net.
The data point is a d -dimensionalvector x ∈ R d drawn from a data distribution p ( x ) and the labels are generated through an arbitraryfunction y
= y ( x ) ∈ R .
For generality, we let different layers have different strengths of weight decayeven though they often take the same value in practice.
We want to minimize the following objective:
[Equation]
where d 1 is the width of the hidden layer and ϵ i are uncorrelated random variables.
γ w > 0 andγ u > 0 are the weight decay parameters.
Here, we consider a general type of independent noise with 95 E [ ϵ i ] =
and E
[ ϵ i ϵ j ] =
δ ij σ 2 + 1 where δ ij is the Kronecker’s delta, and σ 2 > 0 .
While we formallyrequire γ and σ to nonzero, we discuss the cases of zero limit in Appendix D and E.
For shorthand,we use the notation A 0 ∶= E [ xx T ] , and the largest and the smallest eigenvalues of A 0 are denoted asa max and a min respectively.
a i denotes the i -th eigenvalue of A 0 viewed in any order.
For now, it issufficient for us to assume that the global minimum of Eq.
(2) always exists.
We will prove a moregeneral result in Proposition 1, when we deal with multilayer nets.
3.1 Main Result
We first present two lemmas showing that the global minimum can only lie on a rather restrictivesubspace of all possible parameter settings due to invariances in the objective.
Lemma 1.
At the global minimum of Eq.
(2) , U 2 j = γ w γ u
∑ i W 2 ji for all j .
Proof Sketch .
We use the fact that the first term of Eq.
(2) is invariant to a simultaneous rescaling ofrows of the weight matrix to find the optimal rescaling, which implies the lemma statement.
◻
This lemma implies that for all j , ∣ U j ∣ must be proportional to the norm of its corresponding rowvector in W .
This lemma means that using weight decay has the effect of making all layers of adeep neural network have a balanced norm.
The following lemma further shows that, at the globalminimum, all elements of U must be equal.
Lemma 2.
At the global minimum, for all i and j , we have
[Equation]
Proof Sketch .
We show that if the condition is not satisfied, then an “averaging" transformation willstrictly decrease the objective.
◻
This lemma can be seen as a formalization of the intuition suggested in the original dropout
paper(Srivastava et al., 2014).
Namely, using dropout encourages the neurons to be independent of oneanother and results in an averaging effect.
The second lemma imposes strong conditions on thesolution of the problem, and the essence of this lemma is the reduction of the original problem to alower dimension.
We are now ready to prove our first main result.
Theorem 1.
The global minimum U ∗ and W ∗ of Eq.
(2) is U ∗ = 0 and W ∗
=
if and only if
[Equation]
When ∣∣ E [ xy ]∣∣ 2 > γ u γ w , the global minima are
[Equation]
[Equation]
where r = (± 1 ,..., ± 1 ) is an arbitrary vertex of a d 1 -dimensional hypercube, and b satisfies:
[Equation]
Apparently, b = 0 is the trivial solution that has not learned any feature due to overregularization.
Henceforth, we refer to this solution (and similar solutions for deeper nets) as the “trivial" solution.
We now analyze the properties of the nontrivial solution b ∗ when it exists.
The condition for the solution to become nontrivial is interesting: ∣∣
E [ xy ]∣∣ 2 ≥ γ u γ w .
The term 126 ∣∣ E
[ xy ]∣∣ can be seen as the effective strength of the signal, and γ u γ w is the strength of regularization.
This precise condition means that the learning of a two-layer can be divided into two qualitativelydifferent regimes: an “overregularized regime" where the global minimum is trivial, and a “featurelearning regime" where the global minimum involves actual learning.
Lastly, note that our mainresult does not specify the exact value of b ∗ .
This is because b ∗ must satisfy the condition in Eq.
(6),which is equivalent to a high-order polynomial in b with coefficients being general functions of theeigenvalues of A 0 , whose solutions are generally not analytical by Galois theory.
One special casewhere an analytical formula exists for b is when A
=
σ 2 x I .
See Section B for more discussion.
3.2 Bounding
the
General Solution
While the solution to b ∗ does not admit an analytical form for a general A 0 , one can find meaningfullower and upper bounds to b ∗ such that we can perform an asymptotic analysis of b ∗ .
At the globalminimum, the following inequality holds:
[Equation]
where a min and a max are the smallest and largest eigenvalue of A 0 , respectively.
The middle term isequal to γ u / γ w by the global minimum condition in (33), and so, assuming a min > 0 , this inequalityis equivalent to the following inequality of b ∗ :
[Equation]
Namely, the general solution b ∗ should scale similarly to the homogeneous solution in Eq.
(105) ifwe treat the eigenvalues of A 0 as constants.
Exact Solution for An Arbitrary-Depth Linear Net
This section extends our result to multiple layers, each with uncorrelated stochasticity.
We first derivethe analytical formula for the global minimum of a general arbitrary-depth model.
We then show thatthe landscape for a deeper network is highly nontrivial.
4.1 General Solution
The loss function is
[Equation]
where all the noises ϵ are uncorrelated with each other, and for all i and j ,
Proposition 1.
For D ≥ 1 and strictly positive γ u , γ 1 ,...,γ D , the global minimum for Eq.
(9) exists.
Note that the positivity of the regularization strength is crucial.
If one of the γ i is zero, the globalminimum may not exist.
The following theorem is our second main result.
Theorem 2.
Any global minimum of Eq.
(9) is of the form
[Equation]
where µ = ∏ Di = 2 d i , s 2 = ∏ Di =
d i ( σ 2 + d i ) , b u ≥ 0 and b i ≥ 0 , and r i = (± 1 ,..., ± 1 ) is an arbitraryvertex of a d i -dimensional hypercube for all i .
Furthermore, let b 1 ∶= √∣∣ W i ∶ ∣∣ 2 / d and b D + 1 ∶= b u , b isatisfies
[Equation]
[Equation]
Proof Sketch .
We prove by induction on the depth D .
The base case is proved in Theorem 1.
We thenshow that for a general depth, the objective involves optimizing subproblems, one of which is a D − 1layer problem that follows by the induction assumption, and the other is a two-layer problem that hasbeen solved in Theorem 1.
Putting these two subproblems together, one obtains Eq.
◻
Remark.
We deal with the technical case of having a bias term for each layer in Appendix C. Forexample, we will show that if one has preprocessed the data such that E [ x ] = 0 and E
[ y ] = 0 , ourmain results remain precisely unchanged.
The condition in Eq.
(11) shows that the scaling factor b i for all i is not independent of one another.
This automatic balancing of the norm of all layers is a consequence of the rescaling invariance of themultilayer architecture and the use of weight decay.
It is well-known that this rescaling invariancealso exists in a neural network with the ReLU activation, and so this balancing condition is alsodirectly relevant for ReLU networks.
Condition (11) implies that all the b i can be written in terms of one of the b i :
[Equation]
where c 0 = (
[Equation]
2 d
[Equation]
D /
[Equation]
γ u ∏ Di = 2 γ i ∏ Di =
d i √ d 1 and b ≥ 0 .
Consider the first layer ( i = 1 ), Eq (11) shows that theglobal minimum must satisfy the following equation, which is equivalent to a high-order polynomialin b that does not have an analytical solution in general:
[Equation]
[Equation]
Thus, this condition is an extension of the condition (6) for two-layer networks.
At this point, it pays to clearly define the word “solution," especially given that it has a specialmeaning in this work because it now becomes highly nontrivial to differentiate between the two typesof solutions.
Definition 1.
We say that a non-negative real b is a solution if it satisfies Eq.
A solution istrivial if b = 0 and nontrivial otherwise.
Namely, a global minimum must be a solution, but a solution is not necessarily a global minimum.
We have seen that even in the two-layer case, the global minimum can be the trivial one when thestrength of the signal is too weak or when the strength of regularization is too strong.
It is thus naturalto expect 0 to be the global minimum under a similar condition, and one is interested in whether thecondition becomes stronger or weaker as the depth of the model is increased.
However, it turns outthis naive expectation is not true.
In fact, when the depth of the model is larger than 2 , the conditionfor the trivial global minimum becomes highly nontrivial.
The following proposition shows why the problem becomes more complicated.
In particular, we haveseen that in the case of a two-layer net, some elementary argument has helped us show that the trivialsolution b = 0 is either a saddle or the global minimum.
However, the proposition below shows thatwith D ≥ 2 , the landscape becomes more complicated in the sense that the trivial solution is alwaysa local minimum, and it becomes difficult to compare the loss value of the trivial solution with thenontrivial solution because the value of b ∗ is unknown in general.
Proposition 2.
Let D ≥ 2 in Eq.
Then, the solution U = 0 , W ( D ) = 0 , ..., W ( 1 ) = 0 is a localminimum with a diagonal positive-definite Hessian.
Comparing the Hessian of D ≥ 2 and D = 1 , one notices a qualitative difference: for D ≥ 2 , the
Hessian is always diagonal (at 0 ); for D = 1 , in sharp contrast, the off-diagonal terms are nonzero ingeneral, and it is these off-diagonal terms that can break the positive-definiteness of the Hessian.
Thisoffers a different perspective on why there is a qualitative difference between D = 1 and D = 2 .
Lastly, note that, unlike the depth- 1 case, one can no longer find a precise condition such that a b ≠ 0solution exists for a general A 0 .
The reason is that the condition for the existence of the solution isnow a high-order polynomial with quite arbitrary intermediate terms.
The following proposition givesa sufficient but stronger-than-necessary condition for the existence of a nontrivial solution, when allthe σ i , intermediate width d i and regularization strength γ i are the same.
Proposition 3.
Let σ 2 i =
σ 2 > 0 , d i = d 0 and γ i = γ > 0 for all i .
Assuming a min > 0 , the onlysolution is trivial if
[Equation]
Nontrivial solutions exist if
[Equation]
Moreover, the nontrivial solutions are both lower and upper-bounded:
[Equation]
Proof Sketch .
The proof follows from the observation that the l.h.s.
of Eq.
(13) is a continuousfunction and must cross the r.h.s.
under certain sufficient conditions. ◻
One should compare the general condition here with the special condition for D = 1 .
One sees thatfor D ≥ 2 , many other factors (such as the width, the depth, and the spectrum of the data covariance
A 0 ) come into play to determine the existence of a solution apart from the signal strength E [ xy ] andthe regularization strength γ .
4.2 Which Solution is the Global Minimum?
Again, we set γ i = γ > 0 , σ 2 i =
σ 2 > 0 and d i = d 0 > 0 for all i for notational concision.
Using thiscondition and applying Lemma 3 to Theorem 2, the solution now takes the following form, where
[Equation]
[Equation] [Equation]
The following theorem gives a sufficient condition for the global minimum to be nontrivial.
It alsoshows that the landscape of the linear net becomes complicated and can contain more than 1 localminimum when a certain condition is satisfied.
Theorem 3.
Let σ 2 i =
σ 2 > 0 , d i = d 0 and γ i = γ > 0 for all i and assuming a min > 0 .
Then, if
[Equation] [Equation]
the global minimum of Eq.
(9) is one of the nontrivial solutions.
While there are various ways this bound can be improved, it is general enough for our purpose.
Inparticular, one sees that, for a general depth, the condition for having a nontrivial global minimumdepends not only on the E [ xy ] and γ but also on the model architecture in general.
For a moregeneral architecture with different widths etc., the architectural constant c 0 from Eq.
(13) will alsoenter the equation.
In the limit of D → 1 + , relation (18) reduces to
[Equation]
E
[Equation]
(19) which is the condition derived for the 2-layer case.
Implications
We first caution the readers that the following discussion should be taken with a caveat and is based onthe philosophy that deep linear nets can approximate the nonlinear ones, which we try to back up withempirical evidence whenever possible.
This approximation certainly holds for fully connected modelswith differentiable activation functions such as tanh or Swish because they are, up to first-order Taylorexpansion, a deep linear net around zero, which is the region our theory is the most relevant for.
Weempirically demonstrate in Section G that close to the origin, the landscape of linear nets can indeedapproximate that of nonlinear nets quite well.
Landscape of multi-layer neural networks .
The combination of Theorem 3 and Proposition 2shows that the landscape of a deep neural network can become highly nontrivial when there isa weight decay and when the depth of the model is larger than 2 .
This gives an incomplete butmeaningful picture of a network’s complicated but interesting landscape beyond two layers (see
Figure 1 for an incomplete summary of our results).
In particular, even when the nontrivial solution isthe global minimum, the trivial solution is still a local minimum that needs to be escaped.
Our resultsuggests the previous understanding that all local minima of a deep linear net are global and cannotgeneralize to many practical settings where deep learning is found to work well.
For example, a seriesof works attribute the existence of bad (non-global) minima to the use of nonlinearities nonlinearnets (Kawaguchi, 2016) or the use of a non-regular (non-differentiable) loss function (Laurent and
Brecht, 2018).
Our result, in contrast, shows that the use of a simple weight decay is sufficient tocreate a bad minimum.
Moreover, the problem with such a minimum is two-fold: (1) (optimization)it is not global and so needs to be “overcome" and (2) (generalization) it is a minimum that has notlearned any feature at all because the model constantly outputs zero.
To the best of our knowledge,previous to our work, there has not been any proof that a bad minimum can generically exist in arather arbitrary network without any restriction on the data.
Thus, our result offers direct and solidtheoretical justification for the widely believed importance of escaping local minima in the field ofdeep learning (Kleinberg et al., 2018; Liu et al., 2021; Mori et al., 2022).
In particular, previousworks on escaping local minima often hypothesize landscapes that are of unknown relevance to anactual neural network.
With our result, this line of research can now be established with respect tolandscapes that are actually deep-learning-relevant.
Previous works also argue that having a deeper depth does not create a bad minimum (Lu and
Kawaguchi, 2017).
While this remains true, its generality and applicability to practical settings nowalso seem low.
Our result shows that as long as weight decay is used, and as long as D ≥ 2 , there isindeed a bad local minimum at 0 .
In contrast, there is no bad minimum at 0 for a depth- 2 network:the point b = 0 is either a saddle or the global minimum.
7 Having a deeper depth thus alters thequalitative nature of the landscape, and our results agree better with the common observation that adeeper network is harder, if not impossible, to optimize.
In the case of nonlinear networks without regularization, a few works proved the existence of bad minima.
However, the previous results strongly depend on the data and are rather independent of architecture.
For example, one major assumption is that the data cannot be perfected and fitted by a linear model (Yun et al., 2018; Liu, 2021;
He et al., 2020).
Some other works explicitly construct data distribution (Safran and Shamir, 2018;
Venturi et al., 2019).
Our result, in contrast, is independent of the data.
Of course, in practice, the model trained with SGD can still converge to the trivial solution even if it is a saddle point (Ziyin et al., 2021) because SGD with a finite learning rate is, in general, not a good estimator of the local minima.
We note that our result can also be relevant for more modern architectures such as the ResNet. Using ResNet, one needs to change the dimension of the hidden layer after every bottleneck and alearnable linear transformation is applied here.
Thus, the “effective depth” of a ResNet would beroughly between the number of its bottlenecks and its total number of blocks.
For example, for a
ResNet18 applied to CIFAR10, it often has 5 bottlenecks and 18 layers in total.
We thus expect it tohave qualitatively similar behavior to a deep linear net with a depth in between.
See Figure 2.
Theexperimental details are given in Section F.
Learnability of a neural network .
Now we analyze the solution when D tends to infinity.
We first note that the existencecondition bound in (15) becomes exponentially harder to satisfyas D becomes large: [Equation]
[Equation]
When this bound is not satisfied, the given neural network cannot learn the data.
Recall that for a two-layer net, and theexistence condition is nothing but ∣∣
E [ xy ]∣∣ 2 > γ 2 , independentof the depth, width, or the stochasticity in the model.
For adeeper network, however, every factor comes into play, and thearchitecture of the model has a strong (and dominant) influenceon the condition.
In particular, a factor that increases polynomial in the model width and exponentially in the model depthappears.
A practical implication is that the use of weight decay may be too strong for deep networks.
If oneincreases the depth or width of the model, one should also roughly decrease γ according to Eq.
Insufficiency of the existing initialization schemes .
We have shown that 0 is often a bad localminimum for deep learning.
Our result further implies that escaping this local minimum can be highlypractically relevant because standard initialization schemes are trapped in this local minimum fortasks where the signal E [ xy ] is weak.
See Inequality (16): any nontrivial global minimum is lowerbounded by a factor proportional to ( γ /∣∣ E
[ xy ]∣∣ 1 /( D − 1 ) )/ d 0 , which can be seen as an approximationof the radius of the local minimum at the origin.
In comparison, standard deep learning initializationschemes such as Kaiming init.
initialize at a radius roughly 1 /√ d 0 .
Thus, for tasks E
[ xy ] ≪ γ /√ d 0 ,these initialization methods are likely to initialize the model in the basin of attraction of the trivialregime, which can cause a serious failure in learning.
To demonstrate, we perform a numericalsimulation shown in the right panel of Figure 1, where we train D = 2 nonlinear networks with width32 with SGD on tasks with varying ∣∣
E [ xy ]∣∣ .
For sufficiently small ∣∣ E [ xy ]∣∣ , the model clearly isstuck at the origin.
In contrast, linear regression is never stuck at the origin.
Our result thus suggeststhat it may be desirable to devise initialization methods that are functions of the data distribution
Prediction variance of stochastic nets .
A major extension of the standard neural networks isto make them stochastic, namely, to make the output a random function of the input.
In a broadsense, stochastic neural networks include neural networks trained with dropout (Srivastava et al.
,2014; Gal and Ghahramani, 2016), Bayesian networks (Mackay, 1992), variational autoencoders(VAE) (Kingma and Welling, 2013), and generative adversarial networks (Goodfellow et al., 2014).
Stochastic networks are thus of both practical and theoretical importance to study.
Our result canalso be used for studying the theoretical properties of stochastic neural network.
Here, we present asimple application of our general solution to analyze the properties of a stochastic net.
The followingtheorem summarizes our technical results.
Theorem 4.
Let σ 2 i =
σ 2 > 0 , d i = d 0 and γ i = γ > 0 for all i .
Let A
= σ 2 x I .
Then, at any globalminimum of Eq.
(9) , holding other parameters fixed,
in the limit of large d 0 , V ar [ f ( x )]
=
O
( d − 10 ) ;
in the limit of large σ 2 , V ar [ f ( x )]
= O ( 1 ( σ 2 ) D ) ;
In the limit of large D , V ar [ f ( x )]
= O ( e − 2 D log [( σ 2 + d 0 )/
d 0 ] ) .
Interestingly, the scaling of prediction variance in asymptotic σ 2 is different for different widths.
Thethird result shows that the prediction variance decreases exponentially fast in D .
In particular, thisresult answers a question recently proposed in Ziyin et al.
(2022): does a stochastic net trained on
MSE have a prediction variance that scales towards 0 ?
Ziyin et al.
(2022) shows that for a genericnonlinear model, the prediction variance at the global minimum scales is O ( d − 1 ) .
Ziyin et al.
(2022)also hypothesizes that the prediction variance scales towards 0 as the strength of latent variance σ 2increases to infinity.
We improve on their result in the case of a deep linear net by (a) showing thatthe d − 10 is tight in general, independent of the depth or other factors of the model, and (b) proving apower-law bound for asymptotic σ 2 that has been conjectured, and (c) proving a novel bound showingthat the variance also scales towards zero as depth increases, which is a novel result of our work.
Ourresult also offers an important insight into the cause of the vanishing prediction variance.
Previousworks (Alemi et al., 2018; Ziyin et al., 2022) often attribute the cause to the fact that a wide neuralnetwork is too expressive.
However, our result implies that this is not always the case because alinear network, with limited expressivity, can also have a vanishing variance as the model tends to aninfinite width.
in deep learning.
Lastly, our result may also shed light on the well-known problem ofposterior collapse problem in Bayesian deep learning and the neural collapse problem in contrastivelearning.
Tian (2022) used a deep linear network to understand the contrastive learning problem andshowed that the loss function could be decomposed into a min-max problem with a regularizationterm.
Wang and Ziyin (2022) identified the cause of the posterior collapse in a two-layer VAEstructure to be that the regularization of the mean of the latent variable z is too strong.
Althoughappearing in different contexts of deep learning, the two problems share the same phenomenology thatthe model converges to a “collapsed" regime where the learned representation becomes a constant,which is in agreement with the behavior of the trivial regime we identified.
Our result thus shows thatthe two problems may share a common cause that is a unique feature of deep learning.
One importantfuture step is to investigate this connection in greater detail.
Conclusion
In this work, we derived the exact solution of a deep linear net with arbitrary depth and withstochasticity.
The global minimum is shown to take exact forms.
Our work sheds light on thehighly complicated landscape of a deep neural network.
Compared to the previous works that mostlyfocus on the qualitative understanding of the linear net, our result offers a more precise quantitativeunderstanding of deep linear nets.
Quantitative understanding is one major benefit of knowing theexact solution, whose usefulness we have also demonstrated with the various implications.
Theresults, although derived for linear models, are also empirically shown to be relevant for networkswith nonlinear activations.
Lastly, our results strengthen the line of thought that analytical approachesto deep linear models can be used to understand deep neural networks, and it is the sincere hope ofthe authors to attract more attention to this promising field.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] The experiments
are only for demonstration and are straightforward to reproduce following the theory.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No] The fluctuations are visually negligible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] They are done on a single 3080Ti
GPU.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]