D EEP R E LU N ETWORKS P RESERVE E XPECTED L ENGTH
A BSTRACT
Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize.
One natural measure of complexity is how the network distorts length – if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs?
It has been widely believed that this length grows exponentially in network depth.
We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization.
We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes.
These theoretical results are corroborated by our experiments.
I NTRODUCTION
The utility of deep neural networks ultimately arises from their ability to learn functions that are sufﬁciently complex to ﬁt training data and yet simple enough to generalize well.
Understanding the precise sense in which functions computed by a given network have low or high complexity is therefore important for studying when the network will perform well.
Despite the fundamental importance of this question, our mathematical understanding of the functions expressed and learned by different neural network architectures remains limited.
A popular way to measure the complexity of a neural network function is to compute how it distorts lengths.
This may be done by considering a set of inputs lying along a curve and measuring the length of the corresponding curve of outputs.
It has been claimed in prior literature that in a ReLU network this length distortion grows exponentially with the network’s depth Price & Tanner (2019); Raghu et al. and this has been used as a justiﬁcation of the power of deeper networks.
We prove that, in fact, for networks with the typical initialization used in practice, expected length distortion does not grow at all with depth.
Our main contributions are:
We prove that for ReLU networks initialized with the usual 2 / fan-in weight variance, the expected length distortion does not grow with depth at initialization, actually decreasing slightly with depth (Thm. 3.1) and exhibiting an interesting width dependency.
We prove bounds on higher moments of the length distortion, giving upper bounds that hold with high probability (Thm. 4.1).
We also obtain similar results for the distortion in the volume of higher-dimensional manifolds of inputs (Thm. 4.2).
We empirically verify that our theoretical results accurately predict observed behavior for networks at initialization, while previous bounds are loose and fail to capture subtle architecture dependencies.
It is worth explaining why our conclusions differ from those of Price & Tanner (2019); Raghu et al.
First, prior authors prove only lower bounds on the expected length distortion, while we use different methodology to calculate tight upper bounds, allowing us to say that the expected length distortion does not grow with depth.
As we show in Thm. 5.1 and Fig.
1, the prior bounds are in fact quite loose.
Second, Thm. 3(a) in Raghu et al.
(2017) has a critical typo 1 , which has unfortunately been perpetuated in Thm.
1 of Price & Tanner (2019).
Namely, the “2” was omitted in the following statement (paraphrased from the original):
[Equation]
where σ 2 w / width is the variance of the weight distribution.
As we prove in Thm. 3.1, leaving out the 2 makes the statement false, incorrectly suggesting that length distortion explodes with depth for the standard He initialization σ w = √ 2 .
Finally, prior authors drew the conclusion that length distortion grows exponentially with depth by considering the behavior of ReLU networks with unrealistically large weights.
If one multiplies by C the weights and biases of a ReLU network, one multiplies the length distortion by C depth , so it should come as no surprise that there exist settings of the weights for which the distortion grows (or decays) exponentially with depth.
The value of our results comes in analyzing the behavior speciﬁcally at He initialization ( σ w = √ 2 )
He et al.
This initialization is the one used in practice, since this is the weight variance that must be used if the outputs Hanin & Rolnick (2018) and gradients Hanin (2018) are to remain well-controlled at init.
In the present work, we show that this is also the correct initialization for the expected length distortion to remain well-behaved.
R ELATED W ORK
A range of complexity measures for functions computed by deep neural networks have been considered in the literature, dividing the prior work into at least three categories.
In the ﬁrst, the emphasis is on worst-case (or best-case ) scenarios – what is the maximal possible complexity of functions computed by a given network architecture.
These works essentially study the expressive power of neural networks and often focus on showing that deep networks are able to express functions that cannot be expressed by shallower networks.
For example, it has been shown that it is possible to set the weights of a deep ReLU network such that the number of linear regions computed by the network grows exponentially in the depth Daniely (2017); Eldan & Shamir (2016); Mont ´ ufar et al. Telgarsky (2015; 2016).
Other works consider the degree of polynomials approximable by networks of different depths Lin et al.
Rolnick & Tegmark (2018) and the topological invariants of networks Bianchini & Scarselli (2014).
While such work has sometimes been used to explain the utility of different neural network architectures (especially deeper ones), a second strand of prior work has shown that a signiﬁcant gap can exist between the functions expressible by a given architecture and those which may be learned in practice.
Such average-case analyses have indicated that some readily expressible functions are provably difﬁcult to learn Shalev-Shwartz et al. (2017) or vanishingly unlikely for random networks
Hanin & Nica (2020); Hanin & Rolnick (2019; 2020), and that some functions learned more easily by deep architectures are nonetheless expressible by shallow ones Ba & Caruana (2014). (While here we consider neural nets with ReLU activation, it is worth noting that for arithmetic circuits, worst-case and average-case scenarios may be more similar – in both scenarios, a signiﬁcant gap exists between the matricization rank of the functions computed by deep and shallow architectures Cohen et al.
As noted earlier, average-case analyses in Price & Tanner (2019)
; Raghu et al.
(2017) provided lower bounds on expected length distortion, while Poole et al.
(2016) presented a similar analysis for the curvature of output trajectories.
Finally, a number of authors have sought complexity measures that either empirically or theoretically correlate with generalization Jiang et al.
Such measures have been based on classiﬁcation margins Bartlett et al.
(2017), network compressibility Arora et al.
(2018), and PAC-Bayes considerations Dziugaite & Roy (2017).
E XPECTED L ENGTH D ISTORTION 3. M OTIVATION
While neural networks are typically overparameterized and trained with little or no explicit regularization, the functions they learn in practice are often able to generalize to unseen data.
This phenomenon indicates an implicit regularization that causes these learned functions to be surprisingly simple.
Why this occurs is not well-understood theoretically, but there is a high level intuition.
In a nutshell, randomly initialized neural networks will compute functions of low complexity.
Moreover, as optimization by ﬁrst order methods is a kind of greedy local search, network training is attracted to minima of the loss that are not too far from the initialization and hence will still be well-behaved.
While this intuition is compelling, a key challenge in making it rigorous is to devise appropriate notions of complexity that are small throughout training.
The present work is intended to provide a piece of the puzzle, making precise the idea that, at the start of training, neural networks compute tame functions.
Speciﬁcally, we demonstrate that neural networks at initialization have low distortion of length and volume, as deﬁned in §3.2.
An important aspect of our analysis is that we study networks in typical, average-case scenarios.
A randomly initialized neural network could, in principle, compute any function expressible by a network of that architecture, since the weights might with low probability take on any set of values.
Some settings of weights will lead to functions of high complexity, but these settings may be unlikely to occur in practice, depending on the distribution over weights that is under consideration.
As prior work has emphasized Price & Tanner (2019); Raghu et al. (2017), atypical distributions of weights can lead to exponentially high length distortion.
We show that, in contrast, for deep ReLU networks with the standard initialization, the functions computed have low distortion in expectation and with high probability.
our Thm. 5.
(solid lines) to the lower bounds proven in prior work Raghu et al.
(2017) (dashed lines) and the true empirical means (colored dots), which closely track our predictions. (b) zooms in on the upper part of (a), showing the empirical mean length distortion as a function of depth for different widths. The horizontal black line is y = Γ(3) Γ(2 . 5) √ 2 .
the mean length distortion we predict when n 0 = n L in the limit of inﬁnite width for any ﬁxed depth.
In each network, width is constant in all hidden layers, while input and output dimension are both 5 .
The input curve is a ﬁxed line segment of unit length.
Length distortion is calculated for 500 different initializations of the weights and biases of the network (the weight variance is 2 / fan-in).
For further experimental details, see Appendix B.
D EFINITIONS
Let L ≥ 1 be a positive integer and ﬁx positive integers n 0 , . . . , n L .
We consider a fully connected feed-forward ReLU network N with input dimension n 0 , output dimension n L , and hidden layer widths n 1 , . . .
, n L − 1 .
Suppose that the weights and biases of N are independent and Gaussian with the weights W ( (cid:96) ) ij between layers (cid:96) − 1 and (cid:96) and biases b ( (cid:96) )
j in layer (cid:96) satisfying:
[Equation]
Here, C b > 0 is any ﬁxed constant and G ( µ, σ 2 ) denotes a Gaussian with mean 0 and variance σ 2 .
For any M ⊆ R n 0 , we denote by N ( M ) ⊆
R n L the (random) image of M under the map x (cid:55)→ N ( x ) .
Our primary object of the study will be the size of the output N ( M ) relative to that of M .
Speciﬁcally, when M is a 1 -dimensional curve, we deﬁne
[Equation]
Note that while a priori this random variable depends on M , we will ﬁnd that its statistics depend only on the architecture of N (see Thms.
3.1 and 5.1).
3. R ESULTS
We prove that the expected length distortion does not grow with depth – in fact, it slightly decreases.
Our result may be informally stated thus (for a formal statement and proof, see Thm. 5. and App. C.):
Theorem 3.1 (Length distortion: Mean; Informal Statement of Theorem 5.1) .
Consider a ReLU network of depth L , input dimension n 0 , output dimension n L , and hidden layer widths n (cid:96) , with weights given by standard He normal initialization He et al.
The expected length distortion is upper bounded by (cid:112) n L /n 0 .
More precisely:
[Equation] [Figure]
Our experiments align with the theorem.
In Figure 1, we compute the empirical mean of the length distortion for randomly initialized deep
ReLU networks of various architectures with ﬁxed input and output dimension.
The ﬁgure conﬁrms three of our key predictions: (1) the expected length distortion does not grow with depth, and actually decreases slightly; (2) the decrease happens faster for narrower networks (since 1 /n (cid:96) is larger for smaller n (cid:96) ); and (3) for equal input and output dimension n 0 = n L , there is an upper bound of 1 (in fact, the tighter upper bound of C is approximately 0.
for n L = 5 , as shown in the ﬁgure).
The ﬁgure also shows the prior bounds in Raghu et al.
(2017) to be quite loose.
For further experimental details, see Appendix B.
In Fig.
2, we instead ﬁx the set of hidden layer widths but vary input and output dimensions.
The results conﬁrm that indeed the expected length distortion grows as the square root of the ratio of output to input dimension.
Note that our theorem also applies for initializations other than He normal; the result in such cases is simply less interesting.
Suppose we initialize the weights in each layer (cid:96) as i.i.d.
normal with variance 2 c 2 /n (cid:96) − 1 instead of 2 /n (cid:96) − 1 .
This is equivalent to taking a He normal initialization and multiplying the weights by c .
Multiplying the weights and biases in a depth- L ReLU network by c simply multiplies the output (and hence the length distortion) by c L , so the expected distortion is c L times the value given in Thm. 3.1.
This emphasizes why an exponentially large distortion necessarily occurs if one sets the weights too large.
3. I NTUITIVE EXPLANATION
The purpose of this section is to give an intuitive but technical explanation for why, in Theorem 3.1, we ﬁnd that the distortion len( N ( M )) / len( M ) of the length of a 1 D curve M under the neural network N is typically not much larger than 1 , even for deep networks.
Our starting point is that, near a given point x ∈ M , the length of the image under N of a small portion of M with length dx is approximately given by || J x u || dx , where J x is the input-output Jacobian of N at the input x and u is the unit tangent vector to M at x .
Prior work (see Fact 7.2 in Allen-Zhu et al. In fact, in Lemma C.1 of the Supp. Material, we use a simple argument to give upper bounds on the moments of len( N ( M ))
/ len( M ) in terms of the moments of the norm || J x u || of the Jacobian-vector product J x u . on a ﬁxed line segment.
As predicted in Thm.
4.1, the mean length distortion decreases with the sum of width reciprocals.
Empirical standard deviation does not, in general, increase with greater depth, remaining modest throughout, as is consistent with the upper bound on variance in Thm. 4.1.
Thus, upper bounds on the length distortion reduce to studying the Jacobian J x , which in a network with
L hidden layers can be written as a product J x = J L,x · · · J 1 ,x of the layer (cid:96) − 1 to layer (cid:96) Jacobians J (cid:96),x .
In a worst-case analysis, the left singular vectors of J (cid:96),x would align with the right singular vectors of J (cid:96) +1 ,x , causing the resulting product J x to have a largest singular value that grows exponentially with L .
However, at initialization, this is provably not the case with high probability.
Indeed, the Jacobians J (cid:96),x are independent (see Lemma C.3) and their singular vectors are therefore incoherent. We ﬁnd in particular (Lemma C.2) the following equality in distribution:
[Equation]
where e 1 is the ﬁrst standard unit vector and the terms in the product are independent.
On average each term in this product is close to 1 :
[Equation]
This is a consequence of initializing weights to have variance 2 / fan-in.
Put together, the two preceding displayed equations suggest writing
[Equation]
The terms being summed in the exponent are independent and the argument of each logarithm scales like 1 + O ( n − 1 (cid:96) ) .
With probability 1 − δ we have for all (cid:96) that log( || J (cid:96),x e 1 || ) ≤ cn − 1 (cid:96) , where c is a constant depending only on δ .
Thus, all together, [Equation] with high probability.
In particular, in the simple case where n (cid:96) are proportional to a single large constant n , we ﬁnd the typical size of || J x u || is exponential in L/n rather than exponential in L , as in the worst case. If N is wider than it is deep, so that L/n is bounded above, the typical size of || J x u || and hence of len( N ( M )) / len( M ) remains bounded. Theorem 5.1 makes this argument precise.
Moreover, let us note that articles like Daniely (2017);
Giryes et al. ; Poole et al. (2016) show low distortion of angles and volumes in very wide networks. Our results, however, apply directly to more realistic network widths.
F URTHER R ESULTS 4. H IGHER MOMENTS
We have seen that the mean length distortion is upper-bounded by 1, and indeed by a function of the architecture that decreases with the depth of the network.
However, a small mean is insufﬁcient by itself to ensure that typical networks have low length distortion.
For this, we now show that in fact all moments of the length distortion are well-controlled.
Speciﬁcally, the variance is bounded above by the ratio of output to input dimension, and higher moments are upper-bounded by a function that grows very slowly with depth.
Our results may be informally stated thus:
Theorem 4.1 (Length distortion: Higher moments) .
Consider, as before, a ReLU network of depth
L , input dimension n 0 , output dimension n L , and hidden layer widths n (cid:96) , with weights given by He normal initialization.
We have the following bounds on higher moments of the length distortion:
[Equation] for some universal constant c > 0 .
A formal statement and proof of this result are given in Theorem 5.1 and Appendix C.
We consider the mean and variance of length distortion for a wide range of network architectures in Figure 3.
The ﬁgure conﬁrms that variance is modest for all architectures and does not increase with depth, and that mean length distortion is consistently decreasing with the sum of layer width reciprocals, as predicted by Theorem 4.1.
4. H IGHER - DIMENSIONAL VOLUMES
Another natural generalization to consider is how ReLU networks distort higher-dimensional volumes:
Given a d -dimensional input M , the output N ( M ) will in general also be d -dimensional, and we can therefore consider the volume distortion vol d ( N ( M )) /
vol d ( M ) .
The theorems presented in §3.3 when d = 1 can be extended to all d , and the results may be informally stated thus:
Theorem 4.2 (Volume distortion) .
Consider a ReLU network of depth L , input dimension n 0 , output dimension n L , and hidden layer widths n (cid:96) , with weights given by He normal initialization.
Both the squared mean and the variance of volume distortion are upper-bounded by:
[Equation]
A formal statement and proof of this result are given in Theorem 5.2 and Appendix D.
F ORMAL S TATEMENTS
In this section, we provide formal statements of our results.
Full proofs are given in the appendices.
5. O NE - DIMENSIONAL MANIFOLDS
Fix a smooth one-dimensional manifold M ⊆ R n 0 (i.e. a curve) with unit length.
Each point on
M represents a possible input to our ReLU network N , which we recall has input dimension n 0 , output dimension n L , and hidden layer widths n 1 , . .
. , n L − 1 .
The function x (cid:55)→ N ( x ) computed by N is continuous and piecewise linear.
Thus, the image N ( M ) ⊆
R n L of M under this map is a piecewise smooth curve in R n L with a ﬁnite number of pieces, and its length len( N ( M )) is a random variable.
Our ﬁrst result states that, provided N is wider than it is deep, this distortion is small with high probability at initialization.
Theorem 5.1.
Let N be a fully connected ReLU network with input dimension n 0 , output dimension n L , hidden layer widths n 1 , . . .
, n L − 1 , and independent centered Gaussian weights/biases with the weights having variance 2 / fan-in (as in (1)).
Let M be a 1 -dimensional curve of unit length in R n 0 .
Then, the mean length E [len( N ( M ))] equals
[Equation]
where implied constants in O ( · ) are universal and Γ( · ) denotes the Gamma function.
Moreover:
[Equation]
Finally, there exist universal constants c 1 , c 2 > 0 such that if m < c 1 min { n 1 , . . .
, n L − 1 } , then
[Equation]
Theorem 5.1 is proved in §C. Several comments are in order.
First, we have assumed for simplicity that M has unit length.
For curves of arbitrary length, both the equality (2) and the bounds (3) and (4) all hold and are derived in the same way provided len( N ( M )) is replaced by the distortion len( N ( M )) / len( M ) per unit length.
Second, in the expression (2), note that the exponent tends to 0 as n (cid:96) → ∞ for any ﬁxed L .
More precisely, when n (cid:96) = n is large and constant, it scales as − 5 L/ 8 n .
This shows that, for ﬁxed n 0 , n L , n , the mean E [len( N ( M ))] is actually decreasing with L .
This somewhat surprising phenomenon is borne out in Fig.
and is a consequence of the fact that wide fully connected ReLU nets are strongly contracting (see e.g. Thms.
3 and 4 in Giryes et al.
Third, the pre-factor ( n L /n 0 ) 1 / 2 encodes the fact that, on average over initialization, for any vector of inputs x ∈ R n 0 , we have (see e.g. Cor. 1 in Hanin & Rolnick (2018))
[Equation]
(5) where the approximate equality becomes exact if the bias variance C b is set to 0 (see (1)).
In other words, at the start of training, the network N rescales inputs by an average factor ( n L /n 0 ) 1 / 2 .
This overall scaling factor also dilates len( N ( M )) relative to len( M ) .
Fourth, we stated Theorem 5.1 for Gaussian weights and biases (see (1)), but the result holds for any weight distribution that is symmetric around 0 , has variance 2 / fan-in , and has ﬁnite higher moments.
The only difference is that the constant 5 / 8 may need to be slightly enlarged (e.g. around (19)).
Finally, all the estimates (2), (3), and (4) are consistent with the statement that, up to the scaling factor ( n L /n 0 ) 1 / 2 , the random variable len( N ( M )) is bounded above by a log-normal distribution: len( N ( M )) ≤
[Equation]
exp (cid:20) [Equation] G (cid:18) β 2 , β (cid:19)(cid:21) , where β = c
L (cid:88) (cid:96) = n − 1 (cid:96) and c is a ﬁxed constant.
5. H IGHER - DIMENSIONAL MANIFOLDS
The results in the previous section generalize to the case when M has higher dimension.
To consider this case, ﬁx a positive integer d ≤ min { n 0 , . . .
, n L } and a smooth d − dimensional manifold M ⊆
R n 0 of unit volume vol d ( M ) = 1 .
Note that if d > min { n 0 , . . .
, n L } , then N ( M ) is at most ( d − 1) -dimensional and its d -dimensional volume vanishes.
Its image N ( M ) ⊆
R n L is a piecewise smooth manifold of dimension at most d .
The following result, proved in §D, gives an upper bound on the typical value of the d − dimensional volume of N ( M ) .
Theorem 5.2.
Let N be a fully connected ReLU network with input dimension n 0 , output dimension n L , hidden layer widths n 1 , . . .
, n L − 1 and independent centered Gaussian weights/biases with the variance of the weights given by 2 / fan-in (see (1)).
Let M be a d -dimensional smooth submanifold of R n 0 with unit volume and d ≤ min { n 0 , . . .
, n L } .
Then, both the squared mean and the variance of the d − dimensional volume vol d ( N ( M )) of N ( M ) is bounded above by
[Equation]
P ROOF S KETCH
The purpose of this section is to explain the main steps to obtaining the mean and variance estimates (2) and (3) from Theorem 5.1.
In several places, we will gloss over some mathematical subtleties that we deal with in the detailed proof of Theorem 5.1 given in Appendix C. We will also content ourselves here with proving a slightly weaker estimate than (2) and show instead simply that
[Equation]
We refer the reader to Appendix C for a derivation of the more reﬁned result stated in Theorem 5.1.
Since we have E [ X ] 2 ≤ E (cid:2) X 2 (cid:3) and Var[ X ] ≤ E (cid:2) X 2 (cid:3) , both estimates in (7) follow from
[Equation]
To obtain this bound, we begin by relating moments of len( N ( M )) to those of the input-output
Jacobian of N at a single point for which we need some notation.
Namely, let us choose a unit speed parameterization of M ; that is, ﬁx a smooth mapping
[Equation]
for which M is the image under γ of the interval [0 , 1] .
That γ has unit speed means that for every t ∈ [0 , 1] we have || γ (cid:48)
( t ) ||
= 1 , where γ (cid:48) ( t ) :
= (cid:0) γ (cid:48)
( t ) , . . .
, γ (cid:48) n 0 ( t ) (cid:1) .
Then, the mapping Γ :=
N ◦ γ (for Γ : R → R n L ) gives a parameterization of the curve N ( M ) .
Note that this parameterization is not unit speed.
Rather the length of N ( M ) is given by
[Equation]
Intuitively, the integrand || Γ (cid:48)
( t ) ||
dt computes, at a given t , the length of Γ([ t, t
+ dt ]) as dt → 0 .
The following Lemma uses (9) to bound the moments of len( N ( M )) in terms of the moments of the norm of the input-output Jacobian J x , deﬁned for any x ∈ R n 0 by
[Equation]
where N i is the i -th component of the network output.
Lemma 6.1.
We have
[Equation]
Sketch of Proof.
Taking powers in (9) and interchanging the expectation and integrals, we obtain
[Equation]
Applying the inequality ab ≤ 12 ( a 2 + b 2 ) , for a, b ∈ R , to the integrand in (12), we conclude
[Equation]
The chain rule yields Γ (cid:48) ( t ) =
J γ ( t ) γ (cid:48) ( t ) .
Substituting this into (13) completes the proof.
Lemma 6.1, while elementary, appears to be new, allowing us to bound global quantities such as length in terms of local quantities such as the moments of || J x u || .
In particular, having obtained the expression (11), we have reduced bounding the second moment of len( N ( M )) to bounding the second moment of the random variable ||
J x u || for a ﬁxed x ∈ R n 0 and unit vector u ∈ R n 0 .
This is a simple matter since in our setting the distribution of weights and biases is Gaussian and hence, in distribution,
|| J x u || is equal to a product of independent random variables:
Proposition 6.2.
For any x ∈ R n 0 and any unit vector u ∈ R n 0 , the random variable ||
J x u || 2 is equal in distribution to a product of independent scaled chi-squared random variables
[Equation]
where the number of degrees of freedom in the (cid:96) th term of the product (for (cid:96) = 1 , .
, L − 1 ) is given by an independent binomial n (cid:96) d = Bin ( n (cid:96) , 1 / 2) with n (cid:96) trials and success probability 1 / 2 .
The number of degrees of freedom in the ﬁnal term is deterministic and given by n L .
This Proposition has been derived a number of times in the literature (see Theorem 3 in Hanin (2018) and Fact 7.2 in Allen-Zhu et al.
We provide an alternative proof based on Proposition 2 in
Hanin & Nica (2019) in the Supplementary Material.
Note that the distribution of || J x u || is the same at every x and u .
Thus, ﬁxing x ∈ R n 0 and a unit vector u ∈ R n 0 , we ﬁnd
[Equation]
To prove (8), we note that E (cid:2) χ 2 n (cid:3) =
n/ 2 and apply Lemma 6.2 to ﬁnd, as desired,
[Equation]
L IMITATIONS AND F UTURE W ORK
We show that deep ReLU networks with appropriate initialization do not appreciably distort lengths and volumes, contrary to prior assumptions of exponential growth.
Speciﬁcally, we provide an exact expression for the mean length distortion, which is bounded above by 1 and decreases slightly with increasing depth.
We also prove that higher moments of the length distortion admit well-controlled upper bounds, and generalize this to distortion of higher dimensional volumes.
We show empirically that our theoretical results closely match observations, unlike previous loose lower bounds.
There are several notable limitations of this paper, which offer promising directions for future work.
First, we prove statements for networks at initialization, and our results do not necessarily hold after training; analyzing such behavior formally would require consideration of the loss function, optimizer, and data distribution.
Second, our results, as stated, do not apply to convolutional, residual, or recurrent networks.
We envision generalizations for networks with skip connections being straightforward, while formulations for convolutional and recurrent networks will likely be more complicated.
In Appendix A, we provide preliminary results suggesting that expected length distortion decreases modestly with depth in both convolutional networks and those with skip connections.
Third, we believe that various other measures of complexity for neural networks, such as curvature, likely demonstrate similar behavior to length and volume distortion in average-case scenarios with appropriate initialization.
While somewhat parallel results for counting linear regions were shown in Hanin & Rolnick (2019; 2020), there remains much more to be understood for other notions of complexity.
Finally, we look forward to work that explicitly ties properties such as low length distortion to improved generalization, as well as new learning algorithms that leverage this line of theory in order to better control inductive biases to ﬁt real-world tasks.
E THICS S TATEMENT
This paper falls within deep learning theory and is intended to advance innovation for more effective and reliable algorithms.
However, insofar as our work shows that deep ReLU networks at initialization are better-behaved than was previously believed, this could encourage the use of deeper neural networks.
While deeper networks may enhance performance in some contexts, larger numbers of parameters are also associated with increased computational cost, which can both increase greenhouse gas emissions and exacerbate inequities caused by differential access to computational resources.  A CKNOWLEDGEMENTS
The authors gratefully acknowledge a range of funding sources that supported this research.
BH would like to acknowledge NSF grants DMS-1855684 DMS-2133806 as well as an NSF CAREER grant DMS-2143754 and an ONR MURI on Foundations of Deep Learning.
DR would like to acknowledge support from the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants program and the Canada CIFAR AI Chairs program
A E XPERIMENTS FOR C ONVOLUTIONAL AND R ESIDUAL N ETWORKS
We here provide preliminary experimental results for the mean length distortion in networks with convolutional layers and skip connections.
In Figure 4(a), we show results for networks with sequential convolutional layers (without pooling layers), initialized as before with weights i.i.d.
normal with variance 2 / fan-in, and a ﬁnal fully connected layer.
We use input dimension n 0 = 16 × 16 × 3 = 768 and output dimension 5.
Our results indicate that the mean length distortion is, as expected, approximately equal to (cid:112) n L /n
≈ 0 .
08 , and that it decays slightly with depth.
In Figure 4(b), we consider residual networks.
Here, the overall network N is deﬁned in terms of residual modules N 1 , N 2 , . . .
, N L and scales η 1 , η 2 , . .
, η L according to:
[Equation]
We set all residual modules N (cid:96) to be two-layer, fully connected ReLU networks.
In keeping with
Hanin & Rolnick (2018), we initialize all weights i.i.d.
normal with variance 2 / fan-in, while setting η 1 = η 2 = · · · = η L = 1 /L .
Our results suggest that mean length distortion again decays modestly overall, with a somewhat sharper decrease for small depths.
layer.
As expected, the mean length distortion is (cid:112) n L /n
≈ 0 .
08 and decays modestly with depth.
(b) shows results for networks with skip connections occurring between even layers (i.e. each residual block is a fully-connected ReLU network of depth 2 ), again taking a line segment of unit length as the input curve.
Here, depth corresponds to the total number of layers in the network, so that a depth-20 network includes 10 residual blocks.
Both plots (a) and (b) show the empirical mean over 100 initializations of the weights and biases.
B E XPERIMENTAL D ETAILS
For all experiments, weights were initialized from i.i.d.
normal distributions with variance 2 / fan-in and bias variance 0 .
We run several experiments that involve computing the length distortion of a given line segment in R n 0 .
We remark on how this was done, for which we rely on the notion of linear regions and bent hyperplanes associated with ReLU networks, explored in Hanin & Rolnick (2020);
Telgarsky (2015).
Speciﬁcally, ReLU networks partition input space into a collection of convex polytopes, which we call linear regions, as (generically) distinct linear functions are deﬁned on each region corresponding to a different subset of the hidden neurons being activated (where a neuron is activated if its pre-activation is nonnegative).
The boundaries of these linear regions are given by sets of points for which a particular neuron has preactivation equal to 0 .
Say we are given a line segment (cid:96) with endpoints p 1 , p 2 ∈ R n 0 parametrized by unit speed on [0 , 1] , given by the set { p 1 + t ( p 2 − p 1 ) : t ∈ [0 , 1] } , for which we aim to compute the length distortion.
Let w (cid:96) =
p 2 − p 1 .
We ﬁrst approximate the intersections of the line segment with linear region boundaries using a binary search subroutine.
Speciﬁcally, initialize the set S =
5 , 1] , which will contain the parameter values for these approximations.
For each iteration of the subroutine, we run the following procedure: for any three consecutive points t 1 , t 2 , t 3 ∈ S , let x 1 = p 1 + t 1 ( p 2 − p 1 ) , x 2 = p 1 + t 2 ( p 2 − p 1 ) , x 3 = p 1 + t 3 ( p 2 − p 1 ) , and consider the values ||N ( x 2 ) −N ( x 1 )
||
|| x 2 − x 1 || and ||N ( x 3 ) −N ( x 2 )
||
|| x 3 − x 2 || .
These values are equal if the three points are in the same linear region, in which case we eliminate tand t 3 from S .
If not equal, then there exists a linear region boundary in the segment from x 1 to x 3 ; we add the points t 1 + t 2and t 2 + t 3to S .
We iterate this procedure (15 times in our implementation); for the ﬁnal iteration, we ensure that only the last point associated with a given linear region is in S .
Now take the set S =
[ t 1 , t 2 , . . . , t n ] returned by the binary search procedure; we proceed to compute the parameter values denoting the exact intersections of the line segment with region boundaries, which we shall store in S ∗ .
For consecutive points t i , t i +1 ∈ S , i = 1 , . . . , n − 1 , determine the set of activated neurons for both points at x i
= p 1 + t i ( p 2 − p 1 ) , x i +
= p 1 + t i +1 ( p 2 − p 1 ) , and ﬁnd the neuron at which these sets differ; we solve a linear equation to determine the value t ∗ ∈ [0 , 1] for which this neuron is 0 , and replace t i with the exact value t ∗ i in S ∗ .
Finally, we append 0 and 1 to the respective ends of S ∗ =
[ t ∗ 0 = 0 , t ∗ 1 , . . .
, t ∗ n = 1] .
Computing the length distortion is reduced to summing the lengths of the output segments corresponding to consecutive pairs t i , t i +1 ∈ S ∗ .
Namely, for each such pair, the network is given by a single linear function on the segment of inputs between x i = p 1 + t i ( p 2 − p 1 ) and x i +
= p 1 + t i +1 ( p 2 − p 1 ) .
We calculate the weight matrix
W i of this linear function – the length of the corresponding output segment is the product of W i with the vector w (cid:96) .
The total length distortion is then given by the sum
[Equation]
C P ROOF OF T HEOREM 5.
Our ﬁrst step in proving Theorem 5.1 is to obtain bounds on the moments of len( N ( M )) in terms of the input-output Jacobian of N at a single point, which we recall was deﬁned in (10).
To accomplish them, we recall the notation from §6.
Namely, ﬁx a smooth unit speed parameterization of M = γ ([0 , 1]) with γ : R → R n 0 , γ ( t ) =
( γ 1 ( t ) , . . .
, γ n 0 ( t )) .
The mapping Γ :=
N ◦ γ, Γ : R → R n L gives a parameterization of the curve N ( M ) , and we have
[Equation]
Let us note an important but ultimately benign technicality.
The Jacobian J x of the map x (cid:55)→ N ( x ) is not deﬁned at every x (namely those x where some neuron turns from on to off).
Thus, a priori , Γ (cid:48) ( t ) exists only at those t for which J γ ( t ) exists.
However, the formula (14) is still valid.
Indeed, for any setting of weights and biases of N the map Γ is Lipschitz.
Thus, by Rademacher’s theorem, Γ (cid:48) ( t ) exists for almost every t ∈ [0 , 1] and the length of the curve is given by integrating the norm of this almost-everywhere deﬁned derivative.
The following simple Lemma is a generalization of
Lemma 6.1 and allows us to bound all moments of len( N ( M )) in terms of the moments of the norm of the Jacobian vector product
|| J x u || .
Lemma C.1.
For any integer m ≥ 1 , we have
[Equation]
Proof.
Taking powers in (14) and using Tonelli’s theorem to interchange the expectation and integrals, we obtain
[Equation]
To proceed, let us recall a special case of the power mean inequality which says that for any a i ≥we have
[Equation]
Applying this to the integrand in (16), we conclude
[Equation]
Next, ﬁx t ∈ [0 , 1] .
For any neuron z in N , denote by x (cid:55)→ z ( x ) its pre-activation.
Note that P ( z ( γ ( t )) =
=
since our bias variance C b is set to some ﬁxed positive constant and hence the bias of each neuron has a continuous density.
Therefore, with probability 1 over the randomness in the weights and biases of N , there exists a neighborhood U of γ ( t ) on which z ( x ) has constant sign for all x ∈ U .
The Jacobian J γ ( t ) is therefore well-deﬁned and the chain rule yields
[Equation]
Substituting this into (17) completes the proof.
Having obtained the expression (15), we have reduced bounding the moments of len( N ( M )) to bounding the moments of the random variable
||
J x u || for a ﬁxed x ∈ R n 0 and unit vector u ∈ R n 0 .
Prior work (e.g. Thm. 1 in Hanin & Nica (2019)) shows that these moments satisfy
[Equation]
provided
[Equation]
Substituting these moment estimates in (15) completes the derivation of (4).
However, the results in
Hanin & Nica (2019) are subtle because they apply to any distribution of weights and biases.
They also give the slightly sub-optimal restriction (19) that m 2 must be smaller than a constant times the minimum of the n (cid:96) ’s.
In the special case where the distribution of weights and biases is Gaussian, we can do slightly better by computing the moments of || J x u || more directly (note that in the statement of Theorem 5.1, we required only that m is smaller than a constant times the minimum of the n (cid:96) ’s).
We will need this anyway to derive the slightly more reﬁned estimates in (2) and (3).
Let us therefore check that, in distribution,
||
J x u || is equal to a product of independent random variables:
Proposition C.2.
For any x ∈ R n 0 and any unit vector u ∈ R n 0 , the random variable ||
J x u || 2 is equal in distribution to a product of independent scaled chi-squared random variables
[Equation]
where the number of degrees of freedom in the (cid:96) th term of the product (for (cid:96) = 1 , .
, L − 1 ) is given by an independent binomial
[Equation]
with n (cid:96) trials and success probability 1 / 2 .
The number of degrees of freedom in the ﬁnal term is deterministic and given by n L .
Proof.
Consider a ReLU network N with input dimension 1 , output dimension n L , and hidden layer widths n 1 , . .
. , n L − 1 .
Suppose the weight matrices W ( (cid:96) ) and bias vectors b ( (cid:96) ) are independent with i.i.d.
Gaussian components:
[Equation]
where C b > 0 is any ﬁxed constant.
For a ﬁxed network input x ∈ R n 0 , with probability 1 , the input-output Jacobian J x is well-deﬁned.
Moreover, it can be written as
[Equation]
where W ( (cid:96) ) is the matrix of weights from layer (cid:96) − 1 to layer (cid:96) and D ( (cid:96) ) is an n (cid:96) × n (cid:96) diagonal matrix:
[Equation]
whose diagonal entries are 0 or 1 depending on whether the pre-activation z (
[Equation]
i of neuron i in layer (cid:96) is positive at our ﬁxed input x. Next, according to Proposition 2 in Hanin & Nica (2019), the marginal distribution of each D ( (cid:96) ) is that its diagonal entries are independent Bernoulli (1 / 2) random variables.
Moreover, we have the following equality in distribution:
[Equation]
where D ( (cid:96) ) are independent of each other (and of W ( (cid:96) ) ) resampled from the marginal distribution of D ( (cid:96) ) and η is an independent diagonal matrix with independent diagonal entries that are ± 1 with equal probability.
In particular, for a ﬁxed unit vector u ∈ R n 0 , we have
[Equation]
We may rewrite this as
[Equation]
for u (1) :
=
D
[Equation]
1) W [Equation] 1) u ||D (1) W (1) u || , where we interpret the expression (20) as equal to 0 if D (1) is the zero matrix.
To complete the proof, we need the following standard observation.
Lemma C.3.
Suppose W is an n × n (cid:48) matrix with i.i.d.
Gaussian entries and u is a random unit vector in R n (cid:48) that is independent of W but otherwise has any distribution.
Then Wu is independent of u and is equal in distribution to Wv where v is any ﬁxed unit vector in R n (cid:48) .
Proof.
For any ﬁxed orthogonal matrix O , it is a standard fact that W O is equal in distribution to
W .
Thus, for any measurable sets A ⊆ R n and B ⊆ R n (cid:48) , since u, W are independent we have
[Equation]
where P u ( u 0 ) is the distribution of u. Fix any u 0 ∈ S n (cid:48) − 1 and let O
= O ( u 0 ) be any orthogonal matrix so that u
= O e 1 with e 1 = (1 , 0 , . . .
, 0) is the ﬁrst standard unit vector.
Then, since W O is equal in distribution to W , we have
[Equation]
which is independent of u 0 .
We therefore ﬁnd
[Equation]
We
[Equation]
as desired.
We are now in a position to complete the proof of Proposition C.2.
Combining Lemma C.3 with (20), we ﬁnd that, in distribution
||
J x u
|| equals
[Equation]
Note that these two terms are independent.
Repeating this argument, we obtain that
|| J x u || is equal in distribution to the product
[Equation]
The terms in this product are independent.
To complete the proof note that for (cid:96) = 1 , . .
L −the number of non-zero entries in D ( (cid:96) ) is a binomial random variable n (cid:96) with n (cid:96) trials, each with probability of success 1 / {+
+}
and that
[Equation]
is precisely 2 /n (cid:96) times a sum of n (cid:96) squares of independent standard Gaussians.
Thus, for (cid:96) = 1 , . . .
, L − 1 ,
[Equation]
Similarly
[Equation]
Substituting (23) and (24) into (22) completes the proof.
Evaluating the moments of len( N ( M )) is now a matter of computing the moments of some scaled chi-squared random variables with a random number of degrees of freedom.
For instance, recalling that
[Equation]
and applying Lemma C.2 as well as the tower property of the expectation we ﬁnd
[Equation]
Substituting this into (15) with m = 2 yields
[Equation]
yielding (3).
To prove (2), we need to estimate E [len N ( M )] .
By taking expectations in (14) and using (18), we ﬁnd
[Equation]
where we’ve used that by Lemma C.
[Equation]
the distribution of || J x u || is the same for every x ∈ R n 0 and every unit vector u .
Moreover, again using Lemma C.
[Equation]
we see that E
[ || J x u || ] equals
[Equation]
To simplify this, a direct computation shows that
[Equation]
where Γ( · ) is the Gamma function.
Next, for any positive random variable X with E [ X ] = 1 we have
[Equation]
Recall that
[Equation]
Using this and the law of total variance yields
[Equation]
Thus
[Equation]
we ﬁnd that E [len( N ( M ))] equals
[Equation]
as claimed.
Finally, let us check the higher moment estimates (4).
By Lemma C.2 we have the following estimate
[Equation]
where we used that x = x − 1 + 1 ≤ e x − 1 .
For any m , we have
[Equation]
Therefore, || J x u || 2 m is bounded above by
[Equation]
n
[Equation]
(cid:35)
[Equation]
Finally, for any ﬁxed positive integer n we may write
[Equation]
where ξ k are independent Bernoulli (1 / 2) random variables and Z k are independent standard Gaussians.
A direct computation shows that the centered random variables ξ k Z 2 k − 12 are sub-exponential
SE ( ν 2 , α ) with some universal parameters ν 2 , α .
Thus, by the stability of sub-exponential random variables under summation we have
[Equation]
Again using this property we conclude
[Equation]
where n ∗ = min { n 1 , . .
. , n L − 1 } .
Therefore, by (25), we ﬁnd that
[Equation]
where Y ∈ SE (cid:16) 4 ν 2 (cid:80) L − 1 (cid:96) =1 1 n
[Equation]
, 2 αn ∗ (cid:17) .
By deﬁnition of sub-exponential random variables, we have
[Equation]
provided m < n ∗ / 2 α for some universal constant c > 0 .
This completes the derivation of (4) and completes the proof of Theorem 5.1.
[Equation]
D P ROOF OF T HEOREM 5.
The proof of Theorem 5.2 follows closely the argument for Theorem 5.1.
For a given input x ∈ R n 0 , we will continue to write
[Equation]
for the input-output Jacobian of the network function, which exists for Lebesgue almost every x ∈ R n 0 .
We will write Π
T x M : R n 0 → T x M for the orthogonal projection onto this tangent space.
We have
[Equation]
where vol d is the d − dimensional Hausdorff measure.
Indeed, the integrand measures, at each x ∈
M , the volume of the image of an inﬁnitesimal cube on M of volume vol d ( dx ) centered at x under the map x (cid:55)→ N ( x ) .
Arguing precisely as in the proof of Lemma C.1, we ﬁnd that for any integer
[Equation]
Fix x ∈ M and denote by e
[Equation]
[Equation] x ) an orthonormal basis of the tangent space of M .
Then, by the Gram identity, we may write
[Equation]
27) where we recall that the wedge product is the anti-symmetrization of the tensor product.
Just as in the proof of Theorem 5.1, the key observation is that for Gaussian weights we have that
|| J N ( x ) e 1 ( x ) ∧ · · · ∧ J N ( x ) e d ( x ) || is a product of i.i.d.
random variables.
The formal statement is the following:
Proposition D.1.
For any x ∈ R n 0 and collection of orthonormal unit vectors u 1 , . . .
, u d ∈ R n 0 , the random variable
[Equation]
is equal in distribution to a product of independent scaled chi-squared random variables:
[Equation]
where all the terms in the product are independent and for (cid:96)
= 1 , . .
, L − 1 we’ve written n (cid:96) for independent Binomial random variables:
[Equation]
with n (cid:96) trials and success probability 1 / 2 .
Proof.
The proof is identical to that of Lemma C.2.
The only difference is that we must invoke the following ampliﬁcation of Lemma C.3:
Lemma D.2.
Let u 1 , . .
u k ∈ R n (cid:48) be a collection of orthonormal vectors (i.e. a collection) with any distribution.
Let W be an independent n × n (cid:48) matrix with i.i.d.
Gaussian entries.
Then
[Equation]
is independent of u 1 ∧ · · · ∧ u k and is equal in distribution to Wv 1 ∧ · · · Wv k where v 1 , . .
. , v k is any ﬁxed collection of orthonormal vectors.
Proof.
The proof is identical to that of Lemma C.3, except that we note that for that, given u 1 , . .
, u k , there exists an orthogonal matrix
O =
O ( u 1 , . .
u k )
so that
[Equation]
where e j are the standard basis vectors.
With Lemma D.1 in hand, we complete the proof of Theorem 5.2 as follows.
First, note that for any random variable X we have
[Equation]
Hence, Theorem 5.2 will follow once we show that
[Equation]
Combining Proposition D.1 with (26) and (27), this estimate follows by showing that
[Equation]
To check this, recall that
[Equation]
Hence,
E
[Equation]
where in the last line we used that 1 + x ≤ e x .
This completes the proof.
(cid:3)