Deep Mean Field Theory: Variance and Width Variation by Layer as Methods to Control Gradient Explosion
Anonymous authors
Paper under double-blind review
Abstract
A recent line of work has studied the statistical properties of neural networks to great success from a mean ﬁeld theory perspective, making and verifying very precise predictions of neural network behavior and test time performance.
In this paper, we build upon these previous works to explore two methods for taming the behaviors of random residual networks (with only fully connected layers and no batchnorm).
The ﬁrst method is width variation (WV) , i.e. varying the widths of layers as a function of depth.
We show that width decay reduces gradient explosion without aﬀecting the mean forward dynamics of the random network.
The second method is variance variation (VV) , i.e. changing the initialization variances of weights and biases over depth.
We show VV, used appropriately, can reduce gradient explosion of tanh and ReLU resnets from exp(Θ( √ L )) and exp(Θ( L )) respectively to constant Θ(1).
A complete phase-diagram is derived for how variance decay aﬀects diﬀerent dynamics, such as those of gradient and activation norms.
In particular, we show the existence of many phase transitions where these dynamics switch between exponential, polynomial, logarithmic, and even constant behaviors.
Using the obtained mean ﬁeld theory, we are able to track surprisingly well how VV at initialization time aﬀects training and test time performance on MNIST after a set number of epochs: the level set of test/train set accuracies coincide with the level sets of certain gradient norms.
Introduction
Deep mean ﬁeld theory studies how random neural networks behave with increasing depth, in the limit as widths go to inﬁnity.
Some of the landmark results include precise predictions of norms of activation vectors across depth in feedforward (Poole et al., 2016) and residual networks 1 (Yang and Schoenholz, 2017), correlations between corresponding activation vectors computed from two input vectors (Poole et al., 2016; Yang and Schoenholz, 2017), and gradient norms (Schoenholz et al., 2016; Yang and Schoenholz, 2017).
The latter is used to remarkably predict test time performance given only initialization variances (Schoenholz et al., 2016; Yang and Schoenholz, 2017).
Overall, a program emerged toward building a mean ﬁeld theory for state-of-the-art neural architectures as used in the wild, so as to provide optimal initialization parameters quickly for any neural network user.
In this paper, we contribute to this program by studying how width variation (WV), as practiced commonly, can change the behavior of quantities mentioned above, with gradient norm being of central concern.
We ﬁnd that WV can dramatically reduce gradient explosion without aﬀecting the mean dynamics of forward computation, such as the activation norms, although possibly increasing deviation from the mean in the process (Section 4).
We also study a second method, variance variation (VV), for mean ﬁeld dynamics of a random neural network (Section 5).
In this paper, we focus on its application to tanh and
ReLU residual networks, where we show that VV can dramatically ameliorate gradient explosion, and in the case of ReLU resnet, activation explosion 2 .
Previous works (Poole et al., 2016; Schoenholz et al., 2016; Yang and Schoenholz, 2017) have focused on how network architecture and activation functions aﬀect the dynamics of mean ﬁeld quantities, subject to the constraint that initialization variances and widths are constant across layers.
In each combination of (architecture, activation), the mean ﬁeld dynamics remain the same asymptotically regardless of the variances.
For example, tanh feedforward networks have exp(Θ( l )) forward and backward dynamics, while tanh residual networks have poly ( l ) forward and exp(Θ( √ l )) backward dynamics.
Such asymptotics were considered characteristics of the (architecture, activation) combination (Yang and Schoenholz, 2017).
We show by counterexample that this perception is erroneous .
In fact, as discussed above,
WV can control the gradient dynamics arbitrarily and VV can control forward and backward dynamics jointly, all without changing the network architecture or activation.
To the best of our knowledge, this is the ﬁrst time initialization schemes for reducing gradient explosion or vanishing have been proposed that vary variance and/or width across layers.
Using the derived mean ﬁeld theories of VV, we make stunning predictions of test time performances of grids of resnets.
Preliminaries
Notations and Settings.
We adopt the notations of Yang and Schoenholz (2017) and review them brieﬂy.
Consider a vanilla feedforward neural network of L layers, with each layer l having N ( l ) neurons; here layer 0 is the input layer.
Let x (0) =
( x (0)1 , . . .
, x (0) N (0) ) be the input vector to the network, and let x ( l ) for l > 0 be the activation of layer l .
Then a neural network is given by the equations x ( l ) i =
φ ( h ( l ) i ) , h ( l ) i = (cid:80) N ( l − 1) j = w ( l ) ij x ( l − 1)
j + b ( l ) i where (i) h
( l ) is the pre-activation at layer l , (ii) w ( l ) is the weight matrix, (iii) b ( l ) is the bias vector, and (iv) φ is a nonlinearity, for example tanh or ReLU, which is applied coordinatewise to its input.
To lighten up notation, we suppress the explicit layer numbers l and write x i = φ ( h i ) , h i =
(cid:80) Nj =
1 w ij x j + b i where • implicitly denotes • ( l ) , and • denotes • ( l − 1) (and analogously, • denotes • ( l +1) ).
When the widths are constant N ( l ) =
N ( m ) , ∀ m, l , residual network (He et al., 2015; 2016) adds an identity connection or skip shortcut that “jumps ahead” every couple layers.
We adopt one of the simpliﬁed residual architectures deﬁned in
Yang and Schoenholz (2017) for ease of analysis 3 , in which every residual block is given by
[Equation]
where M ( l ) is the width of the “hidden layer” of the residual block, ( v ( l ) ij ) N ( l ) ,M ( l ) i,j = is a new set of weights and ( a ( l ) i ) N ( l ) i =1 is a new set of biases for every layer l .
If we were to change the width of a residual network, as is done in practice, we need to insert “projection” residual blocks (He et al., 2015; 2016)
every couple layers.
We assume the following simpliﬁed projection residual block in this paper, for the ease of presentation 4 :
[Equation]
j = w ij x j
[Equation]
b i , y
[Equation]
=
N (cid:88) j =
[Equation]
where y =
( y i ) N
( l ) i
=1 is a “projection” of x = ( x i ) N
( l − 1)
i = 5 , and ( π ij ) N ( l ) ,N ( l − 1) i,j = is the “projection” matrix.
Note that we only consider fully-connected aﬃne layers instead of convolutional layers.
Deep mean ﬁeld theory is interested in the “average behavior” of these network when the weights and biases, w ( l ) ij , b ( l ) i , π ( l ) ij , v ( l ) ij , and a ( l ) i are sampled i.i.d.
from Gaussian distributions.
Following standard practice, they respectively have standard deviations σ ( l ) w / √ N
( l − 1) , σ ( l ) b , σ ( l ) π / √ N ( l − 1) , σ ( l ) v / √ M ( l ) , and σ ( l ) a ; here we normalize the variance of weight parameters so that, for example, the variance of each h i is σ 2 w , assuming each x j is ﬁxed.
While previous works have all focused on ﬁxing σ ( l ) • to be constant across depth, in this paper we are interested in studying varying σ ( l ) • .
In particular, other than σ ( l ) π which we ﬁx at 1 across depth (so that “projection” doesn’t act like an “expansion” or “contraction”), we let σ ( l ) • =
σ • l − β • for each • = v, w, a, b , where σ • , β • are constant w.r.t.
l .
Hereafter the bar notation • , • , • do not apply to σ s, so that, by σ a , for example, we always mean the constant σ a .
Statistical Assumptions.
We make several key “mean ﬁeld” assumptions, which were formulated in its entirety ﬁrst in Yang and Schoenholz (2017)
(though Axiom 2(a) has been stated ﬁrst by Schoenholz et al.
While these assumptions may be mathematically unsatisfying, identifying them and discovering that they lead to highly precise prediction is in fact one of the most important contributions of deep mean ﬁeld theory.
Axiom 1 (Symmetry of activations and gradients) .
(a) We assume (cid:104) ( h ( l ) i ) 2 (cid:105) =
(cid:104) ( h ( l ) j ) 2 (cid:105) and (cid:104) ( x (0) i ) 2 (cid:105) =
(cid:104) ( x (0) j ) 2 (cid:105) for any
i, j, l .
(b) We also assume that the gradient ∂E/∂x ( l ) i with respect to the loss function E satisﬁes (cid:104) ( ∂E/∂x ( l ) i ) 2 (cid:105) =
(cid:104)
( ∂E/∂x ( l ) j ) 2 (cid:105) for any i, j, l .
Axiom 2 (Gradient independence) .
(a) We assume the we use a diﬀerent set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d.
from the same distributions.
(b) For any loss function E , we assume that the gradient at layer l , ∂E/∂x ( l ) i , is independent from all activations h ( l ) j and x ( l − 1) j from the previous layer.
One can see that Axiom 1(a) is satisﬁed if the input x (0) ∈ {± 1 } N and Axiom 1(b) is satisﬁed if Axiom 2 below is true and the gradient at the last layer ∂E/∂xL ∈ {± 1 } N .
Axiom 2(a) was ﬁrst made in Schoenholz et al.
(2016) for computing the mean ﬁeld theory of gradients for feedforward tanh networks.
This is similar to the practice of feedback alignment Lillicrap et al.
Mean Field Quantities.
Now we deﬁne the central quantities studied in this paper.
Deﬁnition 2.1.
Fix an input x (0) .
Deﬁne the length quantities q ( l ) :=
(cid:104) ( h ( l ) 1 ) 2 (cid:105) and p ( l ) := (cid:104) ( x ( l ) 1 ) 2 (cid:105) for l > 0 and p (0) =
(cid:107) x (0) (cid:107) 2 /N .
Here the expectations (cid:104)•(cid:105) are taken over all random initialization of weights and biases for all layers l , as N → ∞ (large width limit).
Note that in our deﬁnition, the index 1 does not matter by Axiom 1.
Deﬁnition 2.2.
Fix two inputs x (0) and x (0) (cid:48) .
We write • (cid:48) to denote a quantity • with respect to the input x (0) (cid:48) .
Then deﬁne the correlation quantities γ ( l ) :
= (cid:104) h ( l ) 1 h ( l ) 1 (cid:48) (cid:105) and λ ( l ) :
= (cid:104) x
( l ) 1 x ( l ) 1 (cid:48)
(cid:105) for l > 0 and γ (0) =
x (0) · x (0) (cid:48) /N , where the expectations (cid:104)•(cid:105) are taken over all random initialization of weights and biases for all layers l , as N → ∞ (large width limit).
Again, here the index 1 does not matter by Axiom 1.
Additionally, deﬁne the cosine distance quantities e ( l ) :
= γ ( l ) / (cid:112) p ( l ) p ( l ) (cid:48) and c ( l ) :
= λ ( l ) / (cid:112) q ( l ) q ( l ) (cid:48) .
Deﬁnition 2.3.
Fix an input x (0) and a gradient vector ( ∂E/∂x ( L ) i ) i of some loss function
E with respect to the last layer x ( L ) .
Then deﬁne the gradient quantities (cid:107) ( l ) :=
(cid:104)
( ∂E/∂x ( l ) 1 ) 2 (cid:105) , χ ( l ) • := (cid:104) ( ∂E
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Here the expectations are taken with Axiom 2 in mind, over both random initialization of forward and backward weights and biases, as N → ∞ (large width limit).
Again, the index 1 or 11 does not matter by Axiom 1.
Asymptotic notations.
The expressions f
= O ( g ) ⇐⇒ g
= Ω( f ) have their typical meanings, and f = Θ( g )
iﬀ
f
=
O
( g ) ,
g = O ( f ).
We take f ( x ) =
˜ O ( g ( x ))
⇐⇒ g
( x ) = ˜Ω( f ( x )) to mean f ( x )
=
O ( g log k x ) for some k ∈ Z (this is slightly diﬀerent from the standard usage of ˜ O ), and f = ˜Θ( g ) ⇐⇒ f
= ˜
O ( g ) & g =
˜ O
( f ) .
All asymptotic notations are sign-less, i.e. can indicate either positive or negative quantities, unless stated otherwise.
Theory
We recall integral transforms from Yang and Schoenholz (2017):
Deﬁnition 3.1.
Deﬁne the transforms V and W by V φ ( µ, ρ ) :
=
E[ φ ( z ) 2 : z ∼ N ( µ, ρ )] and
W φ ( µ, ρ, ρ (cid:48) , ν ) :
[Equation]
E[
[Equation]
ν ν
[Equation] [Equation]
Table 1 summarizes the equations governing the dynamics of various mean ﬁeld quantities when we vary variances and/or widths; they are proved similarly to how corresponding recurrences were proved in Yang and Schoenholz (2017); Schoenholz et al.
Width Variation
Width variation is ﬁrst passingly mentioned in Schoenholz et al.
as a potential way to guide gradient dynamics for feedforward networks.
We develop a complete mean ﬁeld theory of WV for residual networks.
Via Table 1 and Thm A.2, we see that width variation (WV) has two kinds of eﬀects on the mean gradient norm: when compared to no width variation, it can multiply the squared gradient norm of biases b i or weights w ij by N/M (which doesn’t “stack”, i.e. does not aﬀect the squared gradient norm of lower layers), or it can multiply the squared gradient norm of x i by N/N (which “stacks”, in the sense above, through the dynamics of (cid:107) ).
We will focus on the latter “stacking” eﬀect and assume N ( l ) =
M ( l )
Suppose from layer l to layer m , (cid:107) ( m ) rises to r times (cid:107) ( l ) .
If we vary the width so that N ( m − 1) is rN ( m ) , then this gradient expansion is canceled, and (cid:107) ( m − 1) =
( N ( m ) /N
( m − 1) )( σ 2 v σ 2 w l − β v − β w V ˙ φ (0 , q ( m ) ) + 1) (cid:107) ( m ) =
( σ 2 v σ 2 w l − β v − β w V ˙ φ (0 , q ( m ) )
+ 1) (cid:107) ( l ) so that it is as if we restarted backpropagation at layer m .
Remarkably, changing the width does not change the mean ﬁeld forward dynamics (for example the recurrences for p, q, γ, λ remain the same)
(Thm A.1).
But, as always, if we reduce the width as part of WV (say, keeping N (0) the same but reducing the widths of later
80 layer l     p : WV vs no WV no WV
WVtheoryno WV WV    80 layer l 0.
e : WV vs no WV no WV WV e (0) =
0.01 no WV WV e (0) =0.
0 20 40 60layer l , reversed 10 − 10 − 10 − 10101010gradients: WV vs no WV ℸ ( l )
χ b χ w χ
v ℸ ( l ) WV χ b WV χ w WV χ v WV layers), the variance of the sampled dynamics will also increase; if we increase the width as part of WV (say, keeping N ( L ) the same but increasing the widths of earlier layers), the variance of the sampled dynamics will decrease.
We can apply this theory of WV to tanh residual networks ( φ = tanh in Table 1) without
VV.
By Yang and Schoenholz (2017), tanh residual networks with all β • = 0 have gradient dynamics log( (cid:107) ( m ) / (cid:107) ( l ) )
=
A ( √ l − √ m ) + Θ(log l − log m )
where A = 43 (cid:113) 2 π σ 2 v σ w √ σv + σ 2 a .
If we place projection blocks projecting N ( l − 1) (cid:55)→ N ( l ) / exp( A ) at layer l = n 2 , for n = 1 , 2 , . .
, then the gradient norms would be bounded (above and below) across layers, as reasoned above.
Indeed, this is what we see in Fig.
The rightmost subﬁgure compares, with log scale y-axis, the gradient dynamics with no WV to that with WV as described above, and we see that our theory tracks the mean gradient dynamics remarkably precisely for both the WV and the no WV cases, and indeed, WV eﬀectively cap the gradient norm for l ≥(where WV is applied).
The left two ﬁgures show the forward dynamics of p and e , and we see that the WV does not aﬀect the mean dynamics as predicted by theory.
However, we also see dramatic increase in deviation from the mean dynamics at every projection layer in the forward case, while the deviation from the mean is constant across depth, but nevertheless is much larger than that in the case with no WV.
Therefore, width variation is unique in a few ways among all the techniques discussed in the mean ﬁeld networks literature so far, including variance decay as studied below, adding skip connections, or changing activation functions: 1.
It can ameliorate or suppress altogether gradient explosion (or vanishing) problems without aﬀecting the mean forward dynamics of p, q, λ, γ, c, e . 2.
To do so, it has to choose a trade oﬀ from the following spectrum:
At one end, we truncate neurons from the original network (say, keeping N (0) the same), so that we have fewer parameters, faster training, but larger deviations from the mean dynamics.
At the other, we add neuron to the original network (say, keeping N ( L ) the same), so that we have more parameters, slower training, and smaller deviations from the mean dynamics.
Variance Variation
As discuss in Section 4, WV is essentially a post-hoc technique to tweak an existing gradient dynamic without changing the forward dynamics.
Thus in this section we assume all widths are constant, N ( l ) =
N ( m ) =
M ( n ) for any m, l, n , so that WV can be applied as a “touch-up” if necessary.
10101010 3 layer l 101010β v =0.1, β w =0.1, β a =0.1, β b =0.1; β =0.2, α =0.
log ℸ (0) /
ℸ ( l ) δ 1− β l 1− β 10101010 3 layer l 1010β v =0.5, β w =0.5, β a =0.5, β b =0.5; β =1.0, α =0.
ℸ (0) /
ℸ ( l ) l δ 10101010 3 layer l 10β v =0.8, β w =0.8, β a =0.5, β b =0.5; β =1.6, α =0.
ℸ (0) /
ℸ ( l )
5. Residual Network with ReLU
It was shown in Yang and Schoenholz (2017) that, with no variance decay, in an ReLU resnet, both the mean squared activation norm ( p ) and the mean squared gradient norm ( (cid:107) ) explode exponentially with depth, and this causes training to fail for even 100 layer networks.
We show that this problem is in fact extremely easy to ﬁx, requiring no architectural changes at all, only that β • be increased from 0 so that the randomization variances decay across depth (Thms A.6 and A.8).
Gradient quantities.
The main driver of this gradient molliﬁcation is β :=
β v + β w .
When β ∈ [0 , 1), the gradient norm varies like (cid:107) (0) / (cid:107) ( l ) =
exp(Θ( l 1 − β )); when β =
this recapitulate the exponential behavior derived in Yang and Schoenholz (2017).
When β = 1, it experiences a sharp phase transition, where now (cid:107) (0) / (cid:107) ( l ) =
poly ( l ).
As β becomes larger than 1, (cid:107) (0) / (cid:107) ( l ) = Θ(1), all bounded!
Fig.
2 veriﬁes this result empirically, and in fact show that our computed asymptotic expansions in Thm A.8 are highly accurate predictors.
It is both interesting and important to note that the gradient norms for actual trainable parameters, such as χ w and χ v , are aﬀected diﬀerently by β .
In fact, χ (1) w /χ ( l ) w is bounded (!) with l when β < 1 (the β = 0 case is already observed in Yang and Schoenholz (2017)) but phase transitions to poly ( l ) for β ≥ 1, while χ (1) v /χ ( l ) v is already poly ( l ) when β < 1, and remains so as β is increased to > 1..
Curiously, χ v turns out to be more predictive of performance than (cid:107) in the β > 1 regime, in a surprising way; see Section 6.1.
Length quantities.
Similarly, β is the primary conduit for mollifying the behavior of squared activation norms p and q (Thm A.6).
Like the gradient dynamics, p = exp(Θ
( l 1 − β ))
when β < 1; when β
= 0 this recapitulates the results of Yang and Schoenholz (2017).
As β rise to 1 and above, p experiences a phase transition into polynomial dynamics, but unlike the case of (cid:107) , it is not constant when β > 1.
Instead, a diﬀerent parameter, α := min( β v + β b , β a ) drives the asymptotics of p in the β > 1 regime.
When α ∈ [0 , 1) is small, p grows like Θ( l 1 − α ).
The instant α hits 1, p is just logarithmic, p = Θ(log l ).
As α shoots past 1, p becomes constant.
Thus the dynamics of p is governed by a zigzag through ( β • ) • = v,w,a space.
Fig.
3 goes through each of the ﬁve cases discussed above and veriﬁes that our asymptotics are correct.
Cosine distance.
e = γ/p measures how well the input space geometry (angles, in this case) is preserved as the input space is propagated through each layer.
Its dynamics are much simpler than those of p and (cid:107) above.
If e (0) = 1, then e ( l ) = 1 for all l trivially.
If e (0) < 1, then we have one of the following two cases, (cid:136)
If β ≤ 1 or α ≤ 1, then e →irrespective of initial data p (0) and γ (0) .
(cid:136)
If β > 1 and α > 1, then e converges to a ﬁxed point e ∗ < 1 which depends on the initial data p (0) and γ (0) , at a rate of Θ( l 1 − α ).
Thus
10101010 3 layer l 10 − 10 − 101010β =0.2, α =0.1 β v = β w =0.1, β a =0.1, β b =0.
log p δ 1− β l 1− β log p − δ 1−
β
l 1− β l 1−2 β 10101010 3 layer l 10 − 1010β =
1.0, α =0.5 β v = β w =0.5, β a =0.5, β b =0.
p /log
l l δ Δ p /log
l l δ − 10101010 3 layer l 10 − 101010β =1.6, α =0.5 β v = β w =0.8, β a =0.5, β b =0.
pl 1− α Δ
pl − α 10101010 3 layer l 10 − 10 − 10 − 10β =1.6, α =1 β v = β w =0.8, β a =1, β b =0.
p /log
l 1Δ pl − 10101010 3 layer
l 10 − 10 − 10 − 10 − 1010β =
1.6, α =1.3 β v = β w =0.8, β a =1.5, β b =0.
p 1Δ pl − α
ReLU very much likes to collapse the input space into a single point ( e = 1 means every two input vectors get mapped to the same output vector), and the only way to prevent this from happening is to make the β • s so high, that higher layers barely modiﬁes the computation done by lower layers at all.
Indeed, the second condition β > 1 and α > 1 ensures that p = Θ(1) as discussed above (Thm A.6), so that as l → ∞ , layer l ’s residual adds to x ( l − 1) only a vector of vanishing size compared to the size of x ( l − 1) .
5. Residual Network with Tanh
While ReLU resnet depends heavily on β = β v + β w and α = min( β v + β b , β a ), ˆ µ p :=
min( β v , β a ) and ˆ ν p :=
1 − β w + ˆ µ p − 2 β v are the key quantity determining the dynamics in the case of tanh resnet, with β playing a minor role.
We will study tanh resnets in the setting where q → ∞ ; otherwise, p is bounded, meaning that higher layers become essentially unused by the neural network (similar to the discussion made in Section 5.1(Cosine distance) above).
In this setting, it can be shown that ˆ µ p ≤ 1 (Thm A.3).
Gradient quantities.
By Thm A.4, as long as ˆ µ p stays below 1, the asymptotics of (cid:107) is entirely governed by ˆ ν p , which is 1 when all β • s are 0 and most of the time decreases as β • s are increased.
When ˆ ν p > 0, (cid:107) (0) / (cid:107) ( l ) =
exp(Θ( l 12 ˆ ν p )); the results of Yang and Schoenholz (2017) are recovered by setting all β • s to 0, thus ˆ ν p = 1 and (cid:107) (0) /
(cid:107) ( l ) =
exp(Θ( l 12 )).
When ˆ ν p hits 0, (cid:107) (0) / (cid:107) ( l ) becomes polynomial, and as ˆ ν p dips below 0, gradient expansion becomes bounded.
If ˆ µ p = 1, (cid:107) (0) / (cid:107) ( l ) is automatically suppressed to be subpolynomial.
The only minor phase transition here is going from β = 1 to β > 1 (and β cannot be less than 1 by our assumption that q → ∞ ).
In the former case, the gradient expansion is exp(Θ( √ log l )), while in the latter case it is bounded.
Length quantities have simpler asymptotics determined by ˆ µ p : either ˆ µ p < 1 and p =
Θ( l 1 − ˆ µ p ), or ˆ µ p = 1 and p = Θ(log l ) (Thm A.3).
Cosine distance, unlike the case of ReLU resnets, can be controlled eﬀectively by β a and β v (Thm A.5).
When β a > β v , the magnitude of a ( l ) i drops much more quickly with depth than that of v ( l ) ij , so that higher layers experience the chaotic phase (Schoenholz et al., 2016; Yang and Schoenholz, 2017), driving e ( l ) toward the limit point e ∗ = 0.
On the other end, when β a < β v , v ( l ) ij vanishes in comparison to a ( l ) i with large l , so that the higher layers experience the stability phase (Schoenholz et al., 2016; Yang and Schoenholz, 2017), collapsing all inputs to the same output vector, sending e ( l ) → 1.
Only when β a = β v could the ﬁxed point e ∗ be controlled explicitly by σ v and σ a , with e ∗ given by the equation e ∗ =
σ
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
as in the case with no VV (Yang and
Schoenholz, 2017).
Experiments 6. ReLU Residual Network: Zigzagging through β • -space
As explained in Section 5.1, a zigzag of parameters control various asymptotics of ReLU resnet: “zigging” β from 0 to > 1, and then “zagging” α from 0 to > 1.
To investigate how this zigzag aﬀects neural network training and performance, we trained a grid of ReLU residual networks, pinning all σ • s at 1, but varying the β • s as follows: 1.
The zig: ﬁx β v = β a = 0; ﬁx β w = β b and increase both from 0 to 2 (making β = β w + β v go from 0 toas well); 2.
The zag: ﬁx β v = β a = 0 , β w = β b = 2; increase β a from 0 to 2 (increasing α from 0 to 2 as well).
For each setting of the hyperparameters, we train a network on MNIST with those hyperparameters for 30 epochs.
We then report the accuracy that the network achieved on the training and test sets 6 .
Fig.
4 shows the result, with the zig on the left and the zag on the right.
In the zig, we have overlaid a contour plot of p − γ , which is almost identical to the contour plots of p and (cid:107) (0) / (cid:107) ( l ) , and for simplicity’s sake, in the following we assume that they are exactly identical.
This contour decreases to the right.
The dashed line is given by
δ 1 − β l 1 − β = C for C ≈
In the zag, we have overlaid a contour plot of χ (1) v /χ ( l ) v .
This contour
increases to the upper right.
In addition, we provide heatmap and contour plots of various quantities of interest such as p , e , and χ (1) v /χ ( l ) v in both the zig and the zag regimes in Fig.
A.1 and Fig. A.2 respectively. Discussion.
First notice that training fails in the upper left corner of the zag.
This happens because of numerical instability caused by exploding p and (cid:107) (0) / (cid:107) ( l ) , which grow like exp( δ 1 − β l 1 − β
+ o ( l 1 − β )).
Indeed, one of the contour lines of p traces out almost exactly where training fails.
The dashed line is a level set of the dominant term of asymptotic expansion of log p , and we see it agrees with the contour of p very well.
Second, observe that performance actually dips in the direction where (cid:107) (0) / (cid:107) ( l ) decreases, quite counterintuitively.
This can be explained by noting that gradient against weights, χ w and χ v , in fact respectively remain bounded and polynomial in l (and changes rather mildly with β ; see Fig. A.1); gradient against biases do experience the same behavior as (cid:107) , but in general they are much less important than the weights, as parameters go.
In addition, the performance is also dipping in the direction where p − γ decreases (exponentially) in ( β, L )-space.
This is the quantity that essentially underlies the exponential expressivity result (as told from an extrinsic curvature perspective) of Poole et al.
(2016); as p − γ decreases dramatically, it gets harder and harder for a linear functional in the ﬁnal layer to tell apart two input vectors.
This exponential loss in expressivity dominates the eﬀect on performance more than a polynomial reduction in gradient explosion.
Third, it is remarkable that in the zag regime, the level curves of χ (1) v /χ ( l ) v accurately predict the contour of the test set performance, in such a counterintuitive way that greater gradient explosion χ (1) v /χ ( l ) v correlates with better performance.
It is currently unclear if in certain situations like this, larger gradient expansion is actually beneﬁcial, or if there is a quantity yet undiscovered which has the same level curves and can explain away this seeming paradox (like how p − γ explains away (cid:107) (0) / (cid:107) ( l ) above, in the zig regime). Of the quantities that appear in Fig. A.2, none foots this bill.
Conclusion
In this paper, we derived the mean ﬁeld theory of width and variance variation and showed that they are powerful methods to control forward (VV) and backward (VV + WV) dynamics.
We proved that even with a ﬁxed architecture and activation function, the mean ﬁeld dynamics of a residual neural network can still be manipulated at will by these two methods.
Extraordinarily, the mean ﬁeld theory we developed allowed us to accurately predict the performances of trained MNIST models on test sets, but one puzzling aspect remains where test set performance seems to increase as gradient explosion worsens.
Open Problems.
We solved a small part, width variation, of the program to construct mean ﬁeld theories of state-of-the-art neural networks used in practice.
Many open problems still remain, and the most important of them include but is not limited to 1.
batchnorm, 2.
convolution layers, and 3.
recurrent layers.
In addition, more work is needed to mathematically justify our “physical” assumptions Axiom 1 and Axiom 2 to a “math” problem.
We hope readers will take note and contribute toward deep mean ﬁeld theory.
Appendices A Main Theorems A. Recurrences
Yang and Schoenholz (2017) gave recurrences for mean ﬁeld quantities p, q, γ, λ, (cid:107) under the assumption of constant initialization variances across depth.
The proofs there carry over straightforwardly when variance varies from layer to layer.
Schoenholz et al.
(2016) also derived backward dynamics for when the width of the network is not constant.
Generalizing to the residual network case requires some careful justiﬁcations of independences, so we provide proof for gradient dynamics; but we omit the proof for the forward dynamics as it is not aﬀected by nonconstant width and is almost identical to the constant variance case.
Theorem A.1.
For any nonlinearity φ in an FRN, regardless of whether widths vary across layers,
[Equation]
Theorem A.2.
Suppose a random residual network receives a ﬁxed gradient vector
[Equation]
i
[Equation]
i =1 with respect to some cost function E , at its last layer.
For any non- linearity φ in an FRN, under Axiom 1 and Axiom 2, whenever ˙ φ ( ζ ) 2 has ﬁnite variance for any Gaussian variable ζ ,
[Equation]
if layer l has projection connection
[Equation]
if layer l has identity connection
[Equation]
Proof.
We will show the proof for the projection connection case; the identity connection case is similar but easier.
Write (cid:105) ( l ) i = ∂E
∂x ( l ) i .
We have the following derivative computations:
[Equation]
Then
[Equation]
M (cid:88) k
= v ik ˙ φ ( h k ) w kj )
Thus, (cid:104) (cid:105)
[Equation]
(cid:105) = (cid:104) [
N (cid:88) i = (cid:105) i ( π ij +
M (cid:88) k
= v ik ˙ φ ( h k ) w kj )] 2 (cid:105) =
(cid:104)
N (cid:88) i = (cid:105) 2 i [( π ij +
M (cid:88) k
= [Equation] ik ˙ [Equation]
[Equation] (cid:105) [Equation] (cid:88)
[Equation]
i (cid:48)
[Equation]
i
[Equation]
= N (cid:104) (cid:105) 21 (cid:105) ( (cid:104)
[Equation]
(cid:105)
[Equation]
(cid:104) [
[Equation]
(cid:88) k
[Equation] k ˙ [Equation] [Equation]
k )
[Equation]
[Equation]
(cid:105) )
[Equation]
(cid:88)
[Equation]
<i (cid:48) (cid:104)
[Equation]
i (cid:105) i (cid:48)
[Equation] [Equation] where in the second equality, we expanded algebraically, and in the third equality, we use the symmetry assumption Axiom 1 and the independence assumption Axiom 2.
Now, (cid:104)
[ (cid:80) M k
=1 v 1 k ˙ φ ( h k ) w kj ] 2 (cid:105) = (cid:80) M k =1 (cid:104) v 21 k ˙ φ ( h k ) 2 w 2 kj (cid:105) = (cid:80) M k
=1 σ 2 v l − β v M V ˙ φ (0 , q ) σ 2 w l − β w N = σ 2 v l − β v V ˙ φ (0 , q ) σ 2 w l − βw N by the independence of { v ik } i,k ∪ { h k } k ∪ { w kj } k,j (by our independence assumptions Axiom 2).
Similarly, because { π ij } j ∪ { π i (cid:48) j } j ∪ { v ik } k ∪ { v i (cid:48) k } k for i (cid:54) = i (cid:48) is mutually independent by our assumptions, one can easily see that (cid:104) ( π ij + (cid:80) Mk =1 v ik ˙ φ ( h k ) w kj )
( π i (cid:48)
j + (cid:80) Mk =1 v i (cid:48)
k ˙ φ ( h k ) w kj ) (cid:105) = 0.
Therefore,
[Equation]
For the other gradients
[Equation]
we have (where we apply Axiom 2 implicitly)
[Equation]
A. Residual Networks with Tanh
In this section we derive the asymptotics of various mean ﬁeld quantities for tanh resnet.
The main proof technique is to bound the dynamics in question with known dynamics of diﬀerence equations (as in Yang and Schoenholz (2017)).
Let ˆ µ p :=
min( β v , β a ).
Theorem A.3.
Suppose φ = tanh , and q ( l ) → ∞ as l → ∞ .
If ˆ µ p < 1 , then 1 > β w + ˆ µ p and p ( l ) = Θ( l 1 − ˆ µ p ) q ( l ) = Θ( l 1 − β w − ˆ µ p ) .
More speciﬁcally, ( p ( l ) , q ( l ) ) ∼   ( σ
[Equation]
l
[Equation]
β
[Equation]
, σ
[Equation]
σ 2 w
[Equation]
1 − β
[Equation]
−
β
[Equation]
if
β
v < β a ( σ a l 1 − β a
[Equation] σ
2 a σ 2 w l 1 − β a − β w if
β a < β
v
(( σ 2 a + σ 2 v ) l 1 − β v , ( σ 2 a + σ 2 v ) σ 2 w l 1 − β v − β w ) if β a = β v . If ˆ µ p = 1 , then β w = 0 , and p ( l ) = Θ(log l ) = q ( l ) . More speciﬁcally, ( p ( l ) , q ( l ) ) ∼   ( σ 2 v , σ 2 w σ 2 v log l ) if β v < β a ( σ 2 a log l, σ 2 w σ 2 a log l ) if β v > β a (( σ 2 a + σ 2 v ) log l, σ
[Equation]
ˆ µ p cannot be greater than 1.
Proof.
Claim 1.
We have V φ (0 , q ) = 1 − (cid:113) 2 π q − 1 / 2 + Θ( q − 3 / 2 ) by Lemma B.5.
Thus
[Equation]
where we used the assumption 1 − ˆ µ p > 0.
Thus p ( l ) =
Θ( l 1 − ˆ µ p ) and goes to inﬁnity with l .
Then q = σ 2 w l − β w p + σ 2 b l − β b .
Because we assumed q → ∞ , the ﬁrst term necessarily dominates the second, and 1 − ˆ µ p − β w > 0.
The possible asymptotics of q are then
[Equation]
which gives q ( l )
=
Θ( l 1 − β w − ˆ µ p ) .
Claim 2.
If ˆ µ p = 1, then
[Equation]
Then for q to go to inﬁnity, β w has to be 0, so that q = Θ(log l ) as well, and
[Equation]
l
[Equation]
β a .
Claim 3.
If ˆ µ p > 1, then p = Θ(1) = ⇒ q = Θ(1) , a contradiction.
Theorem A.4.
Suppose φ = tanh , and q ( l ) → ∞ with l .
Then log( (cid:107) (0) / (cid:107) ( l ) ) is
If ˆ µ p <
If 1 − β w + ˆ µ p − 2 β v >Θ( l 12 (1 − β w +ˆ µ p − 2 β v ) )
If 1 − β w + ˆ µ p − 2 β v =Θ(log l )
If 1 − β w + ˆ µ p − 2 β v <Θ(1)
If ˆ µ p =
If β v + β w <
This case cannot happen
If β v + β w =Θ( √ log l )
If β v + β w >Θ(1)
Proof.
By Thm A.2 and Lemma B.4, we have
[Equation]
where C = 23 (cid:113) 2 π .
Since q ( l ) has diﬀerent growth rates depending on the hyperparameters, we need to consider diﬀerent cases:
(cid:136)
If ˆ µ p < 1, then Thm A.3 implies β w + ˆ µ p < 1 and q = Θ( l 1 − β w − ˆ µ p ). So
[Equation]
=
[Equation]
l
[Equation]
(cid:158) If 1 − β w + ˆ µ p − 2 β v > 0, log (cid:107) (0) / (cid:107) ( l ) = (cid:80) lm =1 Θ( m 12 ( − 1 − β w +ˆ µ p − 2 β v ) )
=
Θ( l 12 (1 − β w
+ˆ µ p − 2 β v ) ) .
(cid:158)
If 1 − β w + ˆ µ p − 2 β v = 0, then l − β v − β w q − 1 / 2 = Θ( l − 1 ).
Thus log (cid:107) (0) / (cid:107) ( l ) =
(cid:80) lm =1 Θ( m − 1 ) = Θ(log l ) .
(cid:158)
If 1 − β w +ˆ µ p − 2 β v < 0, then 12 ( − 1 − β w + ˆ µ p − 2 β v ) > 1, so that log (cid:107) (0) / (cid:107) ( l ) = Θ(1).
(cid:136)
If ˆ µ p = 1, then by Thm A.3, β w = 0 and q = Θ(log l ).
So l − β v − β w q − 1 / 2 =
Θ( l − β v − β w / √ log l ) .
(cid:158)
If β v + β w < 1, then β v < 1 = ⇒ ˆ µ p < 1, contradiction.
(cid:158)
If β v + β w = 1, then log (cid:107) (0) / (cid:107) ( l ) = (cid:80) lm =1 Θ( l − 1 / √ log l ) =
Θ( √ log l ) .
(cid:158)
If β v + β w > 1, then log (cid:107) (0) / (cid:107) ( l ) = Θ(1) .
Theorem A.5.
Suppose φ = tanh and q → ∞ .
If e (0) < 1 , then e ( l ) converges to a ﬁxed point e ∗ , given by the equations
[Equation]
Note that in the case β a = β v , we recover the ﬁxed point of tanh residual network without variance decay.
Proof.
We have
[Equation]
Using Lemma B.1, ??
and Lemma B.5, we can see that the LHS is monotonic (increasing or decreasing) for large enough l .
Therefore e ( l ) is a bounded monotonic sequence for large enough l , a fortiori it has a ﬁxed point.
If we express e = e ∗ + (cid:15) , γ ( l ) = ( e ∗ + (cid:15) ( l ) ) p ( l ) , then [Equation] [Equation] [Equation]
[Equation]
RHS It’s easy to verify via Thm A.3 that either p/ ( p − p ) =
Θ( l log l ) (when ˆ µ p = 1) or p/ ( p − p ) =
Θ( l ) (all other cases).
If (cid:15) − (cid:15) =
Ω(( l log l ) − 1 ), then (cid:15) =
Ω(log log l ) (by EulerMacLaurin formula), which would be a contradiction to (cid:15) =
o (1).
Thus ( (cid:15) − (cid:15) )
p p − p
=
o (1).
Hence the RHS goes to 0 with l .
LHS
If β v > β a , then the LHS converges to 1.
So e ∗ = 1.
If β v < β a , then the LHS ∼ W φ (0 ,q,q,cq )
V φ (0 ,q ) .
As l → ∞ , c ∼ e → e ∗ , and W(0 , q, q, e ∗ q ) → 2 π arcsin( e ∗ ), and V(0 , q ) → 1.
Therefore, e ∗ = 2 π arcsin( e ∗ ), for which there are 2 solutions, 0, the stable ﬁxed point, and 1, the unstable ﬁxed point.
In particular, for all e (0) < 1, e ( l ) → 0.
If β v = β a , then taking limits l → ∞ , we get
[Equation]
A. Residual Network with ReLU
The asymptotics of ReLU resnet depends on the following values:
[Equation]
Theorem A.6.
Let φ = ReLU .
Then with δ = 12 σ 2 v σ 2 w and α = min( β v + β b , β a ) =
β v + min( β b , β a − β v ) , we have the following asymptotics of p and q :
(cid:136)
If β v + β w >(cid:158) If α ∈ [0 , 1) (cid:5) p =
Θ( l 1 − α ) and q = Θ( l max(1 − α − β w , − β b ) ) .
(cid:158)
If α =(cid:5) p = Θ(log l ) and q = (cid:26) Θ( l − β w log l ) if β w ≤ β b Θ( l − β b ) otherwise.
(cid:158)
If α >(cid:5) p = Θ(1) , p − p =
Θ( l − α ) and q = Θ( l − min( β w ,β b ) ) .
(cid:136)
If β v + β w =(cid:158)
If δ (cid:54) = 1 − α (cid:5) p
=
Θ( l max( δ, 1 − α ) ) , and q = Θ( l max(max( δ, 1 − α ) − β w , − β b ) ) .
(cid:158) Otherwise (cid:5) p =
Θ( l δ log l ) and q = (cid:26) Θ
( l δ − β w log l ) if β b ≥ β w − δ Θ( l − β b ) otherwise (cid:136)
If β v + β w < [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Equation] In particular, p = [Equation] [Equation]
+ O [Equation] [Equation] ) , R is the same R as in
Lemma B.7, depending on only l , δ , and β v + β w , with
R =  
Θ( l 1 − 2 β
[Equation]
if β < 1 /
[Equation]
= 1 /
[Equation]
[Equation] if β > 1 / 2 .
and q = exp( δ
[Equation]
l
[Equation]
R ( l
[Equation]
w log l + O (1)) .
In particular, p, q = poly ( l ) if β v + β w ≥ 1 .
Proof.
Since V φ (0 , q ) = 12 q , we have
[Equation]
We apply Lemma B.8 with β = β v + β w , δ = 12 σ 2 v σ 2 w , and α = min( β v + β b , β a ) = β v + min( β b , β a − β v ) to upper and lower bound our dynamics here, with appropriately chosen C .
With a little bit of case work, we obtain the desired results.
Theorem A.7.
Let φ = ReLU .
Suppose e (0) < 1 .
(cid:136)
If β v + β w ≤ 1 or α ≤ 1 , then lim l →∞ e ( l ) = 1 .
(cid:136)
If β v + β w > 1 and α > 1 , then e ( l ) converges to a ﬁxed point e ∗ < 1 , dependent on the initial data p (0) and γ (0) , at a rate of Θ( l − α +1 ) where α = min( β v + β b , β a )
Proof.
By Cho and Saul (2009); Yang and Schoenholz (2017), we have W(0 , q, q, cq ) =
V φ (0 , q ) J 1 ( c ) =
12 q · 1 π ( √ 1 − c 2 + ( π − arccos( c )) c ) .
As in the proof of Thm A.5, we have
[Equation]
But σ 2 v l − βvq / 2 J 1 ( c )+ σ 2 a l − βa σv l − βvq / 2 + σ 2 a l − βa ≥
J 1 ( c ) ≥ c ≥ e , where the last inequality holds for all large l , by Lemma B.1.
So the LHS is nonnegative for large l , which implies e ( l ) is a bounded monotonically increasing sequence for large enough l s and thus has a limit e ∗ .
Writing e = e ∗ + (cid:15) , we have
[Equation]
Suppose β v + β w ≤ 1 or α ≤ 1 .
LHS.
By Thm A.6, p = ω (1) (assuming σ v , σ w > 0), so that p/ ( p − p ) =
˜ O
( l ).
As in the proof of Thm A.5, (cid:15) − (cid:15) cannot be ˜Ω( l − 1 ), or else (cid:15) → ∞ .
Thus ( (cid:15) − (cid:15) )
p/ ( p − p )
=
o (1), and the LHS in the limit l → ∞ becomes e ∗ .
RHS.
In all cases, we will ﬁnd e ∗ = 1.
If q = o ( l β v − β a ), in the RHS σ 2 a l − β a dominates, and the RHS converges to 1.
If q = ω ( l β v − β a ), then in the RHS 12 σ 2 v l − β v q dominates.
By Lemma B.1: (cid:136)
If p = ω ( l β w − β b ), then c ∼ e → e ∗ , so that in the limit l → ∞ , e ∗ =
J 1 ( e ∗ ).
This equation has solution e ∗ = 1.
(cid:136)
If p = o ( l β w − β b ), then c → 1, so that e ∗ =
J 1 (1) = 1.
(cid:136)
If p ∼ Cl β w − β b , then c → σ 2 w C e ∗ + σ 2 b σ 2 w C + σ 2 b , and we have an equation e ∗ =
J 1 ( σ 2 w C e ∗ + σ 2 b σ 2 w C + σ 2 b ) .
Note that since e ∗ ∈ [0 , 1], σ 2 w C e ∗ + σ 2 b σ 2 w C + σ 2 b =
[Equation]
with equality iﬀ e ∗ =
Since J 1 is monotonic, e ∗ = J 1 ( σ w C e ∗
+
[Equation] [Equation] σ w C +
σ
2 b ) iﬀ the equality condition above is satisifed, i.e. e ∗ =
If q ∼ Dl β w − β b for some D , then the RHS is
[Equation]
by the same logic as in the proof of Lemma B
As in the above case, Lemma B.1 yields the following results:
(cid:136)
If p = ω ( l β w − β b ), then c ∼ e → e ∗ , so that in the limit l → ∞ , e ∗ = 12 σ 2 v D J 1 ( e ∗ )+ σ 2 a 1 2 σ 2 v D + σ 2 a .
Since the RHS of this equation is a convex combination of J 1 ( e ∗ ) and 1, and J 1 ( e ∗ ) ≥ e ∗ by the monotonicity of J 1 , the equality can hold only if J 1 ( e ∗ ) =
e ∗ .
The only such e ∗ is 1.
(cid:136)
If p = o ( l β w − β b ), then c → 1, so that e ∗ = 12 σ 2 v D J 1 (1)+ σ 2 a 12 σ 2 v D + σ 2 a = 1.
(cid:136)
If p ∼ Cl β w − β b , then c → σ 2 w C e ∗ + σ 2 b σw C + σ 2 b , and we have an equation e ∗ = 1 12 σ 2 v D + σ 2 a ( 12 σ 2 v D J 1 ( σ 2 w C e ∗ + σ 2 b σ 2 w C + σ 2 b ) + σ 2 a ) .
By the monotonicity of J 1 , the RHS is at least 1 12 σ 2 v D + σ 2 a ( 12 σ 2 v D ( σ 2 w C e ∗ + σ 2 b σ 2 w C + σ 2 b ) + σ 2 a ), which is a convex combination of e ∗ and 1.
Since e ∗ ≤ 1, the equality can hold only if e ∗ = 1.
Suppose β v + β w > 1 and α > 1 .
By Thm A.6, p = Θ(1) and therefore γ = Θ(1).
Both converge to ﬁxed points p ∗ and γ ∗ (possibly dependent on initial data p (0) and γ (0) ) because they are monotonically increasing sequences.
Thus e ∗ = γ ∗ /p ∗ .
Unwinding the proof of Thm A.6, we see that p − p =
l − α , so that p/ ( p − p ) =
Θ( l α ) .
Since the
RHS of Eq.
( (cid:63) ) cannot blow up, it must be the case that ( (cid:15) − (cid:15) )
=
O ( l − α )
=
⇒ (cid:15) =
O ( l − α +1 ).
If ( (cid:15) − (cid:15) )
p/ ( p − p ) =
F + o (1) for some constant F , then the LHS becomes e ∗ + F in the limit.
Yet, unless γ (0) = p (0) , e ∗ < 1.
Therefore, F > 0 (or else, like in the case of β v + β w ≤ 1 or α ≤ 1, e ∗ = 1) whenever γ (0) < p (0) , and (cid:15) =
Θ( l − α +1 ).
Theorem A.8.
Suppose φ = ReLU .
Then for α = min( β v + β b , β a ) and δ = 12 σ 2 v σ 2 w as in
Thm A.6: (cid:136)
If β v + β w > 1 , then (cid:107) (0) / (cid:107) ( l ) = Θ(1) .
If α ∈ [0 , 1) [Equation] [Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation] [Equation]
[Equation]
If [Equation] [Equation] [Equation]
[Equation]
[Equation] [Equation] [Equation] [Table]
Otherwise
[Table] w / [Table] w = Θ
[Table] v / [Table]
[Table] 1) b / [Table]
[Table] ) b = Θ [Table]
[Table] v / [Table] =
Θ [Table] [Table] ) a /
[Table]
[Table]
) a =
[Table]
If [Table] [Table] Otherwise
[Table] If
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] = [Table]
[Table] [Table] [Table] Otherwise
[Table] [Table] w = [Table]
/ log [Table] [Table] [Table]
=
[Table]
[Table] [Table] [Equation] (cid:136)
If β v + β w < 1 , then (cid:107)
[Equation]
Proof.
Using Thm A.2 and the fact that V φ (0 , q ) = 12 q, V ˙ φ (0 , q ) = 12 , we get
[Equation]
so
[Equation]
) p ( l − 1) (cid:107) (0) (cid:107) ( l ) χ (1) v χ ( l ) v = 12 (cid:107) (1) q (1) 12 (cid:107) ( l ) q ( l − 1) =
q (0) q
( l − 1) (cid:107) (0) (cid:107) ( l ) χ (1) a χ ( l ) a = (cid:107) (1) (cid:107) ( l ) .
Suppose β v + β w > 1.
Then (cid:107) (0) (cid:107) ( l ) = Θ(1) by Lemma B.7.
By Thm A.6:
(cid:136)
If β v + β w > 1, then (cid:107) (0) (cid:107) ( l ) = Θ(1) by Lemma B.7.
(cid:158)
If α ∈ [0 , 1) (cid:5) p =
Θ( l 1 − α ) and q = Θ( l max(1 − α − β w , − β b ) ) .
So χ (1) w / χ ( l ) w = Θ ( l α − 1+ β v ) [Equation] [Equation]
( l [Equation] v ) χ (1) v / χ ( l ) v
[Equation]
Θ( l − max(1 − α − [Equation] − β b ) )
χ
[Equation]
[Equation] l ) [Equation] Θ
(cid:158)
If α = 1, then p = Θ(log l ) and (cid:5) If β w ≤ β b (cid:5) q = Θ( l − β w log l ) so that χ (1) w / χ ( l ) w = Θ
( l β v / log l ) χ 1) b / χ ( l ) b =
Θ( l β v ) χ (1) v / χ ( l ) v =
Θ( l β w / log l ) χ 1) a / χ ( l ) a = Θ(1) (cid:5) Otherwise (cid:5) q = Θ( l − β b )
so that χ (1) w / χ ( l ) w = Θ ( l β v / log l ) χ (1) b χ ( l ) b =
Θ
( l β v ) χ (1) v / χ l ) v =
Θ
( l β b ) χ ) a / χ ( l
) a = Θ 1) (cid:158)
If α >(cid:5) p = Θ(1) and q = Θ( l − min( β w ,β b ) ) .
So χ (1) w χ ( l ) w = Θ( l β
v ) χ
b / χ ( l b = Θ( l β
v ) χ (
(cid:136)
If β v + β w = 1, then by Lemma B.7 (cid:107) (0) / (cid:107) ( l ) =
Θ( l 12 σ 2 v σ 2 w ) = Θ( l δ ) . (cid:158)
If δ (cid:54) = 1 − α (cid:5) p =
Θ( l max( δ, 1 − α ) ), and q = Θ( l max(max( δ, 1 − α ) − β w , − β b ) ) .
So
l
(cid:158) Otherwise, p =
Θ( l δ log l ), and (cid:5)
If β b ≥ β w − δ
, so ) v
l )
(cid:5) Otherwise (cid:5) q = Θ( l
), so χ
[Equation]
χ
[Equation]
(cid:136)
If β v + β w < 1, then by Lemma B.7, (cid:107) (0) / (cid:107) ( l ) =
exp( δ
[Equation]
β v
[Equation]
β w l [Equation] w + R ( l ; δ, β v + β w )), where R = ˜Θ( l max(1 − 2 β v − 2 β w , 0) ) is as in Lemma B.7.
(cid:158)
We have p = exp( δ 1 − β v − β w l 1 − β v − β w + R ( l ; δ, β v + β w ) + O (1)), where R is the same as above, and q = exp( δ 1 − β l
[Equation]
β [Equation] β w + R ( l ; δ, β v + β w ) − β w log l + O (1)).
Thus χ [Equation] χ [Equation]
χ
[Equation]
v
[Equation]
exp(
[Equation]
w log l [Equation] O [Equation]
[Equation]
χ (1)
[Equation]
χ ( l )
[Equation]
exp(
[Equation] β v − β w [Equation] [Equation]
β
[Equation]
β
[Equation]
B Lemmas
In this section, we present many lemmas used in the proofs of the main theorems.
In all cases, the lemmas here have already appeared in some form in Yang and Schoenholz (2017), and for completeness, we either include them and their proofs here or improve upon and extend them, with the blessing of the authors [Equation]
Lemma B
[Equation]
Asymptotically,
[Equation]
if p =
Cl β w − β b
+ R for some constant C and remainder R =
o ( l β [Equation] β b ) .
where E =
[Equation]
Proof.
If p = o ( l β w − β b ), then
[Equation]
If p = ω ( l β w − β b )
[Equation]
then
[Equation]
p
[Equation]
If p ∼ Cl β w − β b for some C and R = p − Cl β w − β b = o ( l β w − β b )
[Equation]
then
[Equation]
For any function f that is ( k + 1)-times
diﬀerentiable
in a neighborhood of 0, we have the asymptotic expansion
[Equation]
Since
[Equation]
whenever the RHS is integrable, we have
Lemma B.2.
Suppose φ 2 ( z ) z
2 n is integrable over z ∈ R for all 0 ≤ n ≤ N + 1 .
Then
V φ (0 , q ) =
q − 1 / 2 ( (cid:80) Nn =
C n q − n + O ( q − N − 1 ))
as q → ∞ , where
[Equation]
Note that sech d ( z ) =
Θ( e − d | z | ) for z → ∞ as long as d > 0, so that C n from the above result converges when φ = sech d .
Therefore
Lemma B.3.
Let d > 0 .
We have V sech d (0 , q ) (cid:39) q − 1 / 2 (cid:80) n ≥ 0 C n q − n , where
[Equation]
As corollaries, we obtain the following asymptotics.
Lemma B.4.
V ˙ tanh(0 , q ) = 23 (cid:113) 2 π q − 1 / 2 + Θ
q − 3 / 2 ) as q → ∞ .
Proof.
Use Lemma B.3 along with the fact that ˙ tanh( z ) = sech 2 ( z ) and (cid:82) sech 4 z d z = 23 tanh z
+ 12 sech 2 z tanh z .
Lemma B.5.
1 − V tanh(0 , q ) =
(cid:113)
π q − 1 / 2 + Θ( q − 3 / 2 ) as q → ∞ .
Proof.
Use Lemma B.3 along with the fact that 1 − tanh 2 ( z ) = sech 2 ( z ) and (cid:82) sech 2 z d z = tanh z .
Lemma B.6.
Let d ∈ R and 1 < M < N with N
− M ∈
Z ≥ 0 .
Set Σ( M, N, d ) :=
(cid:80) Na =
M a d .
If we ﬁx M and let N → ∞ , Σ
[Equation]
M, N,
[Equation]
=
[Equation]
if d < −
[Equation]
if d = −
[Equation]
if − 1 < d <
[Equation]
if d =
[Equation]
if d >
Proof.
Euler-MacLaurin formula.
Lemma B.7.
Suppose (cid:15) ( l ) satisﬁes the recurrence
[Equation]
for some nonzero constant δ ∈ R independent of l .
(cid:136)
If β > 1 , then (cid:15) ( l ) = Θ(1) .
(cid:136)
If β = 1 , then (cid:15) ( l ) =
Θ( l δ ) .
(cid:136)
If 0 < β < 1 , then (cid:15) ( l ) = exp( δ 1 − β l 1 − β + ˜Θ( l max(0 , 1 − 2 β ) )) .
In particular, (cid:15) ( l )
= exp( δ 1 − β l 1 − β + R ) , where remainder R is
R =  
Θ( l 1 − 2 β ) if β < 1 /Θ(log l ) if β = 1 /Θ(1)
if β > 1 /
Proof.
We have log
[Equation]
) for large l .
If β > 1, then (cid:80) l l − β converges, and log (cid:15) ( l )
= log (cid:15) (0) − Θ(1) (cid:15) ( l ) = Θ(1) .
If β = 1, then log (cid:15) ( l )
[Equation]
[Equation] 1) (cid:15) ( l )
[Equation]
Θ( l δ ) . If β < 1, then log (cid:15) ( l )
[Equation]
for a remainder R that is
[Equation]
Exponentiating gives the desired result.
Lemma B.8.
Suppose (cid:15) ( l ) =
Cl − α + (cid:15) ( l − 1) (1 + δ/l β ) for α ∈ R , C (cid:54) = 0 , and δ (cid:54) = 0 .
Then (cid:136)
If β > 1 , then (cid:158) (cid:15) ( l ) =
Θ( l 1 − α ) if α ∈ [0 , 1) ; (cid:158) (cid:15) ( l ) = Θ(log l ) if α = 1 ; (cid:158) (cid:15) ( l ) = Θ(1) if α > 1 .
(cid:136)
If β = 1 , then (cid:158) (cid:15) ( l ) =
Θ( l max( δ, 1 − α ) )
if 1 − δ (cid:54) = α .
(cid:158) (cid:15) ( l ) =
Θ( l δ log l ) if 1 − δ = α .
(cid:136)
If β < 1 , then (cid:158) (cid:15) ( l ) =
exp( δ 1 − β l 1 − β + ˜Θ( l max(0 , 1 − 2 β ) )) .
In particular, (cid:15) ( l ) =
exp( δ 1 − β l 1 − β + R ( l ; δ, β ) + O (1)) , where R is the same R as in Lemma B.7, depending on only l , δ , and β but not on α or C , with
R =  
Θ( l 1 − 2 β ) if β < 1 /Θ(log l ) if β = 1 /Θ(1)
if β > 1 / 2 .
Furthermore, for β = − δ = 1 : (cid:15) ( l ) ∼ l − 1 if α > 2 , (cid:15) ( l ) ∼ l 1 − α if α < 2 , and (cid:15) ( l ) ∼ l δ log l if α = 2 .
Proof.
We can unwind the recurrence to get
[Equation]
C
[Equation]
Suppose β > 1.
By Lemma B.7, we get (cid:15) ( l ) = Θ(1)
[Equation]
m = m − α + (cid:15) (
[Equation]
) Θ(1) =
  Θ( l 1 − α ) if α ∈ [0 , 1) Θ(log l ) if α =Θ(1) if α > 1.
Now suppose β = 1.
By Lemma B.7, we get (cid:15)
[Equation]
l ) =
[Equation]
(cid:88) m = [Equation] m [Equation]
0) Θ
[Equation]
l δ ) where the constants hidden inside the Θ are the same in every term of the sum.
If α > 1 − δ , then m − δ − α
= o ( m − 1 ), so that (cid:80) lm =1 m − δ − α = Θ(1), and (cid:15) ( l ) =
Θ( l δ ) + (cid:15) (0) Θ( l δ ) =
Θ( l δ ) .
On the other hand, if α < 1 − δ , then (cid:80) lm
=1 m − δ − α = Θ( l 1 − δ − α ).
So (cid:15) ( l ) =
Θ( l 1 − α ) + (cid:15) (0) Θ( l δ ) =
Θ( l 1 − α ) .
If α = 1 − δ , then (cid:80) lm
[Equation]
1 m −
[Equation]
= Θ(log l ).
So
[Equation]
Finally, if β ∈ (0 , 1), then by Lemma B.7,
[Equation]
where R is the remainder as in Lemma B.7.
The sum
[Equation]
m
[Equation]
can be upper and lower bounded by integrals of the form (cid:82) x − α e − µx 1 − β d x for appropriate µ s, which are ﬁnite (they are bounded by values of incomplete Gamma functions).
Thus (cid:15) ( l ) = Θ(1) e δ 1 − β l 1 − β
+ R where
[Equation]
A fortiori,
[Equation]
For our “furthermore” claim: the case of δ = − 1 telescopes, so that the upper and lower constants hidden in Θ can both be taken to be 1.