O N R ANDOM D EEP W EIGHT -T IED A UTOENCODERS : E XACT A SYMPTOTIC A NALYSIS , P HASE T RANSITIONS , AND I MPLICATIONS TO T RAINING
A BSTRACT
We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights.
Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large.
This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature.
Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al.
(2018), and its connection to reversibility considered by several theoretical studies.
Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts.
Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization.
Our analysis is not speciﬁc to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
I NTRODUCTION
The autoencoder is a cornerstone in machine learning, ﬁrst as a response to the unsupervised learning problem (Rumelhart & Zipser (1985)), then with applications to dimensionality reduction (Hinton & Salakhutdinov (2006)), unsupervised pre-training (Erhan et al.
and also as a precursor to many modern generative models (Goodfellow et al.
Its reconstruction power is well utilized in applications such as anomaly detection (Chandola et al.
and image recovery (Mousavi et al.
With the surge of deep learning, thousands of papers have studied multilayer variants of this architecture, but theoretical understanding has been limited, since analyzing the learning dynamics of a highly nonlinear structure is typically a difﬁcult problem even for the shallow autoencoder.
To get around this, we tackle the task with a critical assumption: the weights are random and the autoencoder is weight-tied.
One enjoys much analytical tractability from the randomness assumption, whereas weight tying enforces the random autoencoder to perform “autoencoding”.
We also study this in the high-dimensional setting, where all dimensions are comparably large and ideally jointly approaching inﬁnity.
We consider the simplest setting: vanilla autoencoders (i.e., ones with fully connected layers only) and their reconstruction capability.
This is done for the sake of understanding the effect of depth, while we note our techniques may have broader applicability.
The aforementioned assumptions are not without justiﬁcations.
There is a growing literature on deep neural networks with random weights, (Li & Saad (2018);
Giryes et al.
; Poole et al.
(2016); Schoenholz et al.
; Gabrié et al.
; Amari et al.
(2018)) to name a few, revealing certain properties of deep feedforward networks 1 .
Several recent works have also studied random multilayer feedforward networks through the lens of statistical inference (Manoel et al.
Reeves (2017); Fletcher et al.
The idea of weight tying is considered in the important paper Vincent et al.
(2010) with an empirical ﬁnding that autoencoders with and without weight tying perform comparably, and has become standard in autoencoders.
Similar features of random connection and symmetry also appear in other neural models (Lillicrap et al.
; Scellier et al.
Finally the high-dimensional setting is common in recent statistical learning advances (Bühlmann & Van De Geer (2011)), and not too far from the actual practice where many large datasets have dimensions of at least a few hundreds and are harnessed by large-scaled models.
We seek quantitative answers to three speciﬁc questions that are motivated by previous works:
• In exactly what way does the (vanilla) random weight-tied autoencoder perform “approximate inference”?
This term is coined in Scellier et al.
(2018) in connection with the theoretical results in Arora et al.
(2015), which implicitly studies the said model.
In particular, Arora et al.
(2015) proves an upper bound on (cid:107) ˆ x − x (cid:107) 2 , where x and ˆ x are the input and the output of the network, but is limited in the number of layers and speciﬁc to the ReLU activation.
This direction has been recently extended by Gilbert et al.
In our work, we establish precisely what this approximate inference is by obtaining a general and asymptotically exact characterization 2 of ˆ x , for any number of layers and any Lipschitz continuous activations (Theorem 1 and Section 3.3).
Theorem 1 is the key theoretical result of our work and lays the foundation for all analyses that follow
• In what way is the deep autoencoder different from the shallow counterpart?
Li & Saad (2018); Poole et al.
(2016) reveal this in terms of the candidate function space and expressivity for feedforward networks.
It is unclear how these notions are applicable to weighttied autoencoders, which seek replication of the input rather than a generic mapping.
In this work, we show that the deep autoencoder exhibits a higher order of sensitivity to perturbations of the parameters (Section 3.4).
• Does it have any implications to the training practice?
Many recent works 3 Glorot & Bengio (2010)
; He et al.
; Schoenholz et al.
; Pennington et al.
; Yang & Schoenholz (2017); Xiao et al.
; Chen et al.
; Hayou et al.
; Hanin & Rolnick (2018); Hanin (2018); Burkholz & Dubatovka (2018) demonstrate a connection between the study of random networks, or ones at initialization
and their trainability.
Note that these works either do not study weight-tied structures, or assume the analysis of the untying case for weight-tied structures.
In our work, we derive and experimentally verify insights on how (not) to initialize deep weight-tied autoencoders, demonstrating that it is possible to train them without resorting to techniques such as greedy layer-wise pretraining, drop-out and batch normalization (Section 3.5).
Speciﬁcally we experiment with 200-layer autoencoders.
No prior works have attempted all three tasks.
The quantitative difference between weight-tied and weight-untied networks is in fact not negligible, yet the analysis is non-trivial due to the weight tying constraint (Arora et al. (2015); Chen et al.
To address this issue and obtain Theorem 1, we apply the Gaussian conditioning technique, which ﬁrst appears in the studies of TAP equations in spin glass theory (Bolthausen (2014)) and is extensively used in the approximate message passing algorithm literature (Bayati & Montanari (2011);
Javanmard & Montanari (2013); Berthier et al.
This should be contrasted with untied random networks, whose analysis is typically more straightforward.
More importantly, the difference is not only analytical: the overall picture of deep random weight-tied autoencoders is rich and drastically different from that of feedforward networks.
An analysis in the limit of inﬁnite depth reveals three fundamental equations governing the picture (Section 3.1), which displays multiple phase transition phenomena (Section 3.2).
Finally let us quickly mention other recent theoretical works on autoencoders: Arora et al.
Arpit et al.
; Rangamani et al.
; Nguyen et al.
(2018) studying the learned autoencoders in speciﬁc settings, Baldi (2012); Alain & Bengio (2014); Bengio et al.
(2013) taking a framework where the encoder and the decoder are generic mappings, and Le Roux & Bengio (2008);
Sutskever
& Hinton (2008); Montufar & Ay (2011) exploring representational properties of related architectures.
These works head in different directions from ours.
S ETTING AND MAIN THEOREM 2. S ETTING AND ASSUMPTIONS
Consider the following 2 L -layers autoencoder with weight tying:
[Equation]
Here x ∈ R n 0 is the input, W (cid:96)
∈ R n (cid:96) × n (cid:96) − 1 is the weight, b (cid:96) ∈ R n (cid:96) is the encoder bias, and v (cid:96) ∈ R n (cid:96) − 1 is the decoder bias, for (cid:96) = 1 , ..., L .
Also ϕ (cid:96) : R (cid:55)→ R and σ (cid:96) :
R (cid:55)→ R are the activations (where for a vector u ∈ R n and a function ϕ : R (cid:55)→ R , we write ϕ ( u ) to denote the vector ( ϕ ( u 1 ) , ..., ϕ ( u n ))
(cid:62) ).
It is usually the case in practice that σ 0 ( u ) = u the identity function.
We introduce some convenient quantities inductively:
[Equation]
Note that ˆ x = ϕ
(ˆ x 1 ) .
See Fig.
5 of Appendix A.1 for a schematic diagram.
We assume weights are random.
Speciﬁcally we generate the weights and biases according to
[Equation]
independently of each other.
The scaling of the variances accords with the literature and actual practice (Glorot & Bengio (2010); Vincent et al.
We also consider the asymptotic highdimensional regime, indexed by n :
[Equation]
Here σ W,(cid:96) , σ b,(cid:96) , σ v,(cid:96) and α (cid:96) are ﬁnite constants independent of n .
We enforce σ W,(cid:96) > 0 , but allow σ b,(cid:96) and σ v,(cid:96) to be zero.
We assume that all activations are Lipschitz continuous, and the encoder activations σ (cid:96) ’s are non-trivial in the sense that for any τ > 0 , E z (cid:110) σ (cid:96) ( τz ) 2 (cid:111) > 0 where z ∼ N (0 , 1) .
We also assume that 1 n 0 (cid:107) σ 0 ( x ) (cid:107) 2 tends to a ﬁnite and strictly positive constant as n → ∞ .
We refer to Appendix A.1 for more clariﬁcations of notations.
2. M AIN THEOREM
We motivate our main result via a simpliﬁed shallow autoencoder:
ˆ x = W (cid:62) ϕ
( W x ) = (cid:80) mi
=1 w i ϕ (cid:0) w (cid:62) i x (cid:1) , where W ∈ R m × n , w i ∈ R n .
Notice ˆ x is a sum of independent terms, and by Stein’s lemma (cf.
Appendix E.2), E { ˆ x } ∝ E (cid:8)(cid:80) m i
=1 ϕ (cid:48) (cid:0) w (cid:62) i x (cid:1) /m (cid:9) x .
One thus expects ˆ x ≈ c 1 x + c 2 z for scalars c 1 , c 2 and z ∼ N (0 , I n ) , for large n and m .
It is then important to specify exactly c 1 and c 2 .
Theorem 1 formalizes this intuition with precise formulas for the scalars.
We now deﬁne some scalar sequences, which will then be related to the (vector) quantities of the autoencoders in Theorem 1.
First we deﬁne { τ (cid:96) } (cid:96) =1 ,...,L and { ¯ τ (cid:96) } (cid:96) =0 ,...,L inductively:
[Equation]
) 2 (cid:111) , (cid:96) = 2 , ..., L, for z ∼ N (0 , 1) .
Next
[Equation]
we deﬁne { γ (cid:96) , ρ
[Equation]
} (cid:96) =2 ,...,L +1 inductively:
[Equation]
for z 1 , z 2 ∼ N (0 , 1) independently.
With these sequences deﬁned, we can state the main theorem.
Its statement uses the relational operator ∼ = , which is deﬁned formally in Appendix A.1.
Roughly speaking, a n ∼ = b n means a n and b n are asymptotically equal in distribution as n → ∞ .
Theorem 1.
Consider the settings and assumptions as in Section 2.1, and the sequences { τ (cid:96) , ¯ τ (cid:96) } and { γ (cid:96) , ρ (cid:96) } deﬁned as above.
Then in the limit n → ∞ :
(a) { ¯ τ (cid:96) } describes the
behavior of the encoder output x (cid:96) :
[Equation]
for z ∼ N (0 , I n (cid:96) ) .
(b) { ¯ τ (cid:96) , γ
[Equation]
, ρ (cid:96) } describes the behavior of the decoder output ˆ x (cid:96) :
[Equation]
for
z 1 , z 2 ∼ N (cid:0) 0 , I n (cid:96) − 1 (cid:1) independently
One can replace ¯ τ (cid:96) − 1 z
with x (cid:96) − 1 in
the
above, with z 2 independent of x (cid:96) −
, in which case the
statement also holds for (cid:96) = 1 with x 0 = x .
(c) For the autoencoder’s output ˆ x ,
[Equation]
α 1 σ 2 W, 1 γ 2 σ 0 ( x ) + (cid:113) α 1 σ 2 W
[Equation]
1 ρ [Equation] + σ
[Equation]
v
[Equation]
z
[Equation]
(cid:17) ,
for z 2 ∼ N (0 , I n 0 ) independent of x .
The proof of the theorem, as well as an outline of the key ideas, are in Appendix A.
The theorem says that x (cid:96) , ˆ x (cid:96) and ˆ x admit simple descriptions which are tracked by scalar sequences { ¯ τ (cid:96) , γ (cid:96) , ρ (cid:96) } .
Hence we can learn about the autoencoder by analyzing { ¯ τ (cid:96) , γ (cid:96) , ρ (cid:96) } , which is generally a simpler task than studying x (cid:96) , ˆ x (cid:96) and ˆ x directly.
Numerical simulations in Appendix B suggest that, although the theorem’s statement is in the inﬁnite dimension limit, the agreement is already good for dimensions of a few hundreds.
We note that while the theorem assumes Gaussian biases, the same proof technique allows to obtain a similar result with a more relaxed condition on the biases.
Remark 2 .
While the theorem is speciﬁc to W (cid:96) following the Gaussian distribution, simulations in Appendix B suggest that the conclusion holds for a much broader class of distributions.
We conjecture that it should hold so long as each W (cid:96) has i.i.d.
entries and is independent of each other, its distribution has bounded k -th moment for some sufﬁciently large k , and the activations as well as the input x satisfy certain mild regularity conditions.
Remark 3 .
We comment on the range of ρ (cid:96) and γ (cid:96) .
We have ρ (cid:96) ≥ 0 , which is obvious, and if (cid:107) ϕ
(cid:96) − 1 (cid:107) ∞ ≤ C , then ρ (cid:96) ≤ C 2 .
By Stein’s lemma (cf.
Appendix E.2),
[Equation]
γ
[Equation]
If the activations are non-decreasing, then γ (cid:96) ≥ 0 .
Furthermore, if the activations are Lipschitz, then | γ (cid:96) | ≤ Cc (cid:96) for some constants C and c .
A N ANALYSIS AT INFINITE DEPTH
In the following, we adopt a semi-rigorous approach, with an emphasis on the overall picture.
3. I NFINITE DEPTH SIMPLIFICATION
We make several analytical simpliﬁcations.
First consider α (cid:96) =
α > 0 , σ 2 W,(cid:96)
=
σ 2 W > 0 , σ 2 b,(cid:96) =
σ 2 b ≥ 0 , ϕ (cid:96) =
ϕ and σ (cid:96) =
σ all independent of (cid:96) , except for ϕ L which is chosen separately (but we shall see that the speciﬁc choice of ϕ L is largely immaterial).
We also assume that σ 2 v,(cid:96) = 0 , and σ 0 and ϕ 0 are the identity 4 .
We introduce a parameter ¯ τ ≥ 0 , whose role will be clear shortly, and which satisﬁes:
[Equation]
for z ∼ N (0 , 1) .
Note that this implies σ 2 W ≤ σ 2 W, max = ¯ τ 2 / E (cid:110) σ (¯ τz ) 2 (cid:111) .
If this cannot be satisﬁed, we set ¯ τ 2 = + ∞ .
In addition, let β = ασ 2 W > 0 .
With these, let us consider the following two ﬁxed point equations:
[Equation]
for z 1 , z 2 ∼ N (0 , 1) independently.
Then Eq.
(1), (2) and (3) together form the fundamental equations for deep random weight-tied autoencoders, in either one of the following senses:
Interpretation 1.
For 1 (cid:28) (cid:96) (cid:28) L , in the limit L → ∞ (and (cid:96) → ∞ at a pace sufﬁciently slow compared to L ), we have ¯ τ (cid:96) → ¯ τ , γ (cid:96)
→ γ and ρ (cid:96) → ρ , where ¯ τ is a stable ﬁxed point solution to ¯ τ 2 = T (cid:0) ¯ τ 2 (cid:1) , and ( γ, ρ ) is then jointly a stable ﬁxed point solution to γ = G ( γ, ρ ) and ρ = R ( γ, ρ ) .
In light of Theorem 1, (¯ τ, γ, ρ ) describes the behavior of an intermediate ˆ x (cid:96) : ˆ
[Equation]
where z ∼ N (cid:0) 0 , I n (cid:96) − 1 (cid:1) independent of x (cid:96) − 1 , and x (cid:96) − 1 ∼ =
¯ τ z (cid:48) for z (cid:48) ∼ N (cid:0) 0 , I n (cid:96) − 1 (cid:1) .
If the convergences ¯ τ (cid:96) → ¯ τ , γ (cid:96) → γ and ρ (cid:96) → ρ are fast, then the majority of the intermediate layers are well approximately described by the above, in the regime of large L .
Interpretation 2.
Suppose that for ¯ τ 0 = (cid:107) x (cid:107) / √ n 0 , we impose the constraint ¯ τ 20 = E (cid:110) σ (¯ τz ) 2 (cid:111) .
This should be interpreted as follows: starting with a chosen ¯ τ , we normalize the input data x
according to ¯ τ 20 = E (cid:110) σ (¯ τz ) 2 (cid:111) ; then we choose σ 2 W ≤ σ 2 W, max and σ 2 b according to Eq.
Under this constraint, it is easy to see that ¯ τ (cid:96) =
¯ τ for all (cid:96) ≥ 1 , and hence the norm of the input to each of the encoder’s hidden layers is preserved by Claim (a) of Theorem 1.
We then have that as L → ∞ , at small (cid:96) ≥ 2 (i.e., at the layers near the two ends of the autoencoder), γ (cid:96) → γ and ρ (cid:96) → ρ , where ( γ, ρ ) is jointly a stable ﬁxed point of γ = G ( γ, ρ ) and ρ = R ( γ, ρ ) .
The autoencoder’s output is then, in this limit,
[Equation]
for z ∼ N (0 , I n 0 ) independent of x .
In either cases, we see that ˆ x (cid:96) or ˆ x is a composition of a signal component and a Gaussian variation component.
Their respective strengths S sig and S var admit simple expressions, thanks to the inﬁniteL simpliﬁcation 5 .
We note that G and R do not take σ 2 b as a parameter.
We refer to Appendix C. for the computation of γ and ρ .
Fig.
1 shows that our asymptotic simpliﬁcation is quite accurate already for L on the order of a few tens.
We also remark that the equation ¯ τ 2 =
T (cid:0) ¯ τ 2 (cid:1) is known in the signal propagation analysis of random feedforward networks (Poole et al.
, but the equations γ = G ( γ, ρ ) and ρ = R ( γ, ρ ) are new.
We also observe that in these equations, there is the presence of α (through β ), which is typically missing from such analyses.
Hence unlike untied structures, one may expect to see architectural constraints in analyses of weight-tied structures.
3. P HASE TRANSITION OF THE FIXED POINT
With the inﬁnite depth simpliﬁcation, one question is on the existence of the solutions to Eq.
(2) and (3), and how these ﬁxed points look like.
(Eq.
(1) is well-studied, cf.
Poole et al.
; Hayou et al.
and we will not analyze it here.)
We note that γ = 0 is always a solution to Eq.
(2), in biases.
In addition, it is a common practice to initialize an untrained neural network (hence a random one) with zero biases (Goodfellow et al.
This assumption also helps making the analysis more tractable.
Other assumptions seem restrictive, but can be relaxed without affecting the analysis that follows, and hence are made for ease of presentation.
For example, the assumption α (cid:96) =
α for all (cid:96)
Recall we consider the limit of n → ∞ before taking L → ∞ .
Without a proof we expect that our results hold for L =
o (log n ) , so practically this means 1 (cid:28) L (cid:28) Ω (log n ) .
[Figure]
[Figure]
which case Eq.
(3) also has a solution, for instance, when ϕ (0) = 0 such as ReLU or tanh (which admits ρ = 0 ).
However γ = 0 is trivial, since it implies S sig is zero.
We will be interested in the existence of non-trivial and stable ﬁxed points.
To ease visualization, for the moment, let us consider
Eq.
(2) only.
Fig.
2 shows γ (cid:55)→ G ( γ, ρ ) for different ρ , β , ϕ and σ for ¯ τ 2 = 1 .
For a given ϕ and σ , depending on β and ρ , one may observe one or more ﬁxed points, one of which is at γ = 0 and can be stable or unstable.
When γ = 0 is the only ﬁxed point but is unstable, we have γ = ∞ as the “stable solution” to Eq.
The solution landscape changes drastically with β ; for instance, when σ = ϕ = tanh , γ = 0 is the only and stable ﬁxed point when β is small, but it becomes unstable and a new ﬁxed point at γ > 0 emerges when β is sufﬁciently large.
This hints at certain phase transition behaviors as β varies.
In Appendix C.2, we perform a detailed analysis of Eq.
(2) and (3), supported by several rigorously proven properties.
In the following, by an initialization for γ and ρ , we mean γ L +1 and ρ L +1 as in Section 2.2, and by convergence to γ and ρ , we mean the convergences as in Section 3.1.
We highlight some results from the analysis for speciﬁc pairs of ϕ and σ :
ReLU ϕ and σ .
We have two phase transitions at β = 2 and at β = 4 .
When β < 2 , with any initialization, we have convergence to γ = 0 and ρ = 0 .
When 2 < β < 4 , we have, with certain initializations, convergence to γ = 0 and divergence to ρ = + ∞ , and with certain other initializations, divergence to γ = + ∞ and ρ = + ∞ .
These include almost all possible initializations.
When β ≥ 4 , with any non-zero initialization, we have divergence to γ = + ∞ and ρ = + ∞ .
ReLU ϕ and tanh σ .
We have two phase transitions at β = 2 and β = β 0 (¯ τ ) ∈ (2 , ∞ ) .
When β < 2 , with any initialization, we have convergence to γ = 0 and ρ = 0 .
When 2 < β < β 0 , with any non-zero initialization, we have convergence to γ = 0 and divergence to ρ = + ∞ .
When β > β 0 , with any non-zero initialization, we have divergence to γ = + ∞ and hence ρ = + ∞ .
2 β     γργ / √ ρ    4 β 0.
γργ / √ ρ     5 β 0.
γργ / √ ρ
tanh ϕ and σ .
We have two phase transitions at β = 1 and β = β 0 (¯ τ ) > 1 .
When β ≤ 1 , we have convergence to γ = ρ = 0 .
When 1 < β < β 0 , with any non-zero initialization, we have convergence to γ = 0 and ρ ∈ (0 , 1) . When β > β 0 , with any non-zero initialization, we have convergence to γ > 0 and ρ ∈ (0 , 1) .
For ¯ τ > 0 , γ cannot grow to + ∞ as β varies.
We note that β 0 → 1 if ¯ τ 2 → 0 , and in the case α = 1 , this implies σ 2 W → 1 .
With respect to Eq.
(1), we then have σ 2 b → 0 .
An illustration is given in Fig.
tanh ϕ and ReLU σ .
We have a picture similar to the case ϕ
= σ = tanh , with a crucial difference that one cannot have β 0 be close to 1 .
An illustration is given in Fig.
γ and ρ thus exhibit phase transitions, depend crucially on the choice of activations (especially the decoder activation ϕ ), and can be trivialized (i.e., being zero or inﬁnity) as in the case of ReLU ϕ and σ .
It is remarkable that the above pictures are general for many other activations, as suggested by our analysis.
In the next sections, we explore the implications of these behaviors.
3. A PPROXIMATE INFERENCE AT INFINITE DEPTH
Theorem 1 gives a quantitatively exact sense of how the random weight-tied autoencoder performs “approximate inference”.
Here we will be interested in stronger notions.
A ﬁrst question is: does it explain reversibility?
Reversibility, as mathematically formalized in previous works Arora et al.
; Gilbert et al.
(2017), quantitatively concerns with how small the quantity E = (cid:107) x − ˆ x (cid:107) 2 /nis.
The smaller it is, the better the decoder “reverses” the encoder.
This formalized notion is an attempt to give a theoretical understanding of empirical ﬁndings that the input could be reproduced from the values of hidden layers of a trained feedforward network.
Let us now consider the inﬁnite depth simpliﬁcation under Interpretation 2 of Section 3.1.
We have E (cid:39) ( S sig − 1) 2 ¯ τ 20 + S 2var .
As such, for E ≈ 0 with high probability, one must have S sig
≈ 1 and S var ≈ 0 , hence ρ ≈ 0 .
Consequently, E (cid:110) ϕ ( σ (¯ τz 1 )) 2 (cid:111)
≈ 0 .
For ¯ τ > 0 , E (cid:110) ϕ ( σ (¯ τz 1 ))
2 (cid:111) = 0 is impossible for any non-trivial activations (unless the activation outputs zero almost everywhere).
Strikingly, in light of Section 3.2, when ϕ and σ are both ReLU, we have that S sig and S var are either 0 or + ∞ , in which case E ≥ ¯ τ 20 and can become unbounded.
While this does not contradict the results in Arora et al. (2015) (which also concerns with ReLU activations, but with speciﬁc choices of the biases and limited depth, and hence is in a different setting), our discussion suggests that random weight-tied models may be insufﬁcient to explain reversibility.
A second question is: does the model perform signal recovery?
In this case, we are interested in whether ˆ x ∼ =
c x for some constant c not necessarily 1 .
Similar to the above, this requires S var = 0 , hence ρ = 0 , and E (cid:110) ϕ
( βγσ (¯ τz 1 )) 2 (cid:111)
= 0 .
For non-trivial ϕ and σ , this requires ϕ (0) = 0 and γ = 0 .
Many activations do not conform with the former, and the latter implies ˆ x ∼ = 0 undesirably.
This provides a negative answer to the question.
A critic may argue that in expectation, E { ˆ x } ≈ S sig x , and as per Section 3.2, there are cases where γ > 0 and hence S sig > 0 .
Yet in fact, this in-expectation property can already be observed in the simple setting of linear shallow autoencoders (Arora et al.
What is ignored in such argument is that in many cases, S var > 0 whenever S sig > 0 , in light Section 3.2.
Our analysis hence mitigates the shortcoming of the in-expectation approach, and gives a more precise understanding of what the random weight-tied autoencoder can and cannot achieve when the depth becomes large.
3. C OMPARISON WITH THE SHALLOW CASE
Our result also allows for the case of L = 1 a shallow autoencoder.
In particular, taking a parallel setting with Section 3.1 (in particular, Interpretation 2), by Theorem 1,
[Equation]
in which, with ϕ hid being the activation in the hidden layer,
[Equation]
Some observations follow.
In the shallow case, γ > 0 and ρ > 0 and both are bounded regardless of the parameters, except for trivial edge cases such as ¯ τ 2 = 0 or ϕ hid ( · ) = 0 .
Furthermore, γ and ρ are independent of β , for a ﬁxed ¯ τ .
As such, there is no phase transition in γ and ρ as β changes.
We also have S sig ( β ) ∝ β and S var ( β ) ∝ √ β .
Hence, the signal component dominates with S sig /S var ∝ √ β .
Again this happens regardless of parameter choices.
In comparison with the inﬁnite depth case, for ϕ = σ = tanh or ϕ = tanh and σ being the
ReLU, as observed from Fig.
3, in certain regimes, γ , ρ and γ/ √ ρ can grow (sublinearly) with β , and hence S sig ( β ) =
Ω ( β ) 6 , S var ( β ) =
Ω (cid:0) √ β (cid:1) , and the signal component dominates with
S sig /S var =
Ω (cid:0) √ β (cid:1) .
In particular, near the phase transition of γ , S sig /S var = Ω (cid:0) β 1 . 5 (cid:1) .
Recalling that β = ασ 2 W , this implies for the inﬁnite depth case, as compared to the shallow one, ﬁrstly a slight perturbation in σ 2 W may result in a larger perturbation in the signal’s strength, and secondly an architecture using larger α may gain more in terms of ampliﬁcation of the signals.
In short, the deep autoencoder is more sensitive to slight changes in the parameters.
As evident in Section 3.2, the case ϕ being the ReLU also exhibits extreme sensitivity, in that it is possible for a slight perturbation in β to drastically change γ and ρ .
As suggested by Fig.
1, it should be the case already for L about a few tens.
It is, however, at the expense of much care in the selection of parameters, since there are continuous regimes in which the inﬁnite depth diminishes S sig and S var to zero or boost them to inﬁnity, a situation that never occurs in the shallow case.
Remark 4 .
Sensitivity to perturbations is implied by expressivity, a notion put forth in Poole et al.
(2016) in the study of random feedforward networks.
Hence we expect that sensitivity is a common feature of various types of deep neural networks.
3. I MPLICATIONS TO TRAINING INITIALIZATION
We examine the implications of Interpretation 1 in Section 3.1 to trainability of the weight-tied autoencoder.
We ﬁrst state our hypothesis, then test it with experiments.
The hypothesis.
Since the majority of intermediate layers can be described approximately by γ and ρ (as well as ¯ τ ) and the random weight-tied autoencoder is in fact one at initialization, appropriate values of γ , ρ and ¯ τ (by a suitable choice of σ 2 W and σ 2 b ) should lead to better trainability.
In particular, if one of them is ∞ , we expect numerical errors or too large values resulting in quick saturation, both of which render the autoencoder untrainable.
If γ = ρ = 0 in a neighborhood of the chosen σ 2 W and σ 2 b , we expect that the progress is slowed down in the beginning.
If such pitfalls are avoided, the autoencoder is expected to show a faster progress.
Our analysis in Section 3.2 shows that those pitfalls can occur for a wide range of parameters.
If the hypothesis is true, σ 2 W and σ 2 b must then be chosen carefully when L is large.
As a special remark on the case ϕ is the ReLU, when α = 1 , the hypothesis suggests taking σ 2 W = 2 and σ 2 b = 0 .
This coincides with the celebrated He initialization (He et al.
(2015)), which however considers feedforward networks only.
Interestingly this does require σ to be the ReLU; for instance, σ can be tanh , in which case the argument in (He et al.
(2015)) is not applicable.
We shall also examine edge of chaos (EOC) initializations (Schoenholz et al.
; Pennington et al.
(see Appendix E.1).
The EOC initialization enables better signal propagation in deep feedforward networks, and in our context, is relevant to the encoder part with the activation σ .
[Table]
indicates γ = ρ = 0 in a neighborhood.
“Inf” indicates either γ → ∞ or ρ → ∞ .
The schemes with ϕ = tanh should be reﬂected against Fig.
Experiments.
Table 1 lists several initialization schemes with α = 1 , which are chosen such that the hypothesis can be tested separately for each pair ϕ and σ .
We perform simple experiments on a weight-tied vanilla autoencoder as described in Section 3.1: L = 100 , all hidden dimensions of 400 , identity input activation σ 0 , and decoder biases initialized to zero.
This sets α (cid:96) =
α = 1 for (cid:96) ≥ 2 ; here α 1 (cid:54) = 1 is irrelevant in light of Interpretation 1.
We train the autoencoder on the MNIST dataset with mini-batch gradient descent with a batch size of 250 and without regularizations, for 5 × 10iterations (equivalent to 2500 epochs).
We perform the experiments in two settings:
• Setting 1: The output activation ϕ 0 is tanh , MNIST images are normalized to [ − 1 , +1] , and the learning rate is ﬁxed at 5 × 10 − 3 .
This is standard for MNIST.
• Setting 2: ϕ 0 is the identity, MNIST images are unnormalized (i
e., normalized to [0 , +1] ), and the learning rate is ﬁxed at 3 × 10 − 3 .
This is common for regression.
These learning rates are chosen so that the learning dynamics is typically smooth, in light of recent works Mei et al.
; Smith & Le (2018).
We use the normalized (cid:96) 22 loss (cid:107) ˆ x − x (cid:107) 2 / (cid:107) x (cid:107) 2 , and are primarily interested in this loss as a quality measure, since we only focus on trainability 7 .
We also do not apply techniques such as greedy layer-wise pre-training, drop-out or batch normalization.
The results are plotted in Fig.
See also Appendix D.1 for visualization of the reconstructions, and
Appendix D.2 for the evolution over a broader range of parameters.
Note that we plot the evolution in the logarithmic scale of time, since it is typically smooth and revealing on this scale, as found in prior works Baity-Jesi et al.
Mei et al.
(2018) and also evident from the plots.
Discussion.
The results are in good agreement with our hypothesis.
(Recall we test the hypothesis separately for each pair ϕ and σ , for which the involved schemes share the same architecture and only differ in the initialization.)
Note that as predicted, in Setting 2, Scheme 3 and 6 are trapped with numerical errors, and in Setting 1, they saturate quickly at a high loss.
As such, we do not include the results of Scheme 3 and 6 in Fig. 4.
[Figure]
We see from the ﬁgure that Scheme 2, 5, 8, 10 and 14 show much slower progresses, by a factor of 3 to 10 times in terms of training iterations to reach the same loss.
Hence a good amount of training time can be saved by an appropriate initialization.
Interestingly Scheme 5 is in fact a special
EOC initialization that Pennington et al.
(2017) found to be the better one among all EOC schemes with Gaussian weights for tanh activation.
This last observation shows that having good signal propagation through the encoder is far from being a sufﬁcient condition for trainability.
Among the schemes, only Scheme 1 and 4 in Setting 1 and only Scheme 1, 4 and 7 in Settinghave their eventual trained networks produce meaningful reconstructions, whereas the rest always output some “average” of the training set regardless of the input, at the end of 5 × 10 5 iterations (see Appendix D.1).
It is unclear whether this is a bad local minimum, or whether these schemes take much longer to show further progresses.
An explanation is beyond our current theory, and it is an open question how to create a scheme with meaningful trainability.
Remarkably all the schemes that show slower initial progresses (Scheme 2, 5, 8, 10 and 14) are among those that could not yield meaningful reconstructions.
We observe that in Setting 2, the tanh network under Scheme 7 is best performing in terms of the reconstruction loss, and its progress does not seem to reach a plateau after 5 × 10 5 iterations.
In both settings, Scheme 4, which is a hybrid of ReLU and tanh activations, shows slight improvements over Scheme 1, which is a purely ReLU network.
This extends the conclusion in Pennington et al.
(2017) to the context of weight-tied autoencoders: reasonable training at a large depth is possible even for the notoriously difﬁcult tanh activation, and this necessarily requires careful initializations.
Overall we see that our experiments conﬁrm the hypothesis, showing an intimate connection between the phase transition behaviors found by our theory and trainability of the autoencoders.
D ISCUSSION
This paper has shown quantitative answers to the three questions posed in Section 1.
This feat is enabled by an exact analysis via Theorem 1.
The theorem is stated in a general setting, allowing varying activations, weight variances, etc, but our analyses in Section 3 have made several simpliﬁcations.
This leaves a question of whether these simpliﬁcations can be relaxed, and how the picture changes accordingly, for instance, when the parameters vary across layers, similar to Yang & Schoenholz (2018).
Many other questions also remain.
For example, what would be the covariance structure between the outputs of two distinct inputs?
How does the network’s Jacobian matrix look like?
These questions have been answered in the feedforward case (Poole et al.
; Pennington et al.
, but we believe answering them is more technically involved in our case.
We have also seen that an autoencoder that shows initial progress may not necessarily produce meaningful reconstruction eventually after training, and hence much more work is needed to understand the training dynamics far beyond initialization.
Recent works Mei et al.
; Rotskoff & VandenEijnden (2018); Sirignano & Spiliopoulos (2018); Chizat & Bach (2018) have made progresses in this direction for shallow networks.
A P ROOF OF T HEOREM
In the following, we give an outline of the proof of Theorem 1, and the complete proof.
First, we start with a few notations and deﬁnitions.
A. D EFINITIONS AND N OTATIONS
We recall the setting in Section 2.1 (see also Fig.
5 for a schematic diagram of the autoencoder).
We deﬁne { g (cid:96) : R n (cid:96) − 1 (cid:55)→ R n (cid:96) − 1 } (cid:96) =1 ,...,L inductively as follows:
[Equation]
It is easy to see that ˆ x (cid:96) =
g (cid:96) ( x (cid:96) − 1 ) for (cid:96) = 1 , ..., L .
Essentially g (cid:96) ’s represent the autoencoding mappings computed by the inner layers.
We use bold-face letters (e.g. x , W ) to denote vectors or matrices.
We use C throughout to denote an arbitrary (and immaterial) constant that is independent of the dimensions.
For two vectors x and y , (cid:104) x , y (cid:105) denotes their inner product.
We use I for the identity matrix, and I n to emphasize its dimensions n × n .
We use (cid:107)·(cid:107) to denote the usual Euclidean norm, (cid:107)·(cid:107) ∞ the inﬁnity norm, and (cid:107) M (cid:107) 2 the maximum singular value of a matrix M .
For a sigma-algebra F and two random variables X and Y , X |F d = Y means that for any integrable function φ and any F -measurable bounded
random variable Z , E { φ ( X ) Z } =
E { φ ( Y ) Z } .
We write X d = Y when F is the trivial sigma-algebra.
A sequence of functions φ n : R n (cid:55)→ R is uniformly pseudo-Lipschitz if there exists a constant C , independent of n , such that for any x , y ∈ R n ,
[Equation]
A sequence of functions φ n :
R n (cid:55)→ R n is uniformly Lipschitz if there exists a constant C , independent of n , such that for any x , y ∈ R n , (cid:107) φ n ( x ) − φ n ( y ) (cid:107) ≤ C (cid:107) x − y (cid:107) .
These deﬁnitions are adopted from Berthier et al.
For two sequences of random variables X n ∈ R and Y n ∈ R indexed by n , we write X n (cid:39) Y n to mean that X n − Y n → 0 in probability.
The same meaning holds when Y n is deterministic.
For two sequences of random vectors X n ∈ R n and Y n ∈ R n indexed by n , we write X n ∼ =
Y n if for any sequences of uniformly pseudo-Lipschitz test functions φ n : R n (cid:55)→ R , φ n ( X n ) (cid:39) E { φ n ( Y n ) } (and hence in this context, we do not need X n and Y n to be deﬁned on a joint probability space).
A. O UTLINE OF THE PROOF OF T HEOREM
We state several results that are key to prove Theorem 1.
Proposition 5.
Consider the asymptotic setting n → ∞ , with some sequence m = m ( n ) such that m/n → α > 0 as n → ∞ .
Let ϕ : R (cid:55)→ R and σ : R (cid:55)→ R be Lipschitz continuous scalar functions.
Consider a sequence of uniformly Lipschitz functions g : R m (cid:55)→ R m , and sequences of vectors b ∈ R m , v ∈ R n , u ∈ R n such that
[Equation]
for all sufﬁciently large n .
Let us deﬁne
[Equation]
where W ∈ R m × n with W ij ∼ N (cid:0) 0 , σ 2 W /n (cid:1) i.i.d.
for σ W > 0 .
Let γ =
E ˜ z
[Equation] [Equation]
for ˜ z ∼ N (0 , I m ) .
Then:
[Equation]
where z ∼ N (0 , I n ) (independent of u , b and v ).
Here the randomness is solely due to W .
A corollary follows from this proposition.
Corollary 6.
Consider the same setting as in Proposition 5.
Further assume that b ∼ N (cid:0) 0 , σ 2 b I m (cid:1) , v ∼ N (cid:0) 0 , σ 2 v I n (cid:1) and u ∼ N (cid:0) 0 , σ 2 u I n (cid:1) , independent of each other and of W .
Then:
[Equation]
for z , z
[Equation]
∼ N (0 , I n ) independently.
In particular, for φ : R (cid:55)→ R Lipschitz continuous,
[Equation]
in which
[Equation]
+ σ 2 b , where ˜ z ∼ N (0 , I m ) , and z 1 , z 2 are independently distributed as N (0 , 1) .
In a nutshell, the proposition and the corollary consider a random weight-tied “autoencoder” with a single hidden layer, with a mapping g in the middle.
Since g is not a separable function (i.e., g does not apply entry-wise), this is different from the usual shallow autoencoder, a case that has been investigated in Pennington & Worah (2017);
Louart et al.
(2018) with techniques and objectives different from ours.
By understanding this structure, one can understand the random weight-tied multi-layer autoencoder.
Indeed, for each (cid:96) , the proposition applies with u being x (cid:96) − 1 , f ( u ) being ˆ x (cid:96) and g being g (cid:96) +1 .
In other words, this studies the mapping g (cid:96) .
One can start with g L , then progressively move to outer layers g L − 1 , g L − 2 , etc, and hence analyze the autoencoder completely by repeating the same procedure L times.
We note that in doing so, at each step, one requires certain information about the inner layers to perform calculations for the outer layers, and this is worked out in Corollary 6 via a simple recursive relation.
In particular, the left-hand sides of Eq.
(4) and Eq.
(5) play the role of ρ and γ of the outer layer, whereas their right-hand sides involve ρ and γ from the inner layers.
It is also important to note that the assumption that the weights at different layers are independent is crucial in making u , g and ( W , b , v ) to be independent, allowing the proposition to be applicable at all steps.
This is the idea behind the proof of Theorem 1.
We quickly mention the key proof technique for Proposition 5.
The main technical challenge in working with the weight-tied structure W (cid:62) h ( W u ) , for some h : R m (cid:55)→ R m and W ∈ R m × n
Gaussian, is that whereas y = W u is Gaussian with zero mean thanks to independence between u and W , W (cid:62) h ( y ) is not since y is correlated with W .
It is observed in Bolthausen (2014) that conditioning on a linear constraint y =
W u (or the sigma-algebra F generated by y and u ), one has that W is distributed as a conditional projection component plus an independent Gaussian component:
[Equation]
where P ⊥ u is an appropriate projection.
Here E { W |F} is what propagates the information about u .
In addition, ˜ W is Gaussian and independent of F , and hence exact calculations can be then worked out.
We note that the assumption W is Gaussian is crucial for this identity to hold.
A. P ROOF OF P ROPOSITION
We state the Gaussian Poincar ´e inequality, which will be used multiple times throughout the proof.
We remark that the use of the Gaussian Poincar ´
e inequality is unlikely to lead to a tight nonasymptotic result, but is sufﬁcient for our asymptotic analysis.
Theorem 7 (Gaussian Poincar ´e inequality) .
For z ∼ N (cid:0) 0 , σ 2 I n (cid:1) and φ :
R n (cid:55)→ R continuous and weakly differentiable, there exists a universal constant C such that Var { φ ( z ) } ≤
[Equation]
Now we are ready for the proof.
Step 1.
We perform Gaussian conditioning.
Let y
=
W σ ( u ) .
Let F be the sigma-algebra generated by y and u .
Conditioning on F is equivalent to conditioning on the linear constraint y =
W σ ( u ) .
Following Bayati & Montanari (2011), we have
[Equation]
where ˜ W d = W and is independent of F , P σ ( u )
=
σ ( u ) σ ( u ) (cid:62) / (cid:107) σ ( u ) (cid:107) 2 the projection onto σ ( u ) , and P ⊥ σ ( u ) =
I − P σ ( u ) the corresponding orthogonal projection.
As such, since ϕ ( g ( y + b )) is F -measurable,
[Equation]
For a sequence of uniformly pseudo-Lipschitz functions φ n :
[Equation]
n
[Equation]
R :
[Equation]
Up to this point, there is no need for the asymptotics n → ∞ .
The rest of the proof focuses on Φ n .
Step 2.
We show that τ , γ and ρ are uniformly bounded as n → ∞ .
This is trivial for τ , and we note τ > 0 .
Consider ρ .
We have for any r ∈ R m ,
[Equation]
for sufﬁciently large m .
It is then easy to see that ρ is uniformly bounded.
Regarding γ , | γ |
[Equation]
˜
[Equation]
by Cauchy-Schwarz inequality.
Therefore γ is also uniformly bounded.
Step 3.
We analyze the ﬁrst term in Φ n .
Notice that y d = τ ˜ z for some ˜ z ∼ N (0 , I m ) .
In addition, the mapping y (cid:55)→ (cid:104) y , ϕ ( g ( y + b )) (cid:105) /m is uniformly pseudo-Lipschitz, by noticing that
[Equation]
along with the fact that y (cid:55)→ ϕ
( g ( y + b )) is uniformly Lipschitz, and Eq.
As such, by Theorem 7,
Var
[Equation]
which tends to 0 as n → ∞ .
By Chebyshev’s inequality
[Equation]
we thus have
[Equation]
Step 4.
We analyze the second term in Φ n .
We have:
[Equation]
Note that the mapping y (cid:55)→ 1 m (cid:107) ϕ
( g ( y + b ))
(cid:107) 2 is uniformly pseudo-Lipschitz by a similar argument.
Hence again by Theorem 7, Var (cid:110) 1 m (cid:107) ϕ
( g ( y + b ))
(cid:107) 2 (cid:111) =
O (1 /m ) , which yields
[Equation]
Recall that ˜ W is independent of F , and as such, there exists a random variable z (cid:48) ∼ N
[Equation]
1) independent of y such that
[Equation]
which converges to 0 in probability, where we have used the fact that ρ is uniformly bounded from
Step 2.
Furthermore, there also exists a random vector z ∼ N (0 , I n ) independent of F such that
[Equation]
Step 5.
We ﬁnish the proof.
From the deﬁnition of uniform pseudo-Lipschitz functions and Eq.
(8) and (9), we obtain:
[Equation]
Here notice that (cid:13)(cid:13)(cid:13) P ⊥ σ ( u ) (cid:13)(cid:13)(cid:13) 2 = 1 , and as a standard fact from the random matrix theory, (cid:13)(cid:13)(cid:13) ˜ W (cid:13)(cid:13)(cid:13) 2 ≤ c (cid:0) α, σ 2 W (cid:1) a constant with high probability (see e.g. Vershynin (2012)).
Furthermore, 1 √ n (cid:107) z (cid:107) ≤ C with high probability due to the law of large numbers.
Combining with the facts from Step 2, 3 and 4, it is then easy to see that
[Equation]
Finally, since γ and ρ are uniformly bounded from Step 2, it is easy to show that the mapping
[Equation]
is uniformly pseudo-Lipschitz.
Hence by Theorem 7,
Var
[Equation]
which yields
[Equation]
Together with Eq.
(6) and (10), this completes the proof.
A. P ROOF OF C OROLLARY
By Proposition 5, for any sequence of uniformly pseudo-Lipschitz φ n : R n (cid:55)→ R :
[Equation]
for z ∼ N (0 , I n ) independent of u , v and b , in which
[Equation]
for ˜ z ∼ N (0 , I m ) independent of u and b .
It is easy to see that τ 2 (cid:39) τ
2 = ¯ τ 2 − σ 2 b .
For any t ∈ (0 , ∞ ) , the mapping b (cid:55)→ γ ( t, b ) is uniformly pseudo-Lipschitz, and hence by Theorem 7,
Var b { γ ( t, b ) } =
O (1 /m ) , which yields γ ( t, b ) (cid:39) E b { γ ( t, b ) } .
Recall τ is independent of b .
As such, γ ( τ, b ) (cid:39) E b { γ ( τ, b ) } .
Furthermore, the mapping t (cid:55)→ E b { γ ( t, b ) } is continuous and hence E b { γ ( τ, b ) } (cid:39) E b { γ ( τ, b ) } .
Performing a similar argument for ρ ( τ, b ) , we thus have:
[Equation]
We note by Stein’s lemma,
[Equation]
Also notice that
[Equation]
Direct algebras then yield
[Equation]
and hence γ (cid:39) γ .
Therefore,
[Equation]
The mapping ( u , v ) (cid:55)→ h ( u , v ) is uniformly pseudo-Lipschitz and hence by Theorem 7,
Var u , v { h ( u , v ) } =
O (1 /n ) , which implies
[Equation]
which follows from the fact that u , v and z are independent and normally distributed.
The ﬁrst claim is hence proven.
Eq.
(4) follows immediately from the above.
The proof of Eq.
(5) is similar and hence omitted.
A. P ROOF OF T HEOREM
The following lemma states that, roughly speaking, g (cid:96) is uniformly Lipschitz (indexed by n ) with high probability.
First, recall that W (cid:96) =
W (cid:96) ( n ) is indexed by n , and one can easily construct a joint probability space on which the sequence { W (cid:96) ( n ) } n ≥ 1 is deﬁned.
The exact construction is immaterial, so long as for each n , the marginal distribution satisﬁes the setting in Section 2.1.
We work in this joint space in the following.
Lemma 8.
For each (cid:96) = 1 , ..., L , there exists a ﬁnite constant c (cid:96) > 0 such that P (cid:8) E N(cid:96) (cid:9) → 0 as
N
→ ∞ , where the event E N(cid:96) is deﬁned as
[Equation]
Proof.
Consider (cid:96) =
L .
Recall that g L ( u ) =
W (cid:62) L ϕ L ( W L σ L [Equation] 1 ( u ) + b L ) + v L .
Then for any u 1 and u 2 ,
[Equation]
where steps ( i ) and ( ii ) are because ϕ L and σ L − 1 are Lipschitz.
It is a standard fact in the random matrix theory that, for each (cid:96) = 1 , ..., L , there exist ﬁnite constants η
(cid:96) =
η (cid:96) (cid:16) α (cid:96) , σ 2 W,(cid:96) (cid:17) > 0 and c (cid:48)
[Equation]
=
c (cid:48) (cid:96) (cid:16) α (cid:96) , σ 2 W,(cid:96) (cid:17) >
such that P (cid:110) (cid:107) W (cid:96) (cid:107) 22 > η (cid:96) (cid:111) ≤ e − c (cid:48) (cid:96) n (see e.g. Vershynin (2012)).
As such, using this fact for (cid:96)
=
L , by the union bound,
[Equation]
which proves the claim for (cid:96)
=
L .
To see the claim for general (cid:96)
[Equation]
we have from a similar calculation:
[Equation]
Therefore [Equation] sup
[Equation]
with probability at least
[Equation]
by the union bound.
This proves the claim
[Equation]
Lemma 9.
For each (cid:96) ≥ 0 , τ (cid:96) +1 and ¯ τ (cid:96) +1 are ﬁnite and strictly positive.
Furthermore,n (cid:96) (cid:107)
σ (cid:96) ( x (cid:96) ) (cid:107) 2 (cid:39) τ 2 (cid:96) +1 /σ 2 W,(cid:96)
+1 , and x (cid:96) +1 ∼ =
¯ τ (cid:96) +1 z for z ∼ N (cid:0) 0 , I n (cid:96) +1 (cid:1)
Proof.
We prove the lemma by induction.
The claim is trivially true for (cid:96) = 0 by assumption.
Assume the claim for some (cid:96) ≥ 0 .
Due to independence, conditioning on x (cid:96) ,
[Equation]
where z ∼ N (cid:0) 0 , I n (cid:96) +1 (cid:1) independent of x (cid:96) .
Since 1 n (cid:96) (cid:107) σ (cid:96) ( x (cid:96) ) (cid:107) 2 (cid:39) τ 2 (cid:96) +1 /σ 2 W,(cid:96)
+1 , we then have
[Equation]
which implies
[Equation]
for z (cid:48) ∼ N (0 , 1) .
By the induction hypothesis, ¯ τ (cid:96) +1 > 0 .
Hence by assumption, the right-hand side is ﬁnite and strictly positive.
One also recognizes that it is equal to τ 2 (cid:96)
+2 /σ 2 W,(cid:96)
+2 .
Since σ W,(cid:96)
+2 is strictly positive and ﬁnite, so is τ (cid:96) +2 , and hence so is ¯ τ (cid:96) +2 .
This completes the proof.
We are now ready for the proof of Theorem 1.
Claim (a) of the theorem, which follows directly from
Lemma 9, is just a forward pass through the encoder and hence is the same as in the case of random feedforward networks (without weight tying)
Poole et al.
For (cid:96) = 2 , ..., L , let H (cid:96) denote the following three claims:
[Equation]
σ (cid:96) − 1 (¯ τ (cid:96) − 1 z 1 ) + (cid:113) α
(cid:96) σ 2 W,(cid:96)
ρ (cid:96) +1 + σ 2 v,(cid:96)
z 2 , (2) : n (cid:96) −
[Equation]
(cid:107) ϕ
(cid:96) −
[Equation]
( g (cid:96) (¯
[Equation]
(cid:96) − 1 ˜
[Equation]
(cid:96) −
[Equation]
) (cid:107) 2 (cid:39) ρ (cid:96) , (3) : n (cid:96) − 1 ¯ τ 2 (cid:96) − 1 (cid:104) ¯ τ (cid:96) − 1 ˜ z (cid:96) − 1 , ϕ (cid:96) − 1 ( g (cid:96) (¯ τ (cid:96) − 1 ˜ z (cid:96) − 1 )) (cid:105) (cid:39) γ (cid:96) , for z 1 , z 2 , ∼ N (cid:0) 0 , I n (cid:96) − 1 (cid:1) independently, and ˜ z (cid:96) − 1 ∼ N (cid:0) 0 , I n (cid:96) − 1 (cid:1) independent of g (cid:96) , with the understanding that g L +1 is the identity.
We prove them by induction, and Claim (b) then follows immediately.
We ﬁrst note that with high probability, for any (cid:96) , 1 n (cid:96) (cid:107) b (cid:96) (cid:107) 2 and 1 n (cid:96) − 1 (cid:107) v (cid:96) (cid:107) 2 are bounded by the law of large numbers.
Consider H L .
Note that x L − 1 is independent of g L , and thatn L − 1 (cid:107) σ L − 1 ( x L − 1 ) (cid:107) 2 converges to a ﬁnite non-zero constant in probability by Lemma 9.
Hence by Corollary 6, with g being the identity, f being g L , u being x L − 1 (recalling ˆ x L = g L ( x L − 1 ) ) and
φ being ϕ L − 1 , and the fact x
L − 1 ∼ =
¯
τ L − 1 ˜ z (cid:96) − 1 from Claim (a), H L is proven.
Assuming H (cid:96) +1 for some (cid:96) , we prove H (cid:96) .
Recall that g (cid:96) +1 is independent of Θ (cid:96) =
{ W (cid:96) , b (cid:96) , v (cid:96) , x (cid:96) − 1 , ˜ z (cid:96) − 1 } , and g (cid:96) is independent of x (cid:96) − 1 and ˜ z (cid:96) − 1 .
Also from Claim (a), x (cid:96) − 1 ∼ =
¯ τ (cid:96) − 1 ˜ z (cid:96) − 1 .
Consider some N ∈ N and let E N(cid:96)
+1 denote the event as deﬁned in Lemma 8.
On the event ¬E N(cid:96) +1 (the complement of E N(cid:96) +1 ), by Corollary 6, with respect to the randomness of Θ (cid:96) ,
[Equation]
[Equation] [Equation]
, in which
[Equation]
Note that here γ and ρ are functions of g (cid:96) +1 and hence random, and z (cid:48) ∼ N (0 , I n (cid:96) ) independent of g (cid:96) +1 .
On the event ¬E N(cid:96) +1 , with the fact that ¯ τ (cid:96) is ﬁnite and non-zero by Lemma 9, we have the mapping
[Equation]
is uniformly pseudo-Lipschitz, and hence applying Theorem 7, one obtains that
[Equation]
with the right-hand side independent of g (cid:96) +1 for n > N .
Consequently, due to Lemma 8 and
Chebyshev’s inequality, for any (cid:15) > 0 ,
[Equation]
where o n (1) → 0 as n → ∞ .
On the other hand, since z (cid:48) is independent of g (cid:96) +1 , invoking H (cid:96) +1 , P [Equation] z
[Equation]
− γ
[Equation]
[Equation]
> (cid:15)
[Equation]
n
[Equation]
As such,
[Equation]
Letting n → ∞ then N → ∞ , we then have γ (cid:39) γ (cid:96) +1 .
Similarly, we also have ρ (cid:39) ρ (cid:96) +1 .
It is then easy to deduce H (cid:96) , recalling the deﬁnitions of γ (cid:96) and ρ (cid:96) .
The claim that ¯ τ (cid:96) − 1 z 1 can be replaced with x (cid:96) − 1 in the expression for ˆ x (cid:96) in Claim (b) can be recognized easily by doing the same replacement in the proof of Corollary 6 and the above proof.
The proof of Claim (c) is similar.
We omit these repetitive steps.
B N UMERICAL VERIFICATION OF T HEOREM
We perform simple simulations to verify Theorem 1 at ﬁnite dimensions.
In particular, we simulate a random weight-tied autoencoder, as described in Section 2.1, with L = 50 , σ 2 W,(cid:96)
= 2 .
312 , σ 2 b,(cid:96) = 0 .
211 , σ 2 v,(cid:96) = 0 , ϕ (cid:96) =
σ (cid:96) =
tanh , identity ϕ 0 and σ 0 , α (cid:96) = 1 , and consequently n (cid:96) = n 0 , for an input x ∈ R n 0 whose ﬁrst half of the entries are +1 and the rest are − 1 .
Then we compute the following:
[Equation]
(cid:96) ˆ γ 2 (cid:96) + n (cid:96) − [Equation] (cid:107) σ (cid:96) − 1 ( x (cid:96) − 1 ) (cid:107) 2 − [Equation]
2 v,(cid:96) (cid:33) . We also compute { γ (cid:96) , ρ (cid:96) } (cid:96) =2 ,...,L +1 as in Section 2.2, and γ and ρ as in Section 3.1.
Theorempredicts that ˆ γ (cid:96) (cid:39) γ (cid:96) and ˆ ρ (cid:96) (cid:39) ρ (cid:96) .
Section 3.1 also asserts that γ (cid:96) ≈ γ and ρ (cid:96) ≈ ρ for (cid:96) (cid:28) L .
Fig.
6 shows the results for n 0 = 500 and n 0 = 2000 .
We observe a quantitative agreement already for n 0 = 500 , and this improves with larger n 0 .
Next we verify the normality of the variation component in ˆ x .
We compute
[Equation]
Its empirical distribution should be close to N (0 , 1) , in light of Theorem 1.
We make this comparison in Fig.
7, and again observe good agreement already for n 0 = 500 .
Finally we re-simulate the autoencoder with different distributions of the weights W (cid:96) .
In particular, we try with the Bernoulli distribution, the uniform distribution, and the Laplace distribution, with their means and variances adjusted to zero and σ 2 W,(cid:96) /n
(cid:96) − 1 respectively.
The results for n 0 =are plotted in Fig.
8 and 9.
We observe good quantitative agreements between the simulations for these non-Gaussian distributions and the prediction as in Theorem 1, although the theorem is proven only for Gaussian weights.
[Figure]
ˆ γ ℓ γ ℓ γ ˆ ρ ℓ ρ ℓ ρ
[Figure]
ℓ
[Figure]
[Figure]
ˆ γ ℓ γ ℓ γ ˆ ρ ℓ ρ ℓ ρ
[Figure]
[Figure]
distributions of the weights.
The setting is described in Appendix B.
We take a single run for the simulation of the autoencoder.
Here n 0 = 2000 .
[Figure]
t
[Figure]
o
[Figure]
C O N THE FIXED POINT EQUATIONS OF γ AND ρ C. C OMPUTATION OF γ AND ρ
Computing γ and ρ as from Eq.
(2) and (3) amounts to evaluating double integrals.
For simple ϕ , such as the ReLU, the integrals f ϕ ( a, b ) =
E { ϕ ( a + bz ) } and g ϕ ( a, b ) = E (cid:110) ϕ
( a + bz ) 2 (cid:111) , for z ∼ N (0 , 1) , can be calculated in closed forms.
In such cases, one can make reduction to one-dimensional integrals:
[Equation]
We proceed with computing γ and ρ as follows.
From a random initialization γ (0) > 0 and ρ (0) > 0 , one iteratively updates γ ( t +1) =
G (cid:0) γ ( t ) , [Equation] ( t ) (cid:1) and ρ ( t
[Equation]
[Equation]
R (cid:0) γ ( t ) , ρ ( [Equation] ) (cid:1) , and stops when the incremental update is negligible or the number of iterations exceeds a threshold.
Upon convergence, this procedure ﬁnds a stable ﬁxed point.
C.
[Equation]
P ROPERTIES OF γ AND ρ
We prove several properties of γ and ρ .
We recall G ( γ, ρ ) and R ( γ, ρ ) from Eq.
(2) and (3) for ease of reading:
[Equation]
for z 1 , z 2 ∼ N (0 , 1) independently, where ( i ) is due to Stein’s lemma.
Recall that the ﬁxed points equations are γ = G ( γ, ρ ) and ρ = R ( γ, ρ ) .
We also recall that β
= ασ 2 W > 0 .
In light of
Remark 3, we will consider Lipschitz continuous, non-decreasing ϕ and σ , so that γ ≥ 0 (and of course, ρ ≥ 0 ).
We will study these equations, ﬁrst by stating some propositions, then discussing their implications, although we caution that the link between the propositions and the suggested implications is not entirely rigorous.
All the proofs are deferred to Section C.3.
We note that while the discussions concern with ReLU or tanh activations, the propositions apply to broader classes of functions.
In the following, when we say an initialization for γ and ρ , we mean either an initialization in the context of an iterative process to ﬁnd the ﬁxed points as in Section C.1, or γ L +1 and ρ L +1 as in
Section 2.2 in the context of autoencoders with L → ∞ .
We also say γ is a ﬁxed point, without referencing to ρ , to mean that it is only a ﬁxed point of γ = G ( γ, ρ ) for a given ρ , and similarly for ρ .
When we mention both γ and ρ as a ﬁxed point, we mean a ﬁxed point to both γ =
G ( γ, ρ ) and ρ = R ( γ, ρ ) .
We will use ∂ ku f to denote the k th-order partial derivative of f with respect to u .
C.2. T HE CASE OF R E LU ϕ
The following result is exclusive to ReLU ϕ .
Proposition 10.
Consider that ϕ is the ReLU and σ is Lipschitz continuous and non-decreasing:
(a) γ = 0 and ρ = 0 is a ﬁxed point.
Furthermore, at γ = 0 and β (cid:54) = 2 , the mapping ρ (cid:55)→ R (0 , ρ ) admits ρ = 0 as the only ﬁxed point, which is stable if β < 2 and unstable if β > 2 .
Also at γ = 0 and β = 2 , any ρ is a stable ﬁxed point.
(b) Assume σ is positive on a set of positive Lebesgue measure.
If γ → + ∞ , it must be that ρ → + ∞ .
(c) Consider ¯ τ ∈ (0 , ∞ ) .
Assume σ is non-zero on a set of positive Lebesgue measure, and that σ ( u ) =
for all u ≤ 0 .
Then no γ > 0 is a stable ﬁxed point.
Furthermore, • if β E { σ (cid:48) (¯ τz 1 ) } ≤ 1 , there is only one ﬁxed point at γ = 0 , which is stable; • if β E { σ (cid:48)
(¯ τz 1 ) } ∈ (1 , 2) , there are two: one at γ = 0 , which is stable, and the other at γ > 0 , which is unstable;
• if β E { σ (cid:48) (¯ τz 1 ) } ≥ 2 , there is only one ﬁxed point at γ = 0 , which is unstable.
(d) Assume σ is non-zero on a set of positive Lebesgue measure, and that σ is an odd function.
Then for any ρ , the mapping γ (cid:55)→ G ( γ, ρ ) is a straight line through the point (0 , 0) .
Furthermore, if β E { σ (cid:48) (¯ τz 1 ) } (cid:54) = 2 , there is only one ﬁxed point at γ = 0 , which is stable for β E { σ (cid:48)
(¯ τz 1 ) } < 2 and unstable for β E { σ (cid:48) (¯ τz 1 ) } > 2 ; if β E { σ (cid:48) (¯ τz 1 ) }
= 2 , any γ is a ﬁxed point.
(e) Assume σ is odd, and consider β > 2 .
Given γ ≥ 0 , we have R ( γ, ρ ) > ρ for all ρ ≥ 0 .
The proposition suggests the following picture.
First consider σ is ReLU.
Since E { σ (cid:48) (¯ τz 1 ) }
=
P (¯ τz 1 ≥ 0)
= 0 .
5 , we have two phase transitions at β = 2 and at β = 4 .
In particular, based on
Proposition 10:
• When β < 2 , with any initialization, we have convergence to γ = 0 and ρ = 0 .
This is based on Claim (a) and (c).
• When β ∈ (2 , 4) , with certain initializations, we have convergence to γ = 0 and divergence to ρ = + ∞ ; with certain other initializations, we have divergence to γ = + ∞ and ρ = + ∞ .
This excludes a special initialization at the unstable ﬁxed point, which is a singleton and essentially a rare case.
This is based on Claim (a), (b) and (c).
• When β ≥ 4 , with any non-zero initialization, we have divergence to γ = + ∞ and hence ρ = + ∞ .
This is based on Claim (b) and (c).
Now we consider σ is tanh .
Let β 0 = β 0 (¯ τ ) = 2 / E { σ (cid:48) (¯ τz 1 ) } , and it is easy to see that β 0 >since σ = tanh .
The following picture is then expected:
• When β < 2 , with any initialization, we have convergence to γ = ρ = 0 .
This is based on Claim (a) and (d).
• When β ∈ (2 , β 0 ) , with any non-zero initialization, we have convergence to γ = 0 and divergence to ρ = + ∞ .
This is based on Claim (a) and (d).
• When β > β 0 , with any non-zero initialization, we have divergence to γ = + ∞ and ρ = + ∞ .
This is based on Claim (b) and (d).
• When β = β 0 , we have that γ is unchanged from the initialization.
Since β 0 > 2 , we then have divergence to ρ = + ∞ .
This is based on Claim (b) and (e).
One crucial property of the ReLU is that it is unbounded at inﬁnity and its derivative at inﬁnity is bounded away from zero.
This allows γ and ρ to grow to inﬁnity.
This is a stark contrast to the case ϕ is bounded, for instance, ϕ = tanh as we shall see.
C.2. T HE CASE OF tanh ϕ
We state a result that is relevant to ϕ = tanh .
Proposition 11.
Assume that ϕ thrice-differentiable with ϕ (0) = 0 , ϕ (cid:48) (0)
= κ , and (cid:13)(cid:13) ϕ
( k ) (cid:13)(cid:13)
∞ ≤ C for k = 0 , ..., 3 , where ϕ ( k ) is the k -th derivative of ϕ .
Assume σ is Lipschitz continuous, nondecreasing.
Then:
(a) γ = 0 and ρ = 0 is a ﬁxed point.
Furthermore, assuming that σ is non-zero on a set of positive Lebesgue measure, ϕ is non-zero almost everywhere and ¯ τ ∈ (0 , ∞ ) , we have if ρ = 0 , it must be that γ = 0 ; in other words, if γ > 0 , then ρ > 0 .
(b) If ϕ is bounded, then 0 ≤ ρ ≤ C , and | γ | ≤ C/ ¯ τ .
(c) Given γ = 0 , if β < 1 /κ 2 , ρ = 0 is a stable ﬁxed point, and if β > 1 /κ 2 , ρ = 0 is unstable.
(d) Consider ¯ τ ∈ (0 , ∞ ) .
Assume that • σ is positive on a set of positive Lebesgue measure, and E { σ (cid:48) (¯ τz ) } > 0 for z ∼ N (0 , 1) , • either
– case 1: σ ( u ) = 0 for u ≤ 0 , and ∆ ρ ( u, t ) < 0 for u, t, ρ > 0 , or – case 2: σ is an odd function, and ∆ ρ ( u, t ) <
∆ ρ ( − u, t ) for u, t, ρ > 0 , in which ∆ ρ ( u, t ) = ϕ (cid:48) (cid:0) βu + √ βρt (cid:1) − ϕ (cid:48) (cid:0) βu − √ βρt (cid:1) , • ϕ satisﬁes E { zϕ ( z ) } → + ∞ for z ∼ N (cid:0) 0 , s 2 (cid:1) and s → ∞ , • ϕ satisﬁes
E { z I ( z ) } > 0 for z is any Gaussian with zero mean and non-zero variance, and I ( u ) =
ϕ ( u ) + uϕ (cid:48) ( u ) .
Then for any given ρ > 0 , there exists β ∗ =
β ∗ ( ρ, ¯ τ ) > 0 ﬁnite such that if β ≤ β ∗ , then γ = 0 is the only ﬁxed point of the equation γ =
G
( γ, ρ ) and is stable; if β > β ∗ , γ =is unstable, and there is one more ﬁxed point at γ > 0 ﬁnite, which is stable.
(e) Consider ¯ τ ∈ (0 , ∞ ) .
The same conclusion as in Claim (c) holds for ρ = 0 with β ∗ = 1 / ( κ E { σ (cid:48)
(¯ τz 1 ) } ) , assuming
• σ is positive on a set of positive Lebesgue measure, and E { σ (cid:48) (¯ τz ) } > 0 for z ∼ N (0 , 1) , • ϕ (cid:48)(cid:48) ( u ) < 0 for u > 0 , and either – case 1: σ ( u ) = 0 for u ≤ 0 , or – case 2: ϕ (cid:48)(cid:48) and σ are odd functions.
The assumption (cid:13)(cid:13) ϕ ( k ) (cid:13)(cid:13) ∞ ≤ C for k = 0 , ..., 3 is not critical, only serves to ensure integrability of various terms in the proof and is likely relaxable, but is made for simplicity.
The following lemma establishes certain properties of the tanh function.
Lemma 12.
Consider ϕ = tanh .
Then:
(a) For ∆ ( u, t ) =
ϕ (cid:48) ( u + t ) − ϕ (cid:48)
( u − t ) , we have ∆ ( u, t ) < 0 for u, t > 0 , and ∆ ( u, t ) <
∆ ( − u, t ) for u, t > 0 .
(b) lim s →∞ E { zϕ ( z ) } =
+ ∞ for z ∼ N (cid:0) 0 , s 2 (cid:1) .
(c) E {
z I ( z ) } > 0 for z ∼ N (cid:0) 0 , s 2 (cid:1) , s (cid:54)
= 0 ,
and I ( u ) =
ϕ ( u ) + uϕ (cid:48) ( u ) .
Now let us consider ϕ
=
σ = tanh .
Note that tanh (cid:48) ( u ) ∈ (0 , 1) for any u (cid:54) = 0 , tanh (cid:48) (0) = 1 and
E (cid:8) tanh (cid:48) (¯ τz 1 ) (cid:9) < 1 unless ¯ τ = 0 .
By Lemma 12, Proposition 11 applies.
The following picture is suggested based on Proposition 11:
• When β < 1 , we have convergence to γ = ρ = 0 .
This is based on Claim (c) and (e).
• The phase transition for ρ locates at β = 1 , above which we have convergence to ρ ∈ (0 , 1) given a non-zero initialization, and below which ρ = 0 .
Here ρ < 1 since tanh is bounded by 1 .
This is based on Claim (a), (b), and (c).
• The phase transition of γ locates at some β > 1 , above which we have convergence to γ > 0 given a non-zero initialization, and below which γ =
For ¯ τ > 0 , γ cannot grow to + ∞ as β varies.
This is based on Claim (a), (b) and (d).
The proposition also suggests that the two phase transitions are close to each other if E { σ (cid:48)
(¯ τz 1 ) }
≈ 1 .
This requires that ¯ τ 2 ≈ 0 , and σ 2 W ≈ 1 in the case α = 1 .
With respect to Eq.
(1), we then have σ 2 b ≈ 0 .
Remarkably this is reminiscent of the context of random feedforward networks with tanh activations, in which (Pennington et al.
(2017)) ﬁnds that an initialization at σ 2 W ≈ 1 and σ 2 b ≈works better than most edge-of-chaos initialization schemes with Gaussian weights.
We also expect from the proposition a similar picture for σ being the ReLU, with a crucial difference.
In this case, E { σ (cid:48) (¯ τz 1 ) }
= 0 .
5 , and therefore one cannot have that the two phase transitions being close to each other.
Interestingly Claim (a) implies that the phase transition of ρ never occurs before that of γ , regardless of the speciﬁc ϕ and σ .
One way for the phase transitions to be close to each other is, as above, taking ϕ
=
σ = tanh and ¯ τ 2 ≈ 0 .
Claim (a), (c) and (e) of the proposition also suggests that if E { σ (cid:48) (¯ τz 1 ) } > ϕ (cid:48) (0) , then γ and ρ will share the exact same location of the phase transitions, below which they are zero and above which they are positive.
C. P ROOFS FOR S ECTION C.
Proof of Proposition 10.
Let θ = βγσ
(¯ τz 1 ) + √ βρz 2 for brevity.
Claim (a).
We have G (0 , ρ ) = 0 and R (0 , 0)
= 0 .
Simple calculations yield R (0 , ρ ) =
βρ/ 2 .
Claim (a) is then immediate.
Claim (b).
To study ρ in the case γ → ∞ , we calculate:
[Equation]
where φ ( u ) = 1 √ 2 π exp (cid:8) − u 2 / 2 (cid:9) and Φ ( u ) = (cid:82) u t =
−∞ φ ( t ) dt the Gaussian PDF and CDF.
Recall that we must have 1 ρ R ( γ, ρ )
= 1 unless ρ = ∞ .
Also notice that φ ( ∞ ) = 0 and Φ (+ ∞ ) = 1 .
Since σ is positive on a set of positive Lebesgue measure, if γ 2 /ρ → ∞ as γ → ∞ ,
1 ρ R ( γ, ρ ) → ∞ , and hence it must be that ρ → ∞ when γ → ∞ .
Claim (c).
We compute the ﬁrst partial derivative ∂ γ G ( γ, ρ ) , using the fact ϕ (cid:48)(cid:48) ( u ) =
δ ( u = 0) (in the distributional sense) for the ReLU:
[Equation]
where in the last step [Equation] we use the fact that z 1 and z 2 are independent.
Hence some algebras yield the second partial derivative:
[Equation]
where in step ( i ) [Equation] we deﬁne
[Equation]
in step ( ii ) , we use Stein’s lemma.
Recall ¯ τ ∈ (0 , ∞ ) .
Since σ ( u ) = 0 for all u ≤ 0 , we have
I ( σ (¯ τz ))
=
for z ≤ 0 .
In addition, I ( u ) > 0 almost everywhere and σ is non-zero on a set of positive Lebesgue measure.
It follows that ∂ 2 γ G ( γ, ρ ) > 0 .
Therefore for any ﬁxed ρ , the mapping γ (cid:55)→ G ( γ, ρ ) is strictly convex.
Notice that G (0 , ρ ) = 0 .
Hence it cannot be that there is a stable ﬁxed point at γ > 0 .
Next we have:
[Equation]
Recall that γ (cid:55)→ G ( γ, ρ ) is strictly convex.
As such, when β E { σ (cid:48) (¯ τz 1 ) } ≤ 1 , there is only one ﬁxed point at γ = 0 , which is stable.
When β E { σ (cid:48) (¯ τz 1 ) } ∈ (1 , 2) , there are two: one at γ = 0 , which is stable, and the other at γ > 0 , which is unstable.
When β E { σ (cid:48) (¯ τz 1 ) } ≥ 2 , there is only one ﬁxed point at γ = 0 , which is unstable.
Claim (d).
In the case that σ is an odd function, u (cid:55)→ I ( u ) is even and hence z
(cid:55)→ I ( σ (¯ τz )) is even.
Then from the calculation of Claim (c), it is easy to see that ∂ 2 γ G ( γ, ρ ) = 0 .
Also,
G (0 , ρ ) = 0 .
Therefore for each ρ , the mapping γ (cid:55)→ G ( γ, ρ ) is a straight line which passes through the point (0 , 0) .
Since ∂ γ G (0 , ρ ) = 12 β E { σ (cid:48) (¯ τz 1 ) } , the claim is then immediate.
Claim (e).
We recall the formula for R ( γ, ρ ) derived in the proof of Claim (b).
Since σ is odd, φ is even and Φ − 0 .
5 is odd, we have
[Equation]
Then if β > 2 , we always have R ( γ, ρ ) > ρ .
Proof of Proposition 11.
Let θ = βγσ
(¯ τz 1 ) + √ βρz 2 for brevity.
Claim (a).
The fact γ = ρ = 0 is a ﬁxed point is obvious by assumption.
Note that in order that
R ( γ, 0) = 0 , one must have E (cid:110) ϕ
( βγσ (¯ τz 1 )) 2 (cid:111)
= 0 .
By assumption, this requires γ = 0 .
Claim (b).
The fact that 0 ≤ ρ ≤ C is obvious.
Note that
[Equation]
which yields the claim.
Claim (c).
We have:
[Equation]
where ( i ) is due to Stein’s lemma.
As such, ∂ ρ R (0 , 0) =
βκ 2 .
The claim is then immediate.
Claim (d).
Consider ρ > 0 .
We have:
[Equation]
and therefore,
[Equation]
where we use Stein’s lemma and the fact z 1 and z 2 are independent.
Notice that, for βρ > 0 , by
Stein’s lemma,
[Equation]
Consider case 1.
Since σ ( u ) = 0 for all u ≤ 0 ,
[Equation]
Since ∆ ρ ( u, t ) < 0 for any u > 0 and t > 0 , and by the assumption that σ is positive on a set of positive Lebesgue measure (which cannot intersect with ( −∞ ; 0] since σ ( u ) = 0 for all u ≤ 0 ), we have ∂ 2 γ G ( γ, ρ ) < 0 for γ > 0 .
In case 2, since σ is an odd function,
[Equation]
Since ∆ ρ ( u, t ) <
∆ ρ ( − u, t ) for u > 0 and t > 0 and σ ( u ) ≥ 0 for u ≥ 0 (for σ being odd and non-decreasing), we again have ∂ 2 γ G ( γ, ρ ) < 0 for γ > 0 .
As such, the mapping γ (cid:55)→ G ( γ, ρ ) , for ρ > 0 , is strictly concave on (0 , ∞ ) .
Next we have
[Equation]
since z 1 and z 2 are independent.
Notice that by Stein’s lemma,
[Equation]
By assumption, we thus have that β E (cid:8) ϕ (cid:48) (cid:0) √ βρz 1 (cid:1)(cid:9) and hence ∂ γ G (0 , ρ ) are increasing in β .
Furthermore, ∂ γ G (0 , ρ ) → 0 when β → 0 , and
[Equation]
where ( i ) is by Stein’s lemma, and ( ii ) is by assumption.
Therefore, there must exist β ∗ =
β ∗ ( ρ, ¯ τ ) > 0 ﬁnite such that ∂ γ G (0 , ρ )
= 1 at β = β ∗ , ∂ γ G (0 , ρ ) < 1 for β < β ∗ and ∂ γ G (0 , ρ ) > 1 for β > β ∗ .
Since γ (cid:55)→ G ( γ, ρ ) is strictly concave on (0 , ∞ ) and γ = 0 is a ﬁxed point by Claim (a), β ∗ is the threshold as in the claim.
Claim (e).
Consider ρ = 0 , in which case:
[Equation]
Consider case 1.
Since σ ( u ) = 0 for all u ≤ 0 ,
[Equation]
Since ϕ (cid:48)(cid:48) ( u ) < 0 for u > 0 and σ is positive on a set of positive Lebesgue measure (which cannot intersect ( −∞ ; 0] since σ ( u ) = 0 for all u ≤ 0 ), it is easy to see that ∂ 2 γ G ( γ, 0) < 0 for γ > 0 .
In case 2, since ϕ (cid:48)(cid:48) and σ are odd, the mapping z (cid:55)→ ϕ (cid:48)(cid:48) ( βγσ (¯ τz )) is odd, and hence
[Equation]
Again we have ∂ 2 γ G ( γ, 0) < 0 for γ > 0 .
As such, γ (cid:55)→ G ( γ, ρ ) is strictly concave on (0 , ∞ ) .
Notice that ∂ γ G (0 , 0) =
βκ E { σ (cid:48) (¯ τz 1 ) } , linearly increasing in β .
This proves the claim with β ∗ = 1 / ( κ E { σ (cid:48)
(¯ τz 1 ) } ) .
Proof of Lemma 12.
We prove the ﬁrst claim:
[Equation]
Since tanh 2 is even and increasing on (0 , + ∞ ) , ∆ ρ ( u, t ) < 0 for u, t > 0 and ∆ ρ ( u, t ) > 0 for u > 0 and t < 0 .
Simple algebra yields
[Equation]
( u,
[Equation]
( u
[Equation]
Hence ∆ ρ ( u, t ) <
∆ ρ ( − u, t ) for u > 0 and t > 0 .
To see the second claim, for z ∼ N (0 , 1) ,
[Equation]
To see the third claim, since I is odd,
[Equation]
D V ISUALIZATION FOR S ECTION 3. D. V ISUALIZATION OF THE RECONSTRUCTIONS
In Fig.
10, 11, 12 and 13, we show the reconstructions of several images by the trained networks after 5 × 10 5 training iterations, under the schemes from Table 1, in the experiments of Section 3.5.
We draw 10 digit images from the MNIST test set, as well as 3 letter images from the EMNIST
Letters test set (Cohen et al.
Note that the networks are not trained with any letter images from the EMNIST data set.
The reconstruction quality is visually imperfect even after intensive training, which is entirely expected for vanilla autoencoders and regression problems.
Observe that for the schemes that yield meaningful reconstructions, they output recognizable digits for digit images, while for letter images, most of their reconstructions are hardly recognizable as letters.
As such, the trained networks of these schemes do not simply approximate the identity function, but rather capture some low-dimensional structures of the data.
An exception is Schemeunder Setting 2, which is not surprising since it is a purely tanh network and tanh is almost identity near zero.
D. V ISUALIZATION OF THE EVOLUTION
We show the evolution of the test reconstruction loss on the plane (cid:0) σ 2 W , σ 2 b (cid:1) , in conjunction with the experiments of Section 3.5.
To make the computation more manageable, we opt for L = 50 with less iterations, while maintaining other parameters the same as in Section 3.5.
The results are shown in Fig.
14 and 15.
Several patterns emerge in good agreement with our hypotheses.
Firstly, for ϕ = ReLU , the evolution starts earliest near σ 2 W = 2 , shows almost no progress or is numerically unstable when σ 2 W (cid:29) 2 , and is much slower when σ 2 W (cid:28) 2 .
Secondly, for ϕ = tanh , the evolution is much slower when σ 2 W (cid:28) 1 .
Intriguingly the evolution is almost insensitive to σ 2 b .
E M ISCELLANIES E. E DGE OF CHAOS INITIALIZATION
We quickly review the edge of chaos (EOC) initialization (Schoenholz et al.
For an activation σ , any (cid:0) σ 2 W , σ 2 b (cid:1) such that there exists a ﬁnite ¯ τ ≥ 0 for which
[Equation]
is said to be an EOC initialization scheme, where z ∼ N (0 , 1) .
This is based on the order-to-chaos phenomenon found in (Poole et al.
For ReLU σ , σ 2 W = 2 and σ 2 b = 0 is the only EOC initialization (Hayou et al.
and coincides with the He initialization (He et al.
For σ = tanh , there can be multiple pairs that form EOC initialization.
(Pennington et al.
(2017)) argues that a better EOC scheme should be closer to dynamical isometry and suggests taking σ 2 W ≈ 1 .and
σ 2 b ≈ 2 . 01 × 10 − 5 for σ = tanh .
The EOC initialization is applicable to feedforward networks.
E. U SEFUL FACTS
We state several useful facts, which are used in various places throughout.
First, for a Gaussian matrix W and an independent vector u , we have W u is also Gaussian.
In particular, if entries of
W are i.i.d.
N (cid:0) 0 , s 2 (cid:1) , then W u ∼ N (cid:16) 0 , s 2 (cid:107) u (cid:107)
I (cid:17) .
The second fact is Stein’s lemma:
Lemma 13 (Stein’s) .
E { zf ( z ) } =
E { f (cid:48) ( z ) } for z ∼ N (0 , 1) and f : R (cid:55)→ R weakly differentiable whenever the expectations are deﬁned.