On Scrambling Phenomena for Randomly Initialized Recurrent Networks
Abstract
Recurrent Neural Networks (RNNs) frequently exhibit complicated dynamics, and their sensitivity to the initialization process often renders them notoriously hard to train.
Recent works have shed light on such phenomena analyzing when exploding or vanishing gradients may occur, either of which is detrimental for training dynamics.
In this paper, we point to a formal connection between RNNs and chaotic dynamical systems and prove a qualitatively stronger phenomenon about RNNs than what exploding gradients seem to suggest.
Our main result proves that under standard initialization (e.g., He, Xavier etc.), RNNs will exhibit Li-Yorke chaos with constant probability independent of the network’s width.
This explains the experimentally observed phenomenon of scrambling , under which trajectories of nearby points may appear to be arbitrarily close during some timesteps, yet will be far away in future timesteps.
In stark contrast to their feedforward counterparts, we show that chaotic behavior in RNNs is preserved under small perturbations and that their expressive power remains exponential in the number of feedback iterations.
Our technical arguments rely on viewing RNNs as random walks under non-linear activations, and studying the existence of certain types of higher-order ﬁxed points called periodic points that lead to phase transitions from order to chaos.
Introduction
In standard feedforward neural networks (FNNs), computation is performed “from left to right” propagating the input through the hidden units to the output.
In contrast, recurrent neural networks (RNNs) form a feedback loop, transfering information from their output back to their input (e g.,
LSTMs (Hochreiter and Schmidhuber, 1997), GRUs (Cho et al., 2014)).
This feedback loop allows
RNNs to share parameters across time because the weights and biases of each iteration are identical.
As a result, RNNs can capture long range temporal correlations in the input.
For these reasons, they have been very successful in applications in sequence learning domains, such as speech recognition, natural language processing, video understanding, and time-series prediction (Bahdanau et al., 2014;
Cho et al., 2014; Chung et al., 2014).
Unfortunately, their unique ability to share parameters across time comes at a cost: RNNs are sensitive to their initialization processes, which makes them extremely difﬁcult to train and causes complicated evaluation and training dynamics (Le et al., 2015; Laurent and von Brecht, 2016; Miller and Hardt, 2018).
Roughly speaking, because the hidden units of an RNN are applied to the input over and over again, the ﬁnal output can quickly explode or vanish, depending on whether its Jacobian’s spectral norm is greater or smaller than one respectively.
Similar issues also arise during backpropagation that hinder the learning process (Allen-Zhu et al., 2018).
Besides the hurdles with their implementation, recurrent architectures pose signiﬁcant theoretical challenges.
Several basic questions include how to properly initialize RNNs, what is their expressivity power (also known as representation capabilities), and why do they converge or diverge, all of which require further investigations.
In this paper, we take a closer look at randomly initialized RNNs:
Can we get a better understanding of the behavior of RNNs at initialization using dynamical systems?
We draw on the extensive dynamical systems literature—which has long asked similar questions about the topological behavior of iterated compositions of functions—to study the properties of RNNs with standard random initializations.
We prove that under common initialization strategies, e.g., He or Xavier (He et al., 2015, 2016), RNNs can produce dynamics that are characterized by chaos, even in innocuous settings and even in the absence of external input.
Most importantly, chaos arises with constant probability which is independent of the network’s width.
Our theoretical ﬁndings explain empirically observed behavior of RNNs from prior works, and are also validated in our experiments.
More broadly, our work builds on recent works that aim at understanding neural networks through the lens of dynamical systems; for example, Chatziafratis et al.
(2020b,a) use Sharkovsky’s theorem from discrete dynamical systems to provide depth-width tradeoffs for the representation capabilities of neural networks, and Sanford and Chatziafratis (2022) further give more ﬁne-grained lower bounds based on the notion of “chaotic itineraries”.
1.1 Two Motivating Behaviors of RNNs
Before stating our main result, we illustrate two concrete behaviors of RNNs that inspired our work.
The ﬁrst example demonstrates that randomly initialized RNNs can lead to what is perhaps most commonly perceived as “chaos”, while the second example demonstrates a qualitatively different behavior of RNNs compared to FNNs.
Our main result uniﬁes the conclusions drawn from these two.
Scrambling Trajectories at Initialization Prior works have empirically demonstrated that RNNs can behave chaotically when their weights and biases are chosen according to a certain scheme.
For example, Laurent and von Brecht (2016) consider a simple 4-dimensional RNN with speciﬁc parameters in the absence of input data.
They plot the trajectories of two nearby points x, y with k x  y k  10  7 as they are propagated through many iterations of the RNN.
They observe that the long-term behavior (e.g., after 200 iterations) of the trajectories is highly sensitive to the initial states, because distances may become small, then large again and so on.
We ask:
Are RNNs (provably) chaotic even under standard heuristics for random initialization?
We answer this question in the afﬁrmative both experimentally and theoretically.
This question is helpful to understand for multiple reasons.
First, it informs us about the behavior of most RNNs, as we start with a random setting of the parameters.
Second, proving that a system is chaotic is qualitatively much stronger than simply knowing its gradient to be exploding; this will become evident below, where we describe the phenomenon of scrambling from dynamical systems.
Finally, understanding why and how often an RNN is chaotic can lead to even better methods for initialization.
To begin, we empirically verify the above statement by examining randomly initialized RNNs.
Figure 1 demonstrates that trajectories of different points may be close together during some timesteps and far apart during future timesteps, or vice versa.
This phenomenon, which will be rigorously established in later sections, is called scrambling (Li and Yorke, 1975) and emerges as a direct consequence of the existence of higher order ﬁxed points (called periodic points) of the continuous map deﬁned by the random RNN.
Persistence of Chaos in RNNs vs
FNNs Our second example is related to the expressive power of neural networks.
A standard measure used to capture their representation capabilities is the number of linear regions formed by the non-linear activations (Montufar et al., 2014); the higher this number, the more expressive the network is.
Counting the maximum possible number of linear regions across different architectures has also been leveraged in order to obtain depth vs width tradeoffs in many
Here, we are inspired by a remarkable observation of Hanin and Rolnick (2019a), who showed that real-valued FNNs from R !
R may lose their expressive power if their weights are randomly perturbed or those weights are initialized according to standard methods.
Roughly speaking, they show that in such networks the number of linear regions grows only linearly in the total number of neurons.
These results contrast with the aforementioned analyses of the theoretical maximum number of regions, which is exponential in depth.
We ask the analogous question for RNNs instead of FNNs:
Will randomly initialized or perturbed RNNs lose their high expressivity like FNNs did?
We show a contrast between RNNs and FNNs, by showing that the number of linear regions in random
RNNs remains exponential on the number of its feedback iterations.
Once again, the fundamental difference lies on the fact that RNNs share parameters across time.
Indeed, the analyses of many prior works (Hanin and Rolnick, 2019a,b; Hanin et al., 2021) crucially relies on the “fresh” randomness injected at every layer, e.g., that the weights/biases of each neuron are initialized independently of each other; obviously, this is no longer true in RNNs where the same units are repeatedly used across different iterations.
This shared randomness raises new technical challenges for bounding the number of linear regions, but we manage to indirectly bound them by studying ﬁxed points of random RNNs.
Our Main Results
Our main contribution is to prove that randomly initialized RNNs can exhibit Li-Yorke chaos (Li and
Yorke, 1975) (see deﬁnitions below), and to quantify when and how often this type of chaos appears as we vary the variance of the weights chosen by the random initialization.
To do so, we use discrete dynamical systems which naturally capture the behavior of RNNs: simply start with some shallow
NN implementing a continuous map f , and after t feedback iterations of the RNN, its output will be exactly f t ( f composed with itself t times).
We begin with some basic deﬁnitions:
Deﬁnition 2.1.
(Scrambled Set)
Let ( X, d ) be a compact metric space and let f : X !
X be a continuous map.
Two points x, y 2 X are called proximal if:
[Equation]
and are called asymptotic if
[Equation]
A set Y ✓ X is called scrambled if 8 x, y 2 Y, x
6 =
y , the points x, y are proximal, but not asymptotic.
Deﬁnition 2.2.
(Li-Yorke Chaos)
The dynamical system ( X, f ) is Li-Yorke chaotic if there is an uncountable scrambled set Y ✓ X .
Li-Yorke chaos leads both to scrambling phenomena (Figure 1) and to high expressivity (Right of
Figure 2).
Interestingly, this is not true for FNNs (Left of Figure 2), where the fresh randomness at each neuron leads to concentration of the Jacobian’s norm around 1, which is sufﬁcient to avoid chaos.
In other words, these deﬁnitions capture exactly the fact that trajectories get arbitrarily close, but also move apart inﬁnitely often (as in Fig. 1).
Intuitively, when scrambling occurs in RNNs, their # linear regions will be large and their input-output Jacobian will have spectral norm larger than one.
Deﬁnition 2.3 (Simple RNN Model) .
For k,  > 0 , let RNN ( k,  2 ) be a family of recurrent neural networks with the following properties
: • Input: The input to the network is 1-dimensional,
i
e., a single number x 2 [0 , 1] .
• Hidden Layer: There is only one hidden layer of width at least k , with ReLU( x ) = max( x, 0) activations neurons, each of which has its own bias terms b i ⇠ Unif ([0 , 1]) .
• Output: The output is a real number in [0 , 1] which takes the functional form f k ( x ) =
clip( P ki =1 a i ReLU( x  b i )) , where the weights are i.i.d.
Gaussians a i ⇠ N (0 ,  2 k ) , and clip( · ) ensures that f k ( x ) 2 [0 , 1] similarly to the input (i.e., it “clips” the input so that it remains in [0 , 1] as it is an RNN).
• Feedback Loop: f k ( x ) becomes the new input number, so after t iterations the output is ( f k  f k  . . .
f k )( x ) , i.e., t compositions of f k with itself.
As we show, even this innocuous class of RNNs (see Sec. 3 for general model) leads to scrambling.
Theorem 2.4 (Li-Yorke Chaos at Initialization) .
Consider f k 2 RNN ( k  ,  2 ) initialized according to the He normal initialization (set  2 = 2 , so weight variance is 2 /k ).
Then, there exists some constant  > 0 (independent of the width) and width k
He > 1 , such that for sufﬁciently large k > k
He , f k is Li-Yorke chaotic with probability at least  .
This answers our two questions posed earlier: RNNs may remain chaotic under initialization heuristics, and maintain their high expressivity, because Li-Yorke chaos implies an exponential # linear regions (Theorem 1.5 in Chatziafratis et al.
(2020b)).
Next, we focus on threshold phenomena:
Theorem 2.5.
[Order-to-Chaos Transition]
For f k 2 RNN ( k,  2 ) , we get the following 3 regimes as we vary the variance of the weights a i ⇠ N (0 ,  2 k ) :
• (Low variance, order) Let a i ⇠ N (0 , 1 4 k log k ) .
Then, the probability that f k is Li-Yorke chaotic is at most 1 k .
• (Edge of chaos) Let a i ⇠ N (0 , 2 k ) (i.e., He initialization).
Then, f k is Li-Yorke chaotic with constant (albeit small) probability.
• (High-variance, chaos) Let a i ⇠ N (0 ,  2 k ) for  = !
Then, lim k !1 Pr [ f k is Li-Yorke chaotic ] = 1 .
To the best of our knowledge, we are the ﬁrst to study Li-Yorke chaos in the context of RNNs and to prove that scrambling phenomena appear with constant probability independent of the width.
We emphasize that Li-Yorke chaos is stronger than exploding gradients, as any chaotic system requires the input-output map to be non-contracting (as otherwise it would converge to a ﬁxed point).
Techniques Contrary to many prior works (Laurent and von Brecht, 2016; Miller and Hardt, 2018), we do not require differentiability of activations like tanh( · ) , as we rely on properties of continuous functions.
Most importantly, we deviate from past works (Bertschinger and Natschl¨ager, 2004; Poole et al., 2016), as we do not use tools from mean-ﬁeld analysis (where the number of neurons, width or depth goes to inﬁnity), since here we are interested on the dependence on the width k .
Instead, we study the ﬁxed points and higher-order ﬁxed points of f k 2 RNN ( k,  2 ) (see Sec. 3), bounding the probability that f k has a certain type of ﬁxed point known to lead to chaos.
This is done by viewing RNNs as random walks under non-linear activations.
To give some intuition behind our ﬁndings, consider a random walk starting from 0 in which we consider increments
d i ⇠ N (0 , 1) .
It is quite well-known in the literature of random walks, that the expected number of steps needed for a random walk starting from 0 with increments d i ⇠ N (0 , 1) to reach n and come back to 0 is ⇥ ( n 2 ) iterates.
For an RNN to be chaotic, as it turns out, we need to ﬁnd the probability that it has certain type of ﬁxed points; this can be bounded below by the probability that the random walk reaches point 1 and then goes back to 0 .
As a result, if we rescale appropriately and the increments d i ⇠ N (0 , 1 k ) , then we need ⇥ ( k ) iterates.
Roughly speaking, k plays the role of the size of the width ( d i are the weights).
In reality, the random walk we analyze is more complicated as the (random) biases affect the length of each increment in the random walk.
Experiments Finally, we validate our theory by extensive simulations on a variety of randomly initialized RNN architectures.
Our goal is twofold: Firstly, to demonstrate that Li-Yorke chaos is indeed present across different models and different initialization heuristics, and not just an artifact of the speciﬁcities of our RNN family.
Secondly, to estimate the value of the aforementioned constant probability of chaos   in the Edge of Chaos regime, which is known to be the most interesting (Bertschinger and Natschl¨ager, 2004; Yang and Schoenholz, 2017; Hanin, 2018), yielding a comparison for which initializations are more likely to be Li-Yorke chaotic (see table in Fig. 3).
2.1 Further Related Works
Several works have studied RNNs in the context of chaotic dynamical systems and edge of chaos initialization (Sompolinsky et al., 1988; Bertschinger and Natschl¨ager, 2004; Saxe et al., 2013; Kadmon and Sompolinsky, 2015;
Poole et al., 2016).
They study neural networks in the limit of inﬁnitely large widths or depth and derive properties of the dynamics, typically using mean-ﬁeld analysis.
Moreover, the notion of chaos used there is not from Li and Yorke (1975), as they do not analyze periodic points in the trajectories of the networks, nor do they establish scrambling phenomena.
For example, Bertschinger and Natschl¨ager (2004) observe that the sequence of activations in the neurons of the network exhibit increasingly unpredictable and complicated patterns as the variance increases and use this notion to determine the edge of chaos and threshold phenomena.
Preliminaries
Higher-order Fixed Points Let f : [ a, b ] !
[ a, b ] be a continuous map.
The notion of a periodic point is a generalization of a ﬁxed point:
Deﬁnition 3.1.
We say f contains period n or has a point of period n  1 , if there exists a point x 0 2 [ a, b ] such that:
[Equation]
where it is common to use f n ( x 0 ) to denote the composition of f with itself n times, evaluated at point x 0 .
In particular, C := { x 0 , f ( x 0 ) , f ( f ( x 0 )) , . . .
, f n  1 ( x 0 ) } has n distinct elements (each of which is a point of period n ) and is called a cycle (or orbit) with period n .
Period 3 implies Chaos Of particular interest, is the case of functions f that contain period 3 .
In 1975, a seminal paper by Li and Yorke (1975), introduced the term “chaos” as used in mathematics nowadays, and proved connections with periodic functions as deﬁned above.
Fact 1.
Let f : R !
R be continuous.
If f has a period 3 point then two properties hold: First, f contains points of any period n 2 N .
Second, f will exhibit scrambling as deﬁned in Def.
Sometimes we say that f is “chaotic in the sense of Li and Yorke”.
On a historical note, this turned out to be a special case of an older result by Sharkovsky (1964).
In other words, a sufﬁcient condition to obtain scrambling phenomena is to show that f contains a point of period 3.
In general, the uncountable set of chaotic points may, however, be of measure zero, in which case the map is said to have unobservable chaos or unobservable nonperiodicity (Collet and
Eckmann, 2009).
Our Theorem 2.4 informs us that this is not the case for randomly initialized RNNs.
Other RNN Models Recurrent models have been thoroughly studied because of their success in sequence learning.
Following notation from Laurent and von Brecht (2016), here is a general model for RNNs:
[Equation]
where the index t 2 N represents the current time, u t 2 R d represents the current state of the system, x t the current external input, and W i ’s are weight matrices.
Regarding training or initialization of weights many tricks have been proposed, e.g., the popular “identity matrix” trick of Le et al.
(2015) to preserve lengths.
Laurent and von Brecht (2016) to simplify the above, they study what is referred to as the dynamical system induced by the RNN, i.e, they consider trajectories of u t in the absence of external inputs:
[Equation]
This is much more tractable and gives us insights into the architecture of the RNN as it decouples the inﬂuence of the external data x t ’s (with which any possible response can be produced) from the model itself.
For more on other simpliﬁed RNN variants with speciﬁc statistical assumptions on weights/input, and connections to edge of chaos, see Bertschinger and Natschl¨ager (2004).
Similarly, our RNN family has no external input; we show that even in 1-dimensional settings, chaos emerges.
Persistence of Chaos and Phase Transitions
In this section we prove our main results.
As we alluded to in the previous section, in order to show chaos in the sense of Li and Yorke, we will bound the probability that the family RNN ( k,  2 ) contains a point of period 3, since this will serve as a “witness of chaos”.
We start with some simple observations.
4.1 Triangular Wave with Period
Perhaps the simplest example of a continuous function f that has period 3 is the following triangle wave function: f ( x ) = 2 x, for x 2 [0 , 1 / 2] , and 2(1  x ) , for x 2 (1 / 2 , 1] .
Indeed, one can check that the set { 29 , 49 , 89 } is a periodic orbit of length 3, since f ( 29 ) = 49 , f ( 49 )
= 89 and ﬁnally, f ( 89 )
=closing the loop.
This simple function has played an interesting role in obtaining depth vs width tradeoffs in recent years Telgarsky (2015, 2016); Chatziafratis et al.
(2020b).
4.2 Persistence of Chaos
Now consider a function from RNN : f k ( x ) =
clip(
P ki =1 a i ReLU( x  b i )) for independent
a i ⇠ N
(0 ,  2 k ) and b i ⇠ Unif ([0 , 1])
Theorem 4.1.
Fix any  2 (0 , 1) .
There exists  such that for any sufﬁciently large k (whose minimum value depends on  ), f k is 3-periodic with probability at least 1   .
For the theorem we need some intermediate lemmas.
We begin with a sufﬁcient condition for chaos:
Lemma 4.2.
Let y i =
P i  1 ◆ =
a ◆ ( b i  b ◆ ) so that f k ( b i ) =
clip( y i ) .
If there exists indices i < ` such that y i > 1 and y ` < 0 , then f k has period 3.
Proof.
We can assume without loss of generality that b 1 < b 2 < · · · < b k .
Suppose there are indices i < ` as speciﬁed in the statement.
Since y i > 1 , after clipping we know that f k ( x i ) = 1 , and since y ` < 0 , after clipping we know that f k ( x ` ) = 0 .
Let’s check how intervals are mapped under f : since x i  x ` 2 [0 , 1] and because of continuity of f k , we have that f k ([0 , x i ]) =
[0 , 1] , and similarly that f k ([ x i , x ` ]) =
By combining this with the intermediate value theorem for f 3 k (3 iterations of f k applied to itself), we can exhibit a 3-cycle with two elements in (0 , x i ) and a third in ( x i , x ` ) .
Proof Sketch of Theorem 4.1.
Roughly, the proof operates by choosing a subset of t ordered elements { y i j } j 2 [ t ] ⇢ { y i } i 2 [ k ] that meets several key properties such that there exists j < j 0 such that y j >and y j 0 < 0 .
Using this and Lemma 4.2 (for i = j and ` = j 0 ), we will conclude that f k has period 3.
However, there are three immediate challenges that make such a result challenging to obtain:
(Randomness)
If the variance parameter  is small, then there will be a substantial constant probability of an y j lying in the interval [0 , 1] and thus meeting neither characteristic.
(Correlations)
Due to the function’s expression as a linear combination of ReLU neurons with small coefﬁcients, the function f k is smooth, and y i ’s in the same vicinity are highly correlated.
This makes it more difﬁcult to ensure that sign changes to the y j ’s are likely.
(Biases)
The randomness of the biases b i makes it more difﬁcult to assess the correlations of the y
i ’s, since having small gaps b `  b i will yield stronger correlations between the two consecutive values of the function.
We mitigate these issues by carefully selecting parameters t and  and the subset { y i j } j 2 [ t ] .
Problems from (1) can be avoided by choosing  to be large enough to ensure that for sufﬁciently large i relative to k , it’s highly unlikely for any y i j to lie in the interval [0 , 1] ; if none of these lie in that interval, then it is sufﬁcient to ﬁnd y i j
and y i j 0 that exhibit a positive-to-negative sign change.
To minimize correlation for (2) between nearby y i
j ’s, we choose those i j ’s to ensure that each is a large multiplicative factor times its predecessor; this relies on key intuition from random walks, where increasing temporal differences between random variables makes them increasingly independent.
We avoid the atypical samples of b i of (3) by gradually deﬁning a set of “good” b i ’s that exist with high probability and completing the proof while assuming the b i ’s are good.
Proof of Theorem 4.1.
A primary goal of our analysis is to bound the variances and covariances of y
i ’s to aid in our selection of a subset of y i
j ’s that are nearly independent.
We ﬁrst assume that the biases b i are held ﬁxed, while the weights a i are chosen at random; we will later prove bounds that hold for all typical biases.
We let E a [ · ] denote such an expectation over random a for ﬁxed b .
For indices i < ` , we have that E a [ y i ] =
and also the following:
[Equation]
i
X ◆ =
( b i  b ◆ )( b `  b ◆ ) .
For ﬁxed b i ’s, each y i =
P i  1 ◆ =
a ◆ ( b i  b ◆ ) is drawn from a Gaussian distribution with mean 0 and variance  2 k
P i  1 ◆  1 ( b i  b ◆ ) 2 .
We bound these quantities with high probability over random choices of biases b i ; for the remainder of the argument, we assume that the low-probability “bad case” (i.e., having small gaps b `  b i ) does not occur and apply those bounds without worry about the effects of the bias terms.
By deﬁning b i as the i th smallest element from sample of size k from Unif ([0 , 1]) , the distribution of each is a Beta distribution: b i ⇠ Beta ( i, k + 1  i ) (see Bertsekas and Tsitsiklis (2008)).
We recall the low-order moments of these biases in order to bound the moments of the y i ’s.
Assume ◆ < i .
E [ b i ] = i k + 1 , E ⇥ b 2 i
⇤
=
i ( i + 1) ( k + 1)( k + 2) and Var [ b i ] = i ( k + 1  i ) ( k + 1) 2 ( k + 2)
We establish a simple lower bound on the summation used to express E a ⇥ y
i ⇤ :
E a ⇥
y
i ⇤  i  2 2 k
( b i  b i/ 2 ) 2 .
We apply Chebyshev’s inequality to lower-bound b i and upper-bound b i/ 2 .
To do so, note that for sufﬁciently large k , Var [ b i ] 
[Equation]
[Equation]
k +
[Equation]
12 k .
Assume i  k 3 / 4 .
Then, by Chebyshev’s bound,
Pr
[Equation]
By the same calculation, Pr ⇥ b i/ 2
[Equation]
i 8 k ⇤  32 p k . Then, Pr h
[Equation]
1  64 p k .
Later, this will be important for showing that y j 62 [0 , 1] with high probability with  large enough.
We also upper-bound the covariances.
Note that E a [ y i y ` ]  i  2 k b i b ` .
For i  k 3 / 4 , the above calculations ensure that E a [ y i y ` ]  3 i 2 ` 2 2 k 3 with probability at least 1  64 p k .
Now, we choose the elements
y i
j that guarantee the needed properties.
Let i j :=
k r t  j (assuming that k is divisible by 2 r t ) for t := 6  and r = (12 t ) 8 .
We assume that k  ( 72  ) 192 /  = r 4 t , which ensures that every i j  k 3 / 4 , since
[Equation]
These choices ensure that the variance of each element is high enough to make belonging to [0 , 1] unlikely for any y i,j , while spreading apart successive iterates
y i,j and y i,j +1 to make them nearly independent.
To do so, we analyze the normalized variables x j :=
[Equation]
ij q
[Equation]
[Equation]
] , for j 2 [ t ] .
Note that, given ﬁxed b , x j ⇠ N (0 , 1) and with probability 1  128 t 2 p k , for every j < j 0 ,
E a [ x j x j 0 ] =
[Equation] i r Var [Equation]
Var [Equation] ji  [Equation]
[Equation] 0 )  {+
 [Equation] [Equation]+}
Hence, for a sufﬁciently large r , the x j ’s are essentially uncorrelated.
Note that x = ( x 1 , . . .
, x t )
R t (if we assume a ﬁxed b that satisﬁes the above high probability events) has x ⇠ N (0 , ⌃ ) , where ⌃ j,j = 1 and ⌃ j,j 0  1 t 4 .
The smallest eigenvalue of ⌃ can be bounded.
(Similarly for µ max ( ⌃ ) , see
Lemma B.1.)
Given the lack of correlation between the t random variables for sufﬁciently large r , we bound the probability that the signs of the elements of x follow a certain pattern.
We basically need to show that x has a + sign that precedes a  sign (like triangle waves have +2 slope followed by  2 slope).
Using this, we can estimate the probability of period 3 emerging
and this completes the proof.
(Due to space limitations, some last steps are in Appendix B.)
Corollary 4.3.
For weights drawn as a i ⇠ N (0 , !
( 1 k )) , the lim k !
1 Pr [ f k is 3-periodic ] = 1 .
Combining the fact that f k will have period 3 with some constant probability  , with the work of Chatziafratis et al.
(2020b)
on the expressivity of networks as measured by # linear regions we get:
Corollary 4.4.
With the same probability  , the number of linear regions of f k 2 RNN is exponential in the number t of feedback iterations, with a growth rate of  = 1+ p 5 2 .
Theorem 4.5.
Consider f k initialized according to the He initialization (i.e. f k from Thm.
4.1 with  = 2 ).
For sufﬁciently large k , f k is 3-periodic w.p.
at least some constant.
(Proof in App.
B.2.)
Theorem 4.6.
Consider f k initialized with a i ⇠ N (0 , 1 4 k log k ) .
Then, the probability that f k has no ﬁxed points (therefore exhibits no chaotic behavior) is  1  1 k .
(Proof in App.
B.3.)
Experiments
Our goal is to evaluate how robust our ﬁndings are to speciﬁcations of the RNN model, and to empirically estimate some bounds on how often scrambling phenomena appear 2 .
Robustness across Different Models We try different architectures and initialization techniques from the literature (He et al., 2015; Glorot and Bengio, 2010), truncated Gaussians, and others, for setting the weights and the biases.
Our table in Fig.
3 summarizes the results across 10000 runs of each experiment.
For each experiment, the ﬁrst layer has width 2, where the weights and biases of each neuron are initialized according to each line, whereas the second layer has width 1 with weight and bias as speciﬁed in each line.
Notice that the last column contains the probability that the recurrent model is chaotic, having period 3.
The way we classify whether the output of an RNN is chaotic or not is done by plotting one iteration of the RNN and three iterations of the RNN and then checking whether there are more ﬁxed points in the latter, implying period 3 (for a pictorial representation see Figure 7 in Appendix C).
The purpose of this experiment was to quantify the probability for having period 3 in one-dimensional randomly initialized RNNs, as in higher dimensions there is not a clean mathematical statement for checking periods 3.
This validates our theoretical ﬁndings that chaos (in the sense of Li-Yorke) persists, in contrast to FNNs (see Figure 2).
RNNs transitions in Higher Dimensions In higher dimensions, we used an MNIST dataset as input to a 64-dimensional RNN with 1 hidden layer, width 64, fully connected, with ReLU activation functions and He initialization.
Note that we did not train the RNN as this was not our purpose.
We simply initialized it and observed (Fig. 4) how the values of the output neurons ﬂuctuate with respect to t (the number of compositions of the RNN with itself) for various  .
Since high dimensions do not have clean characterizations of chaos such as period 3, we instead detect chaos by relying on a necessary condition: Jacobian’s spectral norm being more than 1 (See Fig. 5).
Additional Experiments and Remarks.
Regarding the experiments reported in Figure 3, the reported numbers were without the Clipping operation, so that the architecture is more faithful to practice.
We believe it is an interesting direction to further understand the chaotic properties of neural networks at initialization and exactly quantify both experimentally and theoretically the probability of chaos.
Of course, for the simpler one-dimensional case, we were able to use the period 3 property which is a simple to state and simple to verify condition.
For higher dimensional RNNs, that are closer to those being used in practice nowadays, it still remains a challenging problem to characterize when chaos emerges.
Other notions of chaos like Devanay’s chaos (Huang and Ye, 2002) could also be useful when exploring chaotic properties of neural nets (Devaney’s chaos is a stronger notion of chaos compared to Li–Yorke’s chaos).
For completeness, we also performed some additional experiments with the Clipping operation and the numbers are still ranging from 2% - 9% for the probability of chaos.
Speciﬁcally, following the setup of Fig. 3, out of 10000 runs, we got (with Clipping) that the chaos frequency reported in the last column is 3 . 6 , 5 . 8 , 9 . 7 , 5 . 7 , 2 . 2 respectively for the different initializations.
We also performed experiments with other types of activations units and higher values for the width and the depth and we still observe that these variants too exhibit period 3.
For example, with tanh activations, the last column would be 9 .
7 , 11 , 17 . 6 , 2 . 6 , 4 . 6 respectively.
If we instead change the width to be equal to 4, the numbers become 13 . 1 , 8 . 1 , 14 . 09 , 0 . 8 , 2 and in another setting where the depth equals 3 the numbers become 6 .
Conclusion
Using tools from discrete dynamical systems, we show simple models for randomly initialized recurrent neural networks exhibiting scrambling phenomena and we study the associated threshold phenomena for the emergence of Li-Yorke chaos.
Our theory and experiments explain observed RNN behaviors, highlighting the difference between FNNs and RNNs in a simple setting.
More broadly, we believe this connection can be fruitful to obtain a better understanding of RNNs at initialization as it adds to a recent thread of works that give interesting new perspectives on neural networks, borrowing ideas from dynamical systems.
Acknowledgments
The authors would like to thank the anonymous reviewers for their time and their useful feedback that improved the presentation of our paper.
VC would like to acknowledge support from a start-up grant at UC Santa Cruz.
IP would like to acknowledge support from a start-up grant at UC Irvine.
Part of this work was done while the authors VC, IP, SS were visiting UC Berkeley and the Simons
Institute for the Theory of Computing, while VC was a FODSI fellow at MIT and Northeastern.
CS was supported by NSF GRFP and NSF grants CCF-1814873 and IIS-1838154.