T HE E XPRESSIVE P OWER OF G ATED R ECURRENT U NITS AS A C ONTINUOUS D YNAMICAL S YSTEM
Anonymous authors
Paper under double-blind review
A BSTRACT
Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture.
Despite their incredible success in tasks such as natural and artiﬁcial language processing, speech, video, and polyphonic music, very little is understood about the speciﬁc dynamic features representable in a GRU network.
As a result, it is difﬁcult to know a priori how successful a GRURNN will perform on a given data set.
In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system and classify the dynamical features obtainable with such system.
We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies and homoclinic orbits.
In addition, we show that any ﬁnite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor and is limited to ﬁnitely many isolated ﬁxed points in theory.
These ﬁndings were then experimentally veriﬁed in two dimensions by means of time series prediction
I NTRODUCTION
Recurrent neural networks (RNNs) have been widely used to capture and utilize sequential structure in natural and artiﬁcial languages, speech, video, and various other forms of time series.
The recurrent information ﬂow within RNN implies that the data seen in the past has inﬂuence on the current state of the RNN, forming a mechanism for having memory through (nonlinear) temporal traces.
Unfortunately, training vanilla RNNs (which allow input data to directly interact with the hidden state) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem (Hochreiter, 1991).
Several special RNN architectures have been proposed to mitigate this issue, notably the long short-term memory (LSTM; Hochreiter & Schmidhuber (1997)) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary.
Recently, a simpliﬁcation of the LSTM called the gated recurrent unit (GRU; Cho et al.
(2014)) has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017), speech (Prabhavalkar et al., 2017), music (Choi et al., 2017), video (Dwibedi et al., 2018), and extracting nonlinear dynamics underlying neural data (Pandarinath et al., 2018).
As a variant of the vanilla LSTM, GRUs incorporate the use of forget gates, but lack an output gate (Gers et al., 2000).
While this feature reduces the number of required parameters, LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017).
In addition, certain mechanistic tasks, speciﬁcally unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018).
Despite these empirical ﬁndings, we lack systematic understanding of the internal time evolution of GRU’s memory structure and its capability to represent nonlinear temporal dynamics.
In general, a RNN can be written as h t +1 =
f ( h t , x t ) where x t is the current input in a sequence indexed by t , f is a point-wise nonlinear function, and h t represents the hidden memory state that carries all information responsible for future output.
In the absence of input, the hidden state h t can evolve over time on its own:
[Equation]
where f ( · ) :
= f ( · , 0 ) for notational simplicity.
In other words, we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system deﬁned by (1).
Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features.
We develop a novel theoretical framework to study the dynamical features fundamentally attainable, in particular, given the particular form of GRU.
We then validate the theory by training GRUs to predict time series with prescribed dynamics.
C ONTINUOUS - TIME G ATED R ECURRENT U NIT
The GRU uses two internal gating variables: the update gate z t which protects the d -dimensional hidden state h t ∈ R d and the reset gate r t which allows overwriting of the hidden state and controls the interaction with the input x t ∈ R p .
z
[Equation]
where W z , W r , W h ∈ R d × p
and U z , U r ,
U h ∈ R d × d are the parameter matrices, b z , b r , b h ∈ R d are bias vectors, (cid:12) represents the Hadamard product, and σ ( z ) =
1 / (1 + e − z ) is the element-wise logistic sigmoid function.
Note that the hidden state is asymptotically contained within [ − 1 , 1] d due to the saturating nonlinearities, implying if the state is initialized outside of this trapping region, it must eventually enter it in ﬁnite time and remain in it for all later time.
Note that the update gate z t controls how fast each dimension at the hidden state decays, providing an adaptive time constant for memory.
Speciﬁcally, as lim z t → 0 h t = h t − 1 , GRUs can implement perfect memory of the past and ignore x t .
Hence, a d -dimensional GRU is capable of keeping a near constant memory through the update gate—near constant since 0 <
[ z t ] j < 1 , where [ · ] j denotes j -th component of a vector.
Moreover, the autoregressive weights (mainly U h and U r ) can support time evolving memory (Laurent & von Brecht (2016)) considered this a hindrance and proposed removing all complex dynamic behavior in a simpliﬁed GRU).
To investigate the memory structure further, let us consider the dynamics of hidden state in the absence of input, i.e. x t = 0 , ∀ t , which is of the form (1).
To utilize the rich descriptive language of continuous time dynamical system theory, we consider the following continuous time limit of the (autonomous) GRU time evolution:
[Equation]
where ˙
h ≡ d h ( t ) d t .
Since both σ ( · ) and tanh( · ) are smooth, this continuous limit is justiﬁed.
The update gate z ( t ) again plays the role of a state-dependent time constant for memory decay.
Furthermore, since 1 − z ( t ) > 0 , it does not change the topological structure of the dynamics (only speed).
For the following theoretical analysis sections (3 & 4), we can safely ignore the effects of z ( t ) .
A derivation of the continuous-time GRU can be found in appendix A.
S TABILITY A NALYSIS OF A O NE D IMENSIONAL GRU
For a single GRU ∗ ( d = 1 ), (7) reduces to a one dimensional dynamical system where every variable is a scalar.
The expressive power of a single GRU is quite limited, as only three stability structures (topologies) exist (see appendix B): ( A ) a single stable node, ( B ) a stable node and a half-stable node, and ( C ) two stable nodes separated by an unstable node (see Fig. 1).
The corresponding temporal features are (A) decay to a ﬁxed value, (B) decay to a ﬁxed value, but from one direction halt at an intermediate value until perturbed, or (C) decay to one of two ﬁxed values (bistability).
The bistability can be used to capture a binary latent state in the sequence.
It should be noted that a one dimensional continuous time autonomous system cannot exhibit oscillatory behavior, as is the case here (Hirsch et al., 2013).
[Figure]
The topology the GRU takes is determined by its parameters.
If the GRU begins in a region of the parameter space corresponding to (A), we can smoothly vary the parameters to transverse (B) in the parameter space, and end up at (C).
This is commonly known as a saddle-node bifurcation.
Speaking generally, a bifurcation is the change in topology of a dynamical system, resulting from a smooth change in parameters.
The point in the parameters space at which the bifurcation occurs is called the bifurcation point, and we will refer to the ﬁxed point that changes its stability at the bifurcation point as the bifurcation ﬁxed point. This corresponds to the parameters underlying (B) in our previous example. The codimension of a bifurcation is the number of parameters which must vary in order to achieve the bifurcation. In the case of our example, a saddle-node bifurcation is codimension-1 (Kuznetsov, 1998).
Right before transitioning to (B), from (A), the ﬂow near where the half-stable node would appear can exhibit arbitrarily slow ﬂow.
We will refer to these as slow points (Sussillo & Barak, 2012).
A NALYSIS OF A T WO D IMENSIONAL GRU
We will see that the addition of a second GRU opens up a substantial variety of possible topological structures when compared with the use of a single GRU.
For notational simplicity, we denote the two dimensions of h as x and y .
We visualize the ﬂow ﬁelds deﬁned by (7) in 2-dimension as phase portraits which reveal the topological structures of interest.
For starters, the phase portrait of two independent bistable GRUs can be visualized as Figure 2A. It clearly shows 4 stable states as expected, with a total of 9 stable ﬁxed points.
This could be thought of as a continuoustime continuous-space implementation of a ﬁnite state machine with 4 states (Fig. 2B).
The 3 types of observed ﬁxed points—stable (sinks), unstable (sources), and saddle points—exhibit locally linear dynamics, however, the global geometry is nonlinear and their topological structures can vary depending on their arrangement.
[Figure]
We explored stability structures attainable by two GRUs.
Due to the relatively large number of observed topologies, this section’s main focus will be on demonstrating all observed local dynamical features obtainable by two GRUs.
In addition, existence of two non-local dynamical features will be presented.
A complete catalog of all observed topologies can be found in the appendix C, along with the parameters of every phase portrait depicted in this paper.
Before proceeding, let us take this time to describe all the local dynamical features observed.
In addition to the previously mentioned three types of ﬁxed points, two GRUs can exhibit a variety of bifurcation ﬁxed points, resulting from regions of parameter space that separate all topologies restricted to simple ﬁxed points (i.e stable, unstable, and saddle points).
Behaviorally speaking, these ﬁxed points act as hybrids between the previous three, resulting in a much richer set of obtainable dynamics.
These bifurcation ﬁxed points fall into two categories, separated by codimension.
More speciﬁcally, two GRUs have been seen to feature both codimension-1 and codimension-2 bifurcation (ﬁxed) points.
Beginning with codimension-1, we have the saddle-node bifurcation ﬁxed point, as expected from its existence in the single GRU case.
We can further classify these points into two types.
These can be thought of as both the fusion of a stable ﬁxed point and a saddle point, and the fusion of an unstable ﬁxed point and a saddle point.
We will refer to these ﬁxed points as saddle-node bifurcation ﬁxed points of the ﬁrst kind and second kind respectively.
One type of codimension-2 bifurcation ﬁxed point that has been observed in the two GRU system acts as the fusion of all three simple ﬁxed points.
More speciﬁcally, these points arise from the fusion of a stable ﬁxed point, unstable ﬁxed point, and two saddle points.
All of these local structures are depicted in ﬁgure 3.
While the existence of simple ﬁxed points was already demonstrated (see Fig.
2A).
Figure 3A demonstrates the maximum number of ﬁxed points observed in a two GRU system, for a given set of parameters.
A closer look at this system reveals its potential interpretation as a continuous analogue of 5-discrete states with input-driven transitions, similar to that depicted in ﬁgure 2, implying additional GRUs are needed for any Markov process modeled in this manner, requiring more than ﬁve discrete states.
We conjecture that the system depicted in ﬁgure 3A is the only eleven ﬁxed point structure obtainable with two GRUs, as all observed structures containing the same number of ﬁxed points are topologically equivalent to one another.
The addition of bifurcation ﬁxed points opens the door to dynamically realize more sophisticated models.
Take for example the four state system depicted in ﬁgure 3B.
If the hidden state is set to initialize in the ﬁrst quadrant of phase space, the trajectory will ﬂow towards the codimension-2 bifurcation ﬁxed point at the origin.
Introducing noise through the input will stochastically cause the trajectory to approach the stable ﬁxed point at (-1,-1) either directly, or by ﬁrst ﬂowing into one of the two saddle-node bifurcation ﬁxed points of the ﬁrst kind.
Models of this sort can be used in a variety of applications, such as neural decision making (Wong & Wang (2006), Churchland & Cunningham (2014)).  1.5 x -1. - y -3.
[Figure]
x - -
[Figure]
[Figure] [Figure] [Figure] We will begin our investigation into the non-local dynamics observed with two GRUs by showing the existence of homoclinic orbits.
A trajectory initialized on a homoclinic orbit will approach the same ﬁxed point in both forward and backward time.
We observe that two GRUs can exhibit one or two bounded planar regions of homoclinic orbits for a given set of parameters, as shown in ﬁgure 4A and 4B respectively.
Any trajectory initialized in one of these regions will ﬂow into the codimension-2 bifurcation ﬁxed point at the origin, regardless of which direction time ﬂows in.
This featured behavior enables the accurate depiction of various models, including neuron spiking (Izhikevich, 2007).
In regards to the second non-local dynamic feature, it can be shown that two GRUs can exhibit an Andronov-Hopf bifurcation, whereby a stable ﬁxed point bifurcates into an unstable ﬁxed point surrounded by a limit cycle.
Behaviorally speaking, a limit cycle is a type of attractor, in the sense that there exists a deﬁned basin of attraction.
However, unlike a stable ﬁxed point, where trajectories initialized in the basin of attraction ﬂow towards a single point, a limit cycle pulls trajectories into a stable periodic orbit around the unstable ﬁxed point at its center.
To demonstrate this phenomenon, let (8) deﬁne the parameters of (7).
[Equation]
where α ∈ R + .
If α = π 3 , the system has a single stable ﬁxed point (stable spiral), as depicted in ﬁgure 5A. If we continuously decrease α , the system undergoes an Andronov-Hopf bifurcation approximately about α = π 3 . 8 .
As α continuously decreases, the orbital period increases, and as the nullclines can be made arbitrarily close together, the length of this 1.5 x -1. y -4. 1.5 x -1.
y -4.
[Figure]
[Figure] [Figure] orbital period can be set arbitrarily, up to machine accuracy. Figure 5B shows an example of a relatively short orbital period, and ﬁgure 5C depicts the behavior seen for slower orbits.
[Figure] Lemma 1.
Any two dimensional GRU can only have ﬁnitely many simple ﬁxed points.
This follows from Lefschetz theory (Guillemin & Pollack, 2010).
The detailed proof can be found in appendix D, and is intended to give the reader intuition behind the result presented in the claim extended to aribitrary dimensional GRU in theorem 1.
Theorem 1.
Any ﬁnite dimensional GRU can only have ﬁnitely many simple ﬁxed points and bifurcation ﬁxed points.
Proof.
By deﬁnition of simple ﬁxed points, the Jacobian of the dynamics at those ﬁxed points have nonzero real parts, making them Lefschetz ﬁxed points. Since GRU dynamics is asymptotically bounded on the compact set [ − 1 , 1] d , where d is the number of GRUs, it follows from Lefshetz theory (Guillemin & Pollack, 2010) that there are ﬁnitely many simple ﬁxed points. Furthermore, by construction, a bifurcation ﬁxed point can only exist within a stability structure if and only if there exists a separate topology, such that the simple ﬁxed points making up each bifurcation ﬁxed point exist isolated from one another. Therefore, there can only exist ﬁnitely many isolated bifurcation ﬁxed points.
This eliminates the possibility of having countably many ﬁxed points.
Next, we show that there cannot be uncountably many non-isolated ﬁxed points. Theorem 2. Any ﬁnite dimensional GRU cannot have a continuous attractor.
Proof.
We provide a sketch of a proof.
Let h ∗ ∈ R
d be a ﬁxed point of a d -dimensional GRU, that is, h ∗ − tanh ( U h ( r ( h ∗ ) (cid:12) h ∗ ) + b h )
= 0 .
Now for any unit norm vector k ∈ S d − 1 , and for any δ > 0 , we can show that there exist an (cid:15) > 0 such that, (cid:107) ˙
h ( h ∗ + δ k ) − ˙ h ( h ∗ ) (cid:107) =
(cid:107) ˙
h ( h ∗ + δ k ) (cid:107) > (cid:15) . This can be argued by taking three cases into consideration, (a) U {+r k = 
+}
and U h ( σ ( b r ) (cid:12) k ) (cid:54) = 0 , {+(b) U r k = 
+}
and U h ( σ ( b r ) (cid:12) k ) = 0 , and (c)
U r k (cid:54) = 0 .
In each case, it reverts to a 1-dimensional problem where it can be trivially shown to have no continuous attractor around h ∗ along direction k .
Thus, we conclude that there is no continuous attractor in any direction.
Despite this limitation, an approximation of a line attractor can be created using two GRUs.
This approximation can be made arbitrarily close to an actual line attractor on a ﬁnite region in phase space, thereby satisfying computational needs on an arbitrary interval when scaled.
We will refer to this phenomenon as a pseudo-line attractor .
Figuredepicts an example of such an attractor.
[Figure]
We conclude this section with a discussion of slow points in the two GRU system.
As a logical extension to the single GRU system, slow points occur where the nullclines are sufﬁciently close together, but do not intersect, as demonstrated in ﬁgure 7.
Given the previously discussed classes of dynamic behavior for two GRUs, slow points can only exist so long as the potential for a saddle-node bifurcation ﬁxed point is possible in the location of the desired slow point, given an appropriate change in parameters, as they result from the collision and annihilation of two ﬁxed points.
This observation is consistent with the single GRU case, as slow points can only exist for a single ﬁxed point.
This would imply that given the one ﬁxed point case, a maximum of ﬁve slow points are possible.
However, this would imply that there must exist a six ﬁxed point case by which ﬁve of the six ﬁxed points exist at saddle-node bifurcation ﬁxed points, which has not been observed (see appendix C).
Despite this shortcoming, four simultaneous slow points are obtainable, as shown in ﬁgure 5C.
1.5 x -1.
y -3.
25 t -1.
x(t) y(t) passing through slow point
A
B
N UMERICAL E XPERIMENTS
As a means to put our theory to practice, in this section we explore several examples of time series prediction of continuous time planar dynamical systems using two GRUs.
Results from the previous section indicate what dynamical features can be learned by this RNN, and suggest cases by which training will fail.
All of the following computer experiments consist of an RNN, by which the hidden layer is made up of two GRUs, followed by a linear output layer.
The network is trained to make a 29-step prediction from a given initial observation, and no further input through prediction.
As such, to produce accurate predictions, the RNN must rely solely on the hidden layer dynamics.
We train the network to minimize the following multi-step loss function:
[Equation]
(9) where θ are the parameters of the GRU and linear readout, T = 29 is the prediction horizon, w i ( t ) is the i -th time series generated by the true system, and ˆ w ( k ; w 0 ) is k -step the prediction given w 0 .
The hidden states are initialized at zero for each trajectory.
The RNN is then trained for 4000 epochs, using ADAM (Kingma & Ba, 2014) in whole batch mode to minimize the loss function, i.e., the mean square error between the predicted trajectory and the data.
N traj = 667 time series were used for training.
Figure 8 depicts the experimental results of the RNN’s attempt at learning each dynamical system we describe below.
5. L IMIT C YCLE
To test if two GRUs can learn a limit cycle, we use a simple nonlinear oscillator called the FitzHugh-Nagumo Model.
The FitzHugh-Nagumo model is deﬁned by:
[Equation]
where in this experiment we will chose τ = 12 .
5 , a = 0 .
7 , b = 0 .
8 , and I ext
= N (0 . 7 , 0 . 04) .
Under this choice of model parameters, the system will exhibit an unstable ﬁxed point (unstable spiral) surrounded by a limit cycle (Fig. 8).
As shown in section 4, two GRUs are capable of representing this topology.
The results of this experiment verify this claim (Fig.
8), as two GRUs can capture topologically equivalent dynamics.
5. L INE A TTRACTOR
As discussed in section 4, two GRUs can exhibit a pseudo-line attractor, by which the system mimics an analytic line attractor.
We will use the simplest representation of a planar line attractor:
[Equation]
1.5 x -1.
y - - - -   - -0.
1 x - -0.
y - - - - - -  -1.5 -1 -0.50.
1.5 x -1.
y - - - -
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
300 t -    - -   2 x
[Figure]
-     y
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
y
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
x -
[Figure]
[Figure]
[Figure]
y - - [Figure] [Figure] [Figure]
[Figure]
T a r g e t
A n a l y t i c a l D y n a m i c s
I n f e rr e d
G R U
D y n a m
i
c s
Limit Cycle
Line Attractor
Ring Attractor
This system will exhibit a line attractor along the y -axis, at x = 0 (Fig.
Trajectories will ﬂow directly perpendicular towards the attractor. We added white Gaussian noise N (0 , 0 . 1 I ) in the training data. While the hidden state dynamics of the trained network do not perfectly match that of an analytic line attractor, there exists a subinterval near each of the ﬁxed points acting as a pseudo-line attractor (Fig.
As such, the added afﬁne transformation (linear readout) can scale and reorient this subinterval as is required by a given problem, thereby mimicking a line attractor.
For this experiment, a dynamical system representing a standard ring attractor of radius one is used:
[Equation]
This system exhibits an attracting ring, centered around an unstable ﬁxed point.
We added Gaussian noise N (0 , 0 .
1 I ) to the training data.
Two GRUs will not be able to accurately capture the system’s continuous attractor dynamics as expected from theorem 3.
The results of this experiment are demonstrated in ﬁgure 8.
As expected, the RNN fails to capture the proper dynamical features of the ring attractor.
Rather, the hidden state dynamics fall into an observed ﬁnite ﬁxed point topology (see case xxix in appendix C).
In addition, we robustly see this over multiple initializations, and the quality of approximation improves as the dimensionality of GRU increases (Fig. 9).
Four network sizes (2, 4, 8, 16 dimensional GRU) were trained 3 timeswith different initializations.
Initial learning rate for ADAM was tuned using Bayesian optimization procedure.
C ONCLUSION
Our analysis shows the rich but limited classes of dynamics the GRU can approximate in one, two, and arbitrary dimensions.
We developed a new theoretical framework to analyze GRUs as a continuous dynamical system, and showed that two GRUs can exhibit a variety of expressive dynamic features, such as limit cycles, homoclinic orbits, and a substantial catalog of stability structures and bifurcations.
However, we also showed that ﬁnitely many GRUs cannot exhibit the dynamics of an arbitrary continuous attractor.
These claims were then experimentally veriﬁed in two dimensions.
We believe these ﬁndings also unlock new avenues of research on the trainability of recurrent neural networks.
Although we have analyzed GRUs only in 1- and 2- dimensions in near exhaustive, we believe that the insights extends to higher-dimensions.
We leave rigorous analysis of higher-dimensional GRUs as future work.
A C ONTINUOUS T IME S YSTEM D ERIVATION
We begin with the fully gated GRU as a discrete time system, where the input vector x t has been set equal to zero, as depicted in (13) - (15), where (cid:12) is the Hadamard product, and σ is the sigmoid function .
[Equation]
We recognize that (15) is a forward Euler discretization of a continuous time dynamical system.
This allows us to consider the underlying continuous time dynamics on the basis of the discretization.
The following steps are a walk through of the derivation:
Since z t is a bounded function on R ∀ t , there exists a function ˜
z t , such that z t + ˜ z t = 1 at each time step (due to the symmetry of z t , ˜ z t is the result of vertically ﬂipping z t about 0 .
5 , the midpoint of its range).
As such, we can rewrite (15) with ˜ z t as depicted in (16).
[Equation]
where,
[Equation] [Equation]
Let h ( t ) ≡ h t − 1 .
As a result, we can say ˜
z
t ≡ ˜ z ( t ) and r t ≡ r ( t ) , as depicted in (20).
[Equation]
where,
[Equation]
Let ∆ t deﬁne an arbitrary time interval.
Then (20) becomes,
[Equation]
Dividing both sides of the equation by ∆ t yields (24).
[Equation]
If we take the limit as ∆ t → 0 , we get the analogous continuous time system to (13) - (15),
[Equation]
where ˙ h ≡ dh ( t ) dt
Finally, we can rewrite (25) as follows:
[Equation]
[Equation]
where
[Equation]
B S INGLE GRU F IXED P OINT P ROOFS
The ﬁxed points of our continuous time system (25) exist where the derivative ˙ h
= 0 .
In the single GRU case, the
Hadamard product reduces to standard scalar multiplication, yielding,
[Equation]
where z ( t ) and r ( t ) are deﬁned by (27) and (22) respectively, and h ∗ ∈ R represents a solution of (28).
We can divide out z ( t ) − 1 , indicating that the update gate does not play a part in the stability of the system.
For simplicity, lets expand r ( t ) in (28) by its deﬁnition (22).
[Equation]
where U h , b h , U r , b r ∈ R .
Lemma 2.
for all U h , b h , U r , b r , there exists h ∗ such that (29) is satisﬁed.
Proof.
The hyperbolic tangent function is continuous and bounded on R , having a range of ( − 1 , 1) .
− h ( t ) ≡ − h is monotonic and achieves all values on R , as − h is bijective on R .
Thus, their sum is unbounded and obtains every value on R at least once.
By the intermediate value theorem, there is at least one point h ∗ such that (29) is satisﬁed, regardless of the choice of parameters
U h , b h , U r , b r .
Lemma 3.
There exists a set of parameters U h , b h , U r , b r such that there exists one, two, or three solutions to (29) .
Proof.
We will prove this lemma by showing the existence of each case.
Let U r = 80 , b r = 40 , and U h = − 60 .
We then allow b h to vary.
The existence of each of the three cases are shown in ﬁgure 1.
If b h = − 1 , there exists a single solution to (29).
If b h decreases continuously, a second root appears and splits in two.
Analogously, the system (25) goes through a saddle-node bifurcation , where a half-stable ﬁxed point appears, and splits into a stable/unstable ﬁxed point pair.
Theorem 3.
For any choice of parameters U r , b r , U h , b h , there can only exist one, two, or three solutions to (29) , and all solutions exist on the interval (-1,1).
Proof.
We begin with the argument of the hyperbolic tangent function in (29),
[Equation]
Taking the derivative of (30) yields,
[Equation]
Setting (31) to zero and simplifying will allow us to ﬁnd the nontrivial critical points of (30), as shown in (32).
Note that if U h = 0 , (30) is equal to b h ∀ h , yielding no critical points.
[Equation]
Let x ≡ e − b r and ˆ h ≡ U r h , and solve (32) for ˆ h .
[Equation]
where W is principal branch of the Lambert W function .
Therefore, (30) has exactly one local maximum or minimum, so long as U h (cid:54) = 0 .
Now consider,
[Equation]
The hyperbolic tangent function preserves intervals of monotonic behavior in its argument.
Therefore, (34) has at most one local maximum or minimum.
We take into account the fact that the hyperbolic tangent function bounds its argument on the interval (-1,1).
If there exists a subset S =
[ a, 1] , for some a ∈ [ − 1 , 1) such that (34) is increasing, then there exists a k ∈ S such that when h = k
(35) and (36) hold.
[Equation]
This result in conjunction with the previous two lemmas completes the proof.
C A LL T OPOLOGICAL S TABILITY S TRUCTURES O BSERVED WITH T WO GRU S
Table 2 depicts all observed topologies of multiple-ﬁxed point structures using two GRUs.
Figure 10 displays an example of a phase portrait from a two GRU system for each case listed in 2.
Note that all ﬁxed points are denoted by a red dot, regardless of classiﬁcation.
Table 3 lists the parameters used for each of the observed cases.
Note that all the update gate parameters are set to zero.
Each case in this paper was discovered by hand by considering the geometric constraints on the structure of nullclines for both the decoupled and coupled system (i.e reset gate inactive and active respectively). An exhaustive analysis on the one dimensional GRU allowed for a natural extension into the two dimensional decoupled GRU. Upon establishing a set of base cases (a combinatorial argument regarding all possible ways the decoupled nullclines [topologically conjugate to linear and cubic polynomials] can intersect) From these base cases, the reset gate can be used as a means of bending and manipulating structure of the decoupled nullclines in order to obtain new intersection patterns in the coupled system.
[Table]
5b 2. 5c 2.
6a 0.
[Table] [Table]
7a
[Table]
[Table]
- - - 10xxiii xxiv   - - - - 10xxiv xxv     - - 10xxv xxvi  -  - - 10xxvi xxvii  -   - 10xxvii xxviii  - - - 10xxviii xxix     - - 10xxix xxx  -   - 10xxx xxxi    - - - 10xxxi xxxii     - - 10xxxii xxxiii  - - - - 10xxxiii xxxiv     - - 10xxxiv xxxv  - - - 10xxxv xxxvi    - - - 10xxxvi
[Figure]
[Figure]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
D P ROOF OF L EMMA
We begin this proof by showing that all ﬁxed points obtainable with two GRUs are Lefschetz ﬁxed points.
To show this is the case let (37) expand our previous notation.
We set all elements in U z and b z to zero, as the update gate plays no part in the topology of (7) (shown in appendix B).
[Equation]
We can now rewrite (7) expanded in terms of the individual elements of U h ,
U r , b h , and b r , as shown in (38) and (39).
[Equation]
If 0 is not an eigenvalue of the Jacobian matrix of (38) and (39) at a ﬁxed point, the ﬁxed point is said to be Lefshetz.
Let F x = d ˙ xdx , F y =
d ˙ xdy , G x = d ˙ ydx , and F y = d ˙ ydy . where [Equation]
Let J denote the Jacobian matrix of (38) and (39).
[Equation] Note that we can rewrite (38) and (39) as follows: [Equation] where θ represents the set of parameters, f ( x, y, θ )
=
U
[Equation] 1+ [Equation] [Equation]
+
[Equation] [Equation] + [Equation]
1+ e −
[Equation]
Ur 21 x + Ur 22 y + br 2) + b h 1 , and g ( x, y, θ ) =
[Equation]
An ordered pair ( x
[Equation] [Equation] As such [Equation]
we can say the following
[Equation]
If we let λ represent the eigenvalues of (48), the characteristic equation of (48) is as follows:
[Equation] We can rewrite (55) in terms of u ( x, y, θ ) and v ( x, y, θ ) as shown in (56) [Equation] where u x ≡ ∂u∂x , u y ≡ ∂u∂y , v x ≡ ∂v∂x , v y ≡ ∂v∂y .
We can use the quadratic formula to solve for λ . [Equation] Setting λ = 0 and simplifying yields the following constraint: [Equation]
which can be realized as follows:
[Equation] We observe that sech 2 ( f ( x, y, θ ))sech 2 ( g ( x, y, θ )) ∈ (0 , 1) , which implies that ∂f∂y ∂g∂x ∈ (1 , ∞ ) However, from (53) and (54), we see that f ( x, y, θ ) = tanh − 1 ( x ) and g ( x, y, θ ) = tanh − 1 ( y ) at a critical point.
Which implies ∂f∂y ∂g∂x = 0 / ∈ (1 , ∞ ) .
{+Therefore λ (cid:54) = 
+}
∀ θ .
This implies that (38) and (39) is a Lefschetz map.
Since (38) and (39) are asymptotically bound to ( − 1 , 1) 2 , we can always ﬁnd a ﬁnite time t 0 such that x, y ∈ ( − 1 , 1) 2 ∀ t > t 0 .
Therefore, for every trajectory initialized outside of the trapping region, we can always ﬁnd a point on [ − 1 , 1] 2 that arises as the transition of that initial condition ﬂowing into the trapping region.
This implies that (38) and (39) can be thought of as existing on a compact set, and therefore has a ﬁnite number of simple ﬁxed points Guillemin & Pollack (2010).