Long Short-Term Memory Neural Network Equilibria Computation and Analysis
Abstract
This paper presents a comprehensive approach for computing nontrivial equilibria of autonomous Long Short-Term Memory neural networks using a homotopy formulation.
Through simulations, it is shown that the eigenvalues of the linearized models around these nontrivial equilibria tend to move closer to the unit circle as the complexity of the training data increases.
This provides insights into the dynamical properties of the LSTM neural networks
Introduction
Since Long-Short Term Memory networks (LSTMs) were initially introduced in [1], there has been a wealth of successful examples employing these types of networks for audio, written text and video recognition (see, for example, [2] and references reported therein).
While much of the effort has been centered around improving training algorithms for LSTMs, there has been relatively little work in trying to understand their dynamical properties and behaviors.
In particular, it has been well recognized that it is not uncommon for the trained neural networks to operate at the edge of chaos [3, 4].
Recently, it has been shown in [5] that the edge of chaos can be characterized by the stability region for the linearized model of the autonomous LSTM model with an equilibrium at zero.
Motivated by this work, we use a linear path-following homotopy [6] to compute non-trivial equilibria of the autonomous LSTM and the corresponding linearized models.
Through simulations, as reported for the zero equilibrium in [5] and as performed for non-trivial equilibria in the same way (not provided in this paper due to the space limitation), we observed that the edge of chaos for the nonlinear LSTMs is accurately characterized by the eigenvalues’ proximity to the boundary of stability (that is, the unit circle).
In order to provide this characterization, we computed and analyzed the eigenvalues of the linearized model of a single layer LSTM neural network, trained on three benchmark tasks: handwritten digit recognition, text generation, and polyphonic music generation.
For the image classiﬁcation, we use the MNIST data set [7] which has only ten classes.
Then, the Penn Tree Bank (PTB) data set, which is more rich and has many more classes, is used for the text generation task.
The result is that, compared to the MNIST data set, the eigenvalues are pushed even further toward the unit circle.
Finally, we use the Nottingham data set [8] for the music generation task, which also corroborates our claim that increasing the number and the complexity of training signals, particularly from different classes, causes the eigenvalues to approach the unit circle, effectively causing the trained network to operate at the edge of stability, that is, chaos.
We hope that these initial results and observations make LSTMs more comprehensible and therefore will open up new research avenues and the possibility for improved network design, training and inference.
The paper is organized as follows.
In Section 2, we provide the LSTM discrete-time dynamical model and its basic properties.
A homotopy formulation and how it is used to compute the LSTMs’ nonzero equilibria, are provided in Section 3.
Representative sets of simulations are presented in Section 4.
Finally, some concluding remarks are provided in Section 5.
LSTM Model
In this paper, we are interested in studying the properties of the autonomous LSTM Neural Network (NN), that is the standard LSTM model introduced by [1] without input vectors.
Mathematically, the model is written as the following vector difference equation:
[Equation]
where h ( · ) T , c ( · ) T ∈ R n are, respectively, the output and memory state vectors, (cid:12) is the Hadamard product, and σ ( · ) :
= 1 1+ e − ( · ) is the standard logistic function.
W f,i,o,g ∈ R n × n and b f,i,o,g ∈ R n are constant and represent the weights and biases of the neural network.
Studying the properties of autonomous LSTMs is of much interest because they are representative of the intrinsic dynamical behavior of the general models with input and therefore exploring their behavior may indicate why these systems perform well.
In this paper, we study the behavior of autonomous LSTMs with non-trivial equilibria.
It is straightforward to see that for b g (cid:54) =
the equilibria are not trivial and computing them is very hard given that the steady-state equations are transcendental, yet the equilibria’s numerical values are needed to compute the linearized models.
Thus, in the following section, we introduce a path-following homotopy method to compute the nonzero equilibria.
Homotopy Approach to Computing LSTM Nonzero Equilibria
The homotopy continuation method is a well-known technique to solve the problem of ﬁnding the roots and ﬁxed points of nonlinear functions.
In topology, a homotopy between two continuous functions F, G : R n → R m is deﬁned as another continuous function H : [0 , 1] × R n → R m such that H (0 , · ) ≡ G ( · ) and H (1 , · ) ≡ F ( · ) .
Now, If x 0 ∈ R n is a known root of G , and H is continuously differentiable then the homotopy method allows us to track the path x : [0 , 1] → R n that starts at x (0) = x 0 and satisﬁes H ( t, x ( t )) ≡ 0 for all t ∈ [0 , 1] .
It is clear that, at t = 1 ,
H (1 , x (1))
=
F ( x (1)) = 0 , thus x (1) is a root of F .
A more rigorous introduction can be found in [6].
In this section, we explain, brieﬂy, our procedure to ﬁnd the equilibria of system (1).
To this end, we deﬁne a new linear homotopy map that allows us to compute the nontrivial equilibria for LSTMs when b g (cid:54) = 0 by starting from the trivial solution when b g = 0 .
To start, consider the LSTM model in equation (1) and let h ( k )
=
q ( k )
+
q c , where q ( · ) ∈ R n is a new variable and q c ∈ R n satisﬁes
W g q c + b g = 0 .
The existence of such q c is guaranteed, if b g is in the span of columns of W g , which is a very mild requirement.
Therefore, system (1) can be, equivalently, rewritten as:
[Equation]
where x ( k ) :
=
[ c ( k ) T , q ( k ) T ] T and F
[Equation]
such that, the new biases are deﬁned as ¯
b j = b j + W j q c , for j ∈ { f, i, o } .
Note that the equilibria of (2) are the roots of the map x → x − F ( x ) − x c .
Therefore, we deﬁne our homotopy map as:
[Equation]
such that the equilibria of (2) are the solutions, at t = 1 , to H ( t, x ) = 0 .
Now, deﬁne
[Equation]
as the set of all the solution curves to H ( t, x ) = (cid:15) , for some ﬁxed (cid:15) ∈ R n .
Let these curves be parameterized using a new independent variable θ ∈ R such that ( t ( θ ) , x ( θ ))
∈
H − 1 (cid:15) for all θ ∈ R .
Since H ∈ C ∞ ( R n × n × [0 , 1]) , and moreover if we assume that the Jacobian matrix
H (cid:48)
( t, x ) :=
[ ∂H/∂x ∂H/∂t ] is of full rank, that is rank ( H (cid:48)
( t, x )) ≥ n for all ( t, x ) ∈
H − 1 (cid:15) , then the implicit function theorem ensures, that H − 1 (cid:15) is composed solely of continuously differentiable curves that are solutions to the following differential equation:
[Equation]
Therefore, the equilibrium point of system (2) is simply the solution, at t = 1 and for (cid:15) = 0 , of the differential equation (5) along with the initial condition ( t ( θ ) , x ( θ ))
= (0 , 0) .
A singular case may occur when the Jacobian matrix [ ∂H/∂x ∂H/∂t ] is not of full rank yet Sard’s theorem ensures that the set of singular values are of measure zero.
Thus, there exists always an (cid:15) ∈ R n , arbitrarily close to the origin, such that H − 1 (cid:15) is composed only by continuously differentiable curves that are solutions to (5).
Therefore, the homotopy method allows us to get arbitrarily close to any equilibria of (2).
Simulations
In this section, we provide a number of simulations corresponding to nonzero equilibria analysis to show how the behavior of the LSTMs are correlated with the location of the eigenvalues and how they propagate during training depending on the number of training samples, classes, and tasks.
The ﬁrst simulation is related to the training of the LSTM neural network to recognize handwritten digits from
MNIST database [7] and we observe that the eigenvalues, generally, move towards the unit circle during training.
However, they do not reach it, which we believe is a consequence of the training set consisting of only 10 different labels/classes (that is, digits from 0 to 9).
In the second experiment, we use a more rich class of training signals; the Penn Tree Bank (PTB) data set [9] to train an LSTM neural network, in the context of language modeling and word prediction.
Results show that the eigenvalues are pushed more closely toward the unit circle which conﬁrms our proposition that the more rich and complex the input data is, the more eigenvalues are pushed toward the unit circle.
To conﬁrm this trend, we train an LSTM network, using the Nottingham data set [8], to predict melodies and harmonics in the context of polyphonic music modeling.
Our general procedure is to train single-layer LSTMs on the data sets mentioned previously, where the inputs are vectors of ﬁxed size representing the pixel intensities for a given row in the context of image recognition, words in the context of language modelling, and melodies/harmonics in the context of music modeling.
We perform a softmax operation on the output of the LSTM in order to get a probability distribution.
Then, the loss is deﬁned to be the cross-entropy between the distribution and the corresponding label/prediction in "one-hot" encoding format.
Training is ﬁnally accomplished via backpropagation through time (BPTT) with various batch sizes.
Throughout the training, the weight matrices and bias vectors are stored at constant intervals.
Then, equilibria of the autonomous LSTMs, constructed using each of these weights and biases, are computed using the homotopy method.
4.1 Equilibria of LSTM neural networks used for MNIST digit recognition
This ﬁrst simulation set is used to demonstrate the dependence of the absolute value of eigenvalues during training.
A neural network conﬁguration with the input layer consisting of 10 neurons, hidden layer with 128 neurons and 10 output neurons has been chosen.
The particular training task, as mentioned earlier, is for the LSTM network to recognize handwritten digits from the MNIST data set [7].
The inputs were vectors with ﬁxed size representing the pixel intensities for a given row in an image.
To demonstrate how the eigenvalues propagate during training itself, we trained the
LSTM with a data set of 100 samples and stored the weights and biases every 200 gradient steps.
The corresponding eigenvalues are plotted in Fig.
The magnitude of the largest eigenvalues are shown as a function of the training step in Fig.
1, where an increasing trend can be clearly observed.
Fig.
1c indicates that their maximum is ∼ 0 .
81 , which can be explained by the small number of classes in the data set.
This result corresponds to the previous analysis.
[Figure]
4.2 Equilibria of LSTM neural network used for language modeling
To show that the eigenvalues can be pushed further toward the unit circle, we use the Penn Tree Bank (PTB) data set to train the LSTM to predict the next word given some previous words.
This data set has ∼ 10 k words in its vocabulary, which is much larger than the number of classes in the previous example.
The conﬁguration of the network and its training is done as explained above.
The size of the input, output, and cell vectors is 200.
Fig.
2a shows how the largest eigenvalues propagate during
12 Training Steps
E i gen v a l ue s 0.1 0.
[Figure]
12 Training steps 0.
E i gen v a l ue s
[Figure]
training.
It is clear that they are pushed toward the unit circle.
Fig.
2b indicates, in fact, that two eigenvalues have a magnitude of ∼ 0 .
99 (very close to the unit circle).
Note that, because the real part of these two eigenvalues was also ∼ 0 .
99 , the convergence of the homotopy solver was very slow.
This is due to the fact that, in that case, ˙ t ( θ ) in equation (5) was very close to zero which makes t increase very slowly toward one.
4.3 Equilibria of LSTM neural network applied for polyphonic music generation
In this experiment [Figure] we use the Nottingham data set [8] to train a single layer LSTM neural network in the same manner explained above.
The input vectors are of dimension 68; the ﬁrst half of the input vector represents the basic musical notes (or the melodies) and the second part the chords (or harmonics).
The dimension of the output vector is 200.
The results conﬁrm that the eigenvalues are pushed toward the unit circle during training.
Fig 3a-b exhibits clearly this tendency.
Fig 3c
[Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
E
[Figure]
depicts that, although the spectral radius gets closer and crosses the unit circle, all real parts of the corresponding eigenvalues are well smaller than 1, which implies that ˙ t ( θ ) in equation (5) is non-zero, and therefore, t can monotonically increase towards 1 without causing any numerical issues for the homotopy.
This has a consequence that the convergence of the homotopy solver was faster in this case than in the previous simulation.
Note that, it is the maximal eigenvalue magnitude that characterizes the boundary of stability, which we stipulate does coincide with the edge of chaos, as shown for the trivial equilibrium in [5].
Conclusion
A homotopy-based approach to compute equilibria of autonomous LSTM neural networks, is provided in this paper.
Through simulations, it has been shown how the locations of the eigenvalues of the linearized models around the computed equilibria depend on the training sample sizes and tasks.
Some of our current research activities are focused on providing connections among the computation of the weights and biases, complexity of the training data, and the location of the equilibria.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No.