D ON ’ T J UDGE A B OOK BY ITS C OVER - O N THE D YNAMICS OF R ECURRENT N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
To be effective in sequential data processing, Recurrent Neural Networks (RNNs) are required to keep track of past events by creating memories .
Consequently
RNNs are harder to train than their feedforward counterparts, prompting the developments of both dedicated units such as LSTM and GRU and of a handful of training tricks.
In this paper, we investigate the effect of different training protocols on the representation of memories in RNN.
While reaching similar performance for different protocols, RNNs are shown to exhibit substantial differences in their ability to generalize for unforeseen tasks or conditions.
We analyze the dynamics of the network’s hidden state, and uncover the reasons for this difference.
Each memory is found to be associated with a nearly steady state of the dynamics whose speed predicts performance on unforeseen tasks and which we refer to as a ’slow point’.
By tracing the formation of the slow points we are able to understand the origin of differences between training protocols.
Our results show that multiple solutions to the same task exist but may rely on different dynamical mechanisms, and that training protocols can bias the choice of such solutions in an interpretable way.
I NTRODUCTION
Recurrent Neural Networks (RNN) are the key tool currently used in machine learning when dealing with sequential data (Sutskever et al., 2014), and in many tasks requiring a memory of past events (Oh et al., 2016).
This is due to the dependency of the network on its past states, and through them on the entire input history.
This ability comes with a cost - RNNs are known to be hard to train (Pascanu et al., 2013a).
This difﬁculty is commonly associated with the vanishing gradient that appears when trying to propagate errors over long times (Hochreiter, 1998), and has been addressed by introducing various architectures such as Long Short-Term Memory (LSTM) cells (Hochreiter &
Schmidhuber, 1997), Gated Recurrent Units (Chung et al., 2015).
In addition to network architecture, training protocols can also facilitate training success.
Inspired by shaping in behavioral training, curriculum learning (Bengio et al., 2009; Cirik et al., 2016) suggests gradually increasing task demands until arriving at the full task.
This was shown to accelerate training, albeit reaching fairly similar results after convergence of training (Jozefowicz et al., 2015;
Cirik et al., 2016).
For complex tasks, however, there are many possibilities on how to gradually increase difﬁculty.
In this work, we ask whether different curricula indeed lead to the same outcome.
We choose a task in which there is a natural division of computational resources that is common for many RNN tasksmemory and processing.
RNNs are not only required to memorize the input, but are also trained to process it.
This processing, which could perhaps be easier for a feed-forward network, utilizes the same connections that are trained for the memory component of the task.
We study a simple task combining these elements, which fails for naive training.
We thus start training the network on one aspect of the task; either ﬁrst learning to extract features or ﬁrstly learn to memorize, and when successful moving to learn the second aspect as well.
Both approaches work, leading to comparable performance on the full task.
A more careful analysis, however, reveals that the resulting networks differ in their extrapolation abilities and reﬂect their training histories.
Understanding the underlying cause for these differences requires some form of reverse engineering.
This can be done by focusing on individual recurrent units (Karpathy et al., 2015;
Oh et al., 2016), or by analyzing global network properties.
We opt for the latter, analyzing the RNN’s hidden states as a discrete-time dynamical system.
In this framework, memories might be associated with a wide range of dynamical objects. On one extreme, transient dynamics can be harnessed for memory operations Manjunath & Jaeger (2013); Maass (2011); Maass et al.
On the other extreme, there are memory networks Sukhbaatar et al. (2015) that memorize everything and later use only the relevant memories while ignoring all the rest.
The idealized dynamical scenario where each memory is associated with a ﬁxed point in the RNN state space (Hopﬁeld, 1982; Sussillo, 2014; Barak, 2017; Amit, 1989) was reﬁned in (Sussillo & Barak, 2013; Mante et al., 2013) where points which are only approximately ﬁxed (slow points), with a drift that is slower than the task duration were shown to represent memory.
To this end, we extend tools used in continuous-time systems in neuroscience (Sussillo & Barak, 2013) to address the aforementioned differences in extrapolation abilities.
Examining relation between drift speed of memories, which we ﬁnd to be represented by slow points of varying slowness, and the extrapolation properties of their associated class, we establish such a correlation in a large variety of settings.
Finally, we obtain an instructive insight on how the memory robustness is altered along the training course. Detailed analysis of individual training trajectories makes it possible to monitor the formation of slow points under a speciﬁc training protocol.
This technique offers us an explanation of interplay between newly created and existing slow points decreasing stability of the latter in a systematic and tractable manner.
This provides a link between training curriculum, dynamical objects, memory and performance.
T ASK DEFINITION
Inspired by real-world applications of Recurrent Neural Networks (Oh et al., 2016), we designed a task where the RNN has to combine stimulus processing and memorization (ﬁgure 1).
The network is presented with a series of noisy images, among which appears a single target image (from MNIST or CIFAR) at time t s .
At a later time point, t a , the network receives a response trigger in a separate input channel, prompting it to output the label of the image.
At all other times, the network should report a null label.
The stimulus and reporting times are chosen randomly each trial from a uniform distribution on [1 , T max ] subject to the constraint t a − t s > 4 .
The total stimulation time T max = 20 , and the network was requested to distinguish between | V | = 10 different classes of MNIST (LeCun et al., 2010) or CIFAR-10 (Krizhevsky et al.).
Each pixel of the noise mask was sampled from a Gaussian distribution with mean and variance matching its counterpart at the image corpus (cid:15) ∼ N ( µ n , σ 2 n ).
We tested the RNNs ability to extrapolate from this task in two directions, the ﬁrst was increasing the delay T max , and the second was increasing the noise σ 2 .
The motivation behind this task is three-fold.
First, as explained, this task is comparable to realworld scenarios where RNNs are used for, combining the need for both stimulus memorization and feature extraction.
Second, the task lends itself to parametric variations, allowing to compare both different training protocols and generalization abilities.
Third, desiring to understand the dynamical nature of memorization in discrete Gated-RNNs, the delay between stimulus and response trigger allows for evolution of RNN hidden-state (HS), which can be reliably analyzed using well known methods from dynamical systems (Sussillo & Barak, 2013), which we modify to our discrete setting.
M ODEL
For MNIST, the network consists of a single recurrent layer of d = 200 gated recurrent units, an output layer of | V | + 1 = 11 neurons, | V | = 10 neuron for the different classes, and an additional neuron for the null indicator.
The input layer has n + 1 neurons, where n is the number of pixels in the image and an extra binary input channel for the response trigger X r ( t ) deﬁned by:
[Equation]
For CIFAR-10, the network was expanded to d = 400 recurrent units, along with a convolutional front-end composed of three convolutional layers and two dense layer.
To eliminate issues of translational invariance regarding the response trigger and the convolutional front-end, the trigger was added as an extra channel to the ﬁnal dense layer, right before the recurrent units.
The gated units are either GRU or LSTM:
[Equation]
For the analysis of the network’s phase space, we denote the state of the recurrent layer by ξ , which for LSTM is ξ = (cid:18) h tanh [Equation] and for GRU ξ = h .
The network was trained using the ’Adam’ optimizer (Kingma & Ba, 2014) with a soft-max crossentropy loss function with a increased loss on reporting at t =
t a in proportion to T max .
Full description of each protocol, including schedules and other hyper-parameters is given in Appendix
C.
T RAINING PROTOCOL : T WO TYPES OF CURRICULA
We found that training failed when using straightforward SG optimization on the full task.
The network converged to a state where it consistently reports ’null’ without regarding neither the output trigger nor the images it has received as inputs.
This suboptimal behavior did not improve upon further training.
On the other hand, we observed that simpler versions of the task are learn-able.
If the maximal delay between stimulus and reporting time was short or when we introduced only a limited number of different digits, the network was able to perform the task.
This led us to try two different protocols of curriculum learning in order teach the network the full required task:
Vocabulary curriculum (VoCu) - here we started from two classes (cid:48) V (cid:48)
= { c 1 , c 2 } and then increased the vocabulary gradually until reaching the full class capacity.
This protocol is similar to the original concept of (Bengio et al., 2009) except the fact that in our vocabulary all the classes occur with the same frequency, and the selected order of curricula is in fact arbitrary.
Delay curriculum (DeCu) - starting from short delays between stimulus and reporting time ( T max = 6 ), we progressively extended it toward the desired values.
Implicitly mentioned in (Hochreiter, 1998), this regime is expected to mitigate the vanishing gradient problem, at least during initial phase of training.
E XTRAPOLATION A BILITY D EPENDS ON T RAINING P ROTOCOL
We found that, in good accordance with existing literature (Bengio et al., 2009; Jozefowicz et al., 2015) results for the nominal test-set were fairly indifferent to the training protocol (Appendix A).
Once we evaluated the ability of each setting to extrapolate to more challenging task settings, however, similarity ends and differences emerge.
We tested two different extrapolation settings, motivated by different hypotheses on the underlying mechanism of the network.
As mentioned in the Introduction, prior work provides a wide spectrum of alternative hypotheses – from systems without stable states, through slow points, and to robust ﬁxed points.
We evaluated these hypotheses by either extending the delay or adding noise.
First, we observed how each setting performs when the delay between stimulus and response trigger is extended further beyond T max = 20 .
If | V | = 10 robust ﬁxed-point attractors have formed, retrieval accuracy should not be affected by the growing delay.
If the computation is based on transients, then all class information is expected to eventually vanish.
Experiments revealed that neither of these exterme options was the case - performance deteriorated with increasing delay, but did not reach chance levels (ﬁgure 2).
This deterioration implies that not every memorized digit corresponds to a stable ﬁxed point attractor, but some do.
Furthermore, the deterioration was curriculum-dependent, with DeCu outperforming VoCu for all cases except
CIFAR-GRU.
Similarly, we can evaluate reporting accuracy when a the noise mask has a higher variance than used for training 1 .
Once more, this serves to evaluate the hypothetical ﬁxed points, and ﬁgure 2 shows that not all curricula are equally able to mitigate input-driven noise which can indicates differences in the basin-of-attraction of the hypothetical attractors formed under each setting.
Both extrapolation experiments show that despite comparable results superﬁcially, the governing dynamics and representation of the networks vary with training protocol, resulting in different extrapolation performance.
An additional extrapolation experiment requiring reporting the class twice with two triggers also shows large differences between the protocols (Appendix B).
Both extrapolation experiments reveal a similar trend, with the DeCu protocol being superior to
VoCu in three out of the four conditions studied.
For the CIFAR-10 dataset with a GRU architecture, we found that DeCu extrapolates worse than DeCu.
We will return to this difference in the next section.
D YNAMICS OF HIDDEN REPRESENTATION
The relevant phase space of this dynamical system is the recurrent layer state ξ .
We thus begin by visually inspecting (in the ﬁrst 3 PCA components) the activity of the network for the maximal training delay, ∆ t = 20 .
We show here results for the MNIST dataset with a GRU architecture, but similar behavior is seen for other conditions and the statistics of all conditions is analyzed below.
The left panels of Figure 3 show that different trials of each digit are well separated into | V | regions with a one to one correspondence to data classes.
Following these trajectories for a longer delay of ∆ t = 1000 shows that some regions converge into what appears to be ﬁxed points, while others vanish (middle panels).
This is true even when a smaller noise amplitude is used (right panels), although here more regions survive.
These ﬁgures also clearly show the difference between the two protocols.
While both achieve a good separation with the nominal delay (left), it is already apparent that VoCu leads to clouds of points with a larger spread, possibly indicating a weaker attraction.
To verify the existence or absence of ﬁxed points hinted by the above visualization, we apply an algorithm developed for continuous time vanilla RNN Sussillo & Barak (2013) to our setting.
Brieﬂy, ﬁxed points (stable or unstable) are local minima of the (scalar) speed S ( ξ, I ) of the hidden state ξ .
[Equation]
where the evolution of state,
[Equation]
is given by equations equation 2 or equation 3 for GRU or LSTM respectively.
It is now possible to use gradient descent on the speed S with respect to state ξ , namely, ∇ ξ S , to locate such minima.
The initial conditions for this gradient descent were obtained by running the network with the mean delay value ∆ t = 15 , and using the average of hidden states of each class.
The external input
States are color codded by their label in the dataset.
For the nominal delay ten distinct regions are observed in the state space corresponding to each of the | V | = 10 classes.
While this picture is in agreement with the naive hypothesis that each class has its dedicated ﬁxed-point, examination of a larger delay ∆ t = 10 3 reveals that while some classes collapse into a single point others vanish completely.
Testing whether this disappearance due to insufﬁcient basin of attraction around true ﬁxed points or because of points being slow rather completely ﬁxed we observe that also under small noises, there is still an evanescence of certain classes for large delays, albeit more classes survive. The Spread of samples along with fewer number of distinct ﬁxed points at ∆ t = 10 3 in VoCu align with our ﬁndings of faster and less stable dynamics compared to DeCu discussed in sequel.
I during gradient descent was the average of the noise images, thus effectively making our system
Time-Invariant where such points and their stability are well deﬁned.
We veriﬁed that using different ﬁxed external inputs did not qualitatively alter the results (not shown).
We repeated the procedure for several realizations, and it always resulted in a local minimum of speed for each class (which we call slow point), with a readout that matches the class label.
Our hypothesis was that slow points with higher speed would be predictive of low extrapolation performance, and thereby explain the variability of Fig 2.
We thus located slow points for every class of several network realizations for all settings.
For every slow point, we computed its speed, and the accuracy of its associated class after a long delay.
Figure 4 shows that the speed of the slow point associated with a certain class can predict how members of that class will react to extrapolation experiments.
This trend holds for all architectures, unit types and datasets tested.
The colored bars in Figure 4 denote the mean and standard deviation (not standard error) of the speeds obtained by the two different protocols.
The picture here is consistent with that observed in Figure 2, with DeCu outperforming VoCu for three of the four cases.
Our results suggest that this difference is mediated by a difference in speed of the associated slow points.
What exactly is the reason for the different behavior of the CIFAR dataset with a GRU architecture is left for future work.
F ORMATION OF SLOW POINTS - WHY CURRICULA DIFFER
We saw that the two training protocols lead to a different representation of the stimulus memory by the network, and to different dynamical objects.
How does training give rise to these differences?
To answer this question, we analyze in detail one setting - a GRU architecture trained on the MNIST database.
We follow the slow points of the velocity backwards in training time to learn how they arise and change throughout training, and correlate these events with network performance.
Except VoCu-LSTM ( p = 0 . 07 ) all differences are signiﬁcant ( p < 0 . 05 ).
The difference in speeds between the protocols underlies the different extrapolation performance shown in Figure 2.
Ten networks were used for MNIST, and ﬁve for CIFAR-10.
We located slow points as described in section 6 , and then used them as initial conditions for gradient descent on the network deﬁned by the previous training step.
We then repeated this procedure iteratively to all training steps.
The assumption is that the change in network parameters at each training step will not induce a very large shift in the locations of the relevant slow points, and thus our continuation procedure can track them.
This is not clear in the case of VoCu, where one might expect discontinuities whenever a new class is added.
Looking at the speeds of the tracked slow points shows that this is indeed the case for VoCu (ﬁgure 5, A,B ), with all tracked speeds exhibiting such jumps.
DeCu, on the other hand, shows a gradual slowing down of the slow points throughout training.
By observing the gradual slowing down in DeCu, it is quite easy to understand why this protocol improves performance - slow points become slower.
But the situation for VoCu is more complicated.
A natural expectation might be that the classes that were presented ﬁrst will have more time to stabilize, and thus will be the slowest.
We saw that this is not the case (not shown), and thus proceeded to look deeper into VoCu.
We inspected the jumps in VoCu more carefully, knowing that this is where the tracking algorithm could fail.
We thus perform a short tracking procedure for each class - starting before the succeeding class is introduced, and following it back to the ﬁrst training iteration in which it appeared.
The result is a training-trajectory of a slow point, from the moment it ﬁrst appears until the next class is introduced.
For each point in this trajectory, we calculated the readout of the network.
By stitching together the results of all short-tracking procedure of all classes, we obtain a full bifurcation diagram (5 C ).
Our results show that slow points do not emerge out of thin air, but rather bifurcate from existing slow points.
For example class ’7’ emerged from classes ’3’ or ’4’.
Such an event might affect the stability of classes ’3’ and ’4’, prompting us to evaluate this possibility.
To more accurately observe the extent of such effect, we add noise to reduce network performance, and then check the difference in performance of the various classes following the introduction of class ’7’.
Figure 5 ( D ) shows that indeed the robustness of classes ’3’ and ’4’ was adversely affected compared to the other classes.
To evaluate the statistics of this phenomenon, we repeat this procedure for many networks, and all class introduction events.
Figure 5 ( E ) com- pares the change in accuracy of the classes that spawned the new class with the decay in accuracy of the other classes.
We thus see that adding new classes corresponds to adding new slow points, which emerge from existing ones.
The performance of the spawning classes is speciﬁcally adversely affected by these events.
D ISCUSSION
Training RNNs to perform tasks is difﬁcult (Pascanu et al., 2013b), and as a consequence many suggestions were made on how to alleviate this difﬁculty.
Changing network architecture or unittypes might be expected to generate different solutions to the same task.
Using different training protocols, however, is thought to allow better convergence to similar solutions.
Here we showed that different training protocols can lead to different locally optimal solutions.
Although these solutions perform similarly under nominal conditions, challenging the networks with unforeseen settings reveals their differences.
An RNN is a dynamical system, and as such its operation can be understood in the language of ﬁxed points and other dynamical objects.
By analyzing the phase space of the network’s hidden states, we showed that the memory of each class was associated with a slow point of the dynamics.
The speeds of these slow points were highly correlated to the functional characteristics of memory longevity, and thus provide a dynamical explanation of the idiosyncrasies observed between curricula and architectures.
Our result proved valid across architectures, datasets and unit types.
For one example, we were able to follow the formation of the aforementioned slow points during the training process.
We characterized the topology of the phase space and showed that slow-points bifurcate one form another upon introduction of new classes.
This bifurcation event is associated with a decrease in network accuracy that is speciﬁc for the classes involved in the bifurcation.
Things could have been different - slow points could emerge in an area of phase space that is distant from existing ones, and the introduction of a new class could have resulted in a uniform effect on all existing classes.
To uncover this bifurcation structure, we introduced a backtracking methodology that could be relevant to any case in which learning modiﬁes the dynamics of the network.
Possible applications include studying the success and failure in creating memories, preventing catastrophic forgetting, understanding memory capacity and more.
Slowly increasing the delay resulted in better stability and increased extrapolation ability for three out of the four conditions tested.
For the CIFAR-10 database with GRU units, we observed the opposite trend.
While we did not uncover the reason for this anomaly, we speculate that once the issue is understood and resolved it will outperform VoCu.
We note that even in this setting the same strong negative correlation between slow-point speed and memory still holds.
The setting studied in this work is a particular case of the more general problem of solution equivalence in gradient based optimization: On the one hand theoretical and numerical evidence does exist for training outcome’s indifference to protocol details.
Speciﬁcally to our case of RNN, it is shown in (Cirik et al., 2016) that, at least for language modeling tasks, performance does not heavily depend on training protocol.
On the other hand, such an indifference is far from being fully established.
In particular, stochastic gradient optimization suffers from known drawbacks (Dauphin et al., 2014; Martens & Sutskever, 2011) and might prove dependent on initialization (Sutskever et al., 2013)
(and in particular pre-training).
Our results show that networks with the same initialization can reach different solutions, and begin to uncover the dynamics underlying the route to these solutions.
A A CCURACIES ON T EST AND T RAIN SETS FOR NOMINAL TASK
Below are the accuracies on the MNIST and CIFAR-10 train and test sets for the nominal task, for both GRU and LSTM architectures and the described curricula.
[Table] [Table]
[Table] [Table]
B A CCURACIES WHEN REQUIRING A SECOND RETRIEVAL
Accuracies when extending the nominal task to two trigger setting.
Speciﬁcally, after the ﬁrst trigger is introduced, we add another one and observe the decay in accuracy upon that second trigger.
[Table]
[Table]
C H YPER - PARAMETERS C. MNIST
The network was trained for a total of 14 · 10 4 Gradient Decent steps, using the ADAM optimizers.
For the ﬁrst 12 · 10 4 steps, the learning rate was 10 − 4 and for the latter steps the learning rate was decreased by a factor of 10 .
The DeCu schedules was starting with a maximal delay of T max = 6 , then incrementally increasing the maximal delay by two timesteps at the following training steps: [20 , 30 , 40 , 50 , 60 , 70 , 80] · 10 3 .
For VoCu, training started by learning the ﬁrst two classes
[Table]
and each class what added in order according at these training steps: [8 , 16 , 26 , 36 , 48 , 62 , 78 , 98] · 10 3 .
The noise mask for this dataset was a normal distribution with mean µ = 0 .
1307 and standard deviation σ = 0 .
C. CIFAR-
For CIFAR-10, a convolutional front-end was added consisting for three convolutional layers with 64 each, and an additional two fully-connected layers with 1024 neurons each.
The network was for 20 · 10 4 Gradient Decent steps, with an initial learning rate of 10 − 4 and was decreased by a factor of 10 for the ﬁnal 2 · 10 − 4 steps.
The DeCu scheduling was at steps [30 , 45 , 60 , 75 , 90 , 105 , 120] · 10and the VoCu scheduling was [12 , 24 , 38 , 54 , 72 , 92 , 124 , 140] · 10 3 .
The noise mask for this dataset was a normal distribution with mean µ = 0 .
4734 and standard deviation σ = 0 .