T HE GEOMETRY OF INTEGRATION IN TEXT CLASSIFICATION RNN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the widespread application of recurrent neural networks (RNNs), a uniﬁed understanding of how RNNs solve particular tasks remains elusive.
In particular, it is unclear what dynamical patterns arise in trained RNNs, and how those patterns depend on the training dataset or task.
This work addresses these questions in the context of text classiﬁcation, building on earlier work studying the dynamics of binary sentiment-classiﬁcation networks (Maheswaranathan et al., 2019). We study text-classiﬁcation tasks beyond the binary case, exploring the dynamics of
RNNs trained on both natural and synthetic datasets. These dynamics, which we ﬁnd to be both interpretable and low-dimensional, share a common mechanism across architectures and datasets: speciﬁcally, these text-classiﬁcation networks use low-dimensional attractor manifolds to accumulate evidence for each class as they process the text.
The dimensionality and geometry of the attractor manifold are determined by the structure of the training dataset, with the dimensionality reﬂecting the number of scalar quantities the network remembers in order to classify. In categorical classiﬁcation, for example, we show that this dimensionality is one less than the number of classes. Correlations in the dataset, such as those induced by ordering, can further reduce the dimensionality of the attractor manifold; we show how to predict this reduction using simple word-count statistics computed on the training dataset.
To the degree that integration of evidence towards a decision is a common computational primitive, this work continues to lay the foundation for using dynamical systems techniques to study the inner workings of RNNs.
I NTRODUCTION
Modern recurrent neural networks (RNNs) can achieve strong performance in natural language processing (NLP) tasks such as sentiment analysis, document classiﬁcation, language modeling, and machine translation.
However, the inner workings of these networks remain largely mysterious.
As RNNs are parameterized dynamical systems tuned to perform speciﬁc tasks, a natural way to understand them is to leverage tools from dynamical systems analysis.
A challenge inherent to this approach is that the state space of modern RNN architectures—the number of units comprising the hidden state—is often high-dimensional, with layers routinely comprising hundreds of neurons.
This dimensionality renders the application of standard representation techniques, such as phase portraits, difﬁcult.
Another difﬁculty arises from the fact that RNNs are monolithic systems trained end-toend.
Instead of modular components with clearly delineated responsibilities that can be understood and tested independently, neural networks could learn an intertwined blend of different mechanisms needed to solve a task, making understanding them that much harder.
Recent work has shown that modern RNN architectures trained on binary sentiment classiﬁcation learn low-dimensional, interpretable dynamical systems (Maheswaranathan et al., 2019).
These
RNNs were found to implement an integration-like mechanism, moving their hidden states along a line of stable ﬁxed points to keep track of accumulated positive and negative tokens.
Later, Maheswaranathan & Sussillo (2020) showed that contextual processing mechanisms in these networks— e.g. for handling phrases like not good —build on top of the line-integration mechanism, employing an additional subspace which the network enters upon encountering a modiﬁer word.
The understanding achieved in those works suggests the potential of the dynamical systems perspective, but it remained to be seen whether this perspective could shed light on RNNs in more complicated settings. In this work, we take steps towards understanding RNN dynamics in more complicated language tasks, illustrating recurrent network dynamics in multiple text-classiﬁcation tasks with more than two categories.
The tasks we study—document classiﬁcation, review score prediction (from one to ﬁve stars), and emotion tagging—exemplify three distinct types of classiﬁcation tasks. As in the binary sentiment case, we ﬁnd integration of evidence to underlie the operations of these networks; however, in multi-class classiﬁcation, the geometry and dimensionality of the integration manifold depend on the type of task and the structure of the training data. Understanding and precisely characterizing this dependence is the focus of the present work.
Our contributions
• We study three distinct types of text-classiﬁcation tasks— categorical , ordered , and multilabeled —and ﬁnd empirically that the resulting hidden state trajectories lie largely in a lowdimensional subspace of the full state space. • Within this low-dimensional subspace, we ﬁnd a manifold of approximately stable ﬁxed pointsnear the network trajectories, and by linearizing the network dynamics, we show that this manifold enables the networks to integrate evidence for each classiﬁcation as they processes the sequence.
We ﬁnd ( N − 1) -dimensional simplex attractors 2 for N -class categorical classiﬁcation, planar attractors for ordered classiﬁcation, and attractors resembling hypercubes for multi-label classiﬁcation, explaining these geometries in terms of the dataset statistics.
• We show that the dimensionality and geometry of the manifold reﬂects characteristics of the training dataset, and demonstrate that simple word-count statistics of the dataset can explain the observed geometries. • We develop clean, simple synthetic datasets for each type of classiﬁcation task. Networks trained on these synthetic datasets exhibit similar dynamics and manifold geometries to networks trained on corresponding natural datasets, furthering an understanding of the underlying mechanism.
Related work Our work builds directly on previous analyses of binary sentiment classiﬁcation by Maheswaranathan et al. and Maheswaranathan & Sussillo (2020).
Apart from these works, the dynamical properties of continuous-time RNNs have been extensively studied (Vyas et al., 2020), largely for connections to neural computation in biological systems.
Such analyses have recently begun to yield insights on discrete-time RNNs: for example, Schuessler et al.
(2020) showed that training continuous-time RNNs on low-dimensional tasks led to low-dimensional updates to the networks’ weight matrices; this observation held empirically in binary sentiment LSTMs as well. Similarly, by viewing the discrete-time GRU as a discretization of a continuous-time dynamical system, Jordan et al. (2019) demonstrated that the continuous-time analogue could express a wide variety of dynamical features, including essentially nonlinear features like limit cycles.
Understanding and interpreting learned neural networks is a rapidly-growing ﬁeld. Speciﬁcally in the context of natural language processing, the body of work on interpretability of neural models is reviewed thoroughly in Belinkov & Glass (2018).
Common methods of analysis include, for example, training auxiliary classiﬁers (e.g., part-of-speech) on RNN trajectories to probe the network’s representations; use of challenge sets to capture wider language phenomena than seen in natural corpora; and visualization of hidden unit activations as in Karpathy et al.
and Radford et al.
S ETUP
Models We study three common RNN architectures: LSTMs (Hochreiter & Schmidhuber, 1997),
GRUs (Cho et al., 2014), and UGRNNs (Collins et al., 2016).
We denote their n -dimensional hidden state and d -dimensional input at time t as h t and x t , respectively.
The function that applies hidden state update for these networks will be denoted by F , so that h t =
F ( h t − 1 , x t ) .
The network’s hidden state after the entire example is processed, h T , is fed through a linear layer to get N output logits for each label:
y =
Wh T + b .
We call the rows of W ‘readout vectors’ and denote the readout corresponding to the i th neuron by r i , for i = 1 , . . .
, N .
Throughout the main text, we will present results for the GRU architecture.
Qualitative features of results were found to be constant across all architectures; additional results for LSTMs and UGRNNs are given in Appendix E.
Tasks
The classiﬁcation tasks we study fall into three categories.
In the categorical case, samples are classiﬁed into non-overlapping classes, for example “sports” or “politics”.
By contrast, in the ordered case, there is a natural ordering among labels: for example, predicting a numerical rating (say, out of ﬁve stars) accompanying a user’s review.
Like the categorical labels, ordered labels are exclusive.
Some tasks, however, involve labels which may not be exclusive; an example of this multi-labeled case is tagging a document for the presence of one or more emotions.
A detailed description of the natural and synthetic datasets used is provided in Appendices C and D, respectively.
Linearization and eigenmodes Part of our analysis relies on linearization to render the complex
RNN dynamics tractable.
This linearization is possible because, as we will see, the RNN states visited during training and inference lie near approximate ﬁxed points h ∗ of the dynamics—points that the update equation leave (approximately) unchanged, i.e. for which h ∗ ≈ F ( h ∗ , x ) .
Near these points, the dynamics of the displacement ∆ h t :
= h t − h ∗ from the ﬁxed point h ∗ is well approximated by the linearization
[Equation]
where we have deﬁned the recurrent and input Jacobians J rec ij ( h , x ) :=
∂F ( h , x ) i ∂h j and J inp ij ( h , x ) :=
∂F ( h , x ) i ∂x j , respectively (see Appendix A for details).
In the linear approximation, the spectrum of J rec plays a key role in the resulting dynamics.
Each eigenmode of J rec represents a displacement whose magnitude either grows or shrinks exponentially in time, with a timescale τ a determined by the magnitude of the corresponding (complex) eigenvalue λ a via the relation τ a := | log | λ a || − 1 .
Thus, eigenvalues within the unit circle thus represent stable (decaying) modes, while those outside represent unstable (growing) modes.
The Jacobians we ﬁnd in practice almost exclusively have stable modes, most of which decay on very short timescales (a few tokens).
Eigenmodes near the unit circle have long timescales, and therefore facilitate the network’s storage of information.
Latent semantic analysis
For a given text classiﬁcation task, one can summarize the data by building a matrix of word or token counts for each class (analogous to a document-term matrix (Manning & Schutze, 1999), where the documents are classes).
Here, the i, j entry corresponds to the number of times the i th word in the vocabulary appears in examples belonging to the j th class.
In effect, the column corresponding to a given word forms an “evidence vector”, i.e. a large entry in an particular row suggests strong evidence for the corresponding class.
Latent semantic analysis (LSA) (Deerwester et al., 1990) looks for structure in this matrix via a singular value decomposition (SVD); if the evidence vectors lie predominantly in a low-dimensional subspace, LSA will pick up on this structure.
The top singular modes deﬁne a “semantic space”: the left singular vectors correspond to the projections of each class label into this space, and the right singular vectors correspond to how individual tokens are represented in this space.
Below, we will show that RNNs trained on classiﬁcation tasks pick up on the same structure in the dataset as LSA; the dimensionality and geometry of the semantic space predicts corresponding features of the RNNs.
Regularization While the main text focuses on the interaction between dataset statistics and resulting network dimensionality, regularization also plays a role in determining the dynamical structures.
In particular, strongly regularizing the network can reduce the dimensionality of the resulting manifolds, while weakly regularizing can increase the dimensionality.
Focusing on (cid:96) 2 -regularization, we document this effect for the synthetic and natural datasets in Appendicies D.1 and F, respectively.
R ESULTS 3. C ATEGORICAL CLASSIFICATION YIELDS SIMPLEX ATTRACTORS
We begin by analyzing networks trained on categorical classiﬁcation datasets, with natural examples including news articles (AG News dataset) and encyclopedia entries (DBPedia Ontology dataset)
We ﬁnd dynamics in these networks which are largely low-dimensional and governed by integration.
Contrary to our initial expectations, however, the dimensionality of the network’s integration manifolds are not simply equal to the number of classes in the dataset.
For example, rather than exploring a three-dimensional cube, RNNs trained on 3-class categorical tasks exhibit a largely twodimensional state space which resembles an equilateral triangle (Fig.
1a, d).
As we will see, this is an example of a pattern that generalizes to larger numbers of classes
Synthetic categorical data To study how networks perform N -class categorical classiﬁcation, we introduce a toy language whose vocabulary includes N + 1 words: a single evidence word “evid i ” for each label i and a neutral word “neutral”.
Synthetic phrases, generated randomly, are labeled with the class for which they contain the most evidence words (see Appendix D for more details).
This is analogous to a simple mechanism which classiﬁes documents as, e.g., “sports” or “ﬁnance” based on whether they contain more instances of the word “football” or “dollar”.
The main features of the categorical networks’ integration manifolds are clearly seen in the 3-class synthetic case.
First, the dynamics are low-dimensional: performing PCA on the set of hidden states explored from hundreds of test phrases reveals that more than 97% of its variance is contained in the top two dimensions.
Projected onto these dimensions, the set of network trajectories takes the shape of an equilateral triangle (Fig. 1a).
Diving deeper into the dynamics of the trained network, we examine the deﬂections, or change in hidden state, ∆ h t , induced by each word.
The deﬂections due to evidence words “evid i ” align with the corresponding readout vector r i at all times (Fig. 1b).
Meanwhile, the deﬂection caused by the “neutral” word is much smaller, and on average, nearly zero.
This suggests that the RNN dynamics approximate that of a two-dimensional integrator: as the network processes each example, evidence words move its hidden state within the triangle in a manner that is approximately constant across the phrase.
The location of the hidden state within the triangle encodes the integrated, relative counts of evidence for each of the three classes.
Since the readouts are of approximately equal magnitude and align with the triangle’s vertices, the phrase is ultimately classiﬁed by whichever vertex is closest to the ﬁnal hidden state.
This corresponds to the evidence word contained the most in the given phrase.
Natural categorical data Despite the simplicity of the synthetic categorical dataset, its working mechanism generalizes to networks trained on natural datasets.
We focus here on the 3-class AG
News dataset, with matching results for 4-class AG news and
3- and 4-class DBPedia Ontology in Appendix E. Hidden states of these networks, as in the synthetic case, ﬁll out an approximate equilateral triangle whose vertices once again lie parallel to the readout vectors (Fig. 1d).
While these results bear a strong resemblance to their synthetic counterparts, the manifolds for natural datasets are, unsurprisingly, less symmetric.
Though the vocabulary in natural corpora is much larger than the synthetic vocabulary, the network still learns the same underlying mechanism: by suitably arranging its input Jacobian and embedding vectors, it aligns an input word’s deﬂection in the direction that changes relative class scores appropriately (Fig. 1e).
Most words behave like the synthetic word “neutral”, causing little movement within the plane; certain words, however, (like “football”) cause a large shift toward a particular vertex (in this case, “Sports”).
Again, the perturbation is relatively uniform across the plane, indicating that the order of words does not strongly inﬂuence the network’s prediction.
In both synthetic and natural cases, the two-dimensional integration mechanism is enabled by a manifold of approximate ﬁxed points, or slow points , near the network’s hidden state trajectories, which allow the network to maintain its position in the absence of new evidence for all t = 1 , . . , T .
As the position within the plane encodes the network’s integrated evidence, this maintenance is essential.
In all 3-class categorical networks, we ﬁnd a planar, approximately triangle-shaped manifold of ﬁxed points which lie near the network trajectories (Fig. 1c, f); vertices of this manifold align with the readout vectors.
PCA reveals the dimensionality of this manifold to be very similar to that of the hidden states.
Since we ﬁnd the network’s hidden state trajectories always lie close to the ﬁxed point manifold, we can use the ﬁxed points’ stability as an approximate measure of the network’s ability to store integrated evidence.
We check for stability directly by linearizing the dynamics around each ﬁxed point and examining the spectra of the recurrent Jacobians.
Almost all of the Jacobian’s eigenvalues are well within the unit circle, corresponding to perturbations which decay on the timescale of a few tokens.
Only two modes, which lie within the ﬁxed-point plane, are capable of preserving information on timescales on the order of the mean document length (Fig. 2a). This linear stability analysis conﬁrms our picture of a two-dimensional attractor manifold of ﬁxed points; the network [Figure] [Figure]
[Figure] [Figure] [Figure]
P C [Figure] dynamics quickly suppress activity in dimensions outside of the ﬁxed-point plane. Integration, i.e. motion within the ﬁxed-point plane is enabled by two eigenmodes with long time constants (relative to the average phrase length).
LSA predictions Intuitively, the two-dimensional structure in this three-class classiﬁcation task reﬂects the fact that the network tracks relative score between the three classes to make its prediction.
To see this two-dimensional structure quantitatively in the dataset statistics, we apply latent semantic analysis (LSA) to the dataset, ﬁnding a low-rank approximation to the evidence vectors of all the words in the vocabulary.
This analysis (Fig. 1h) shows that two modes are necessary to capture the variance, just as we observed in the RNNs.
Moreover, the class vectors projected into this space (Fig. 1g) match exactly the structure observed in the RNN readouts.
The network appears to pick up on the same structure in the dataset’s class counts identiﬁed by LSA.
General N -class categorical networks The triangular structure seen in the 3 -class networks above is an example of a general pattern: N -class categorical classiﬁcation tasks result in an ( N − 1) - dimensional simplex attractor (Fig. 3a).
We verify this with synthetic data consisting of up toclasses, analyzing the subspace of R n explored by the resulting networks.
More than 95% of the variance of the hidden states is contained in N − 1 dimensions, with the subspace taking on the approximate shape of a regular ( N − 1) -simplex centered about the origin (Fig. 1b).
The readout vectors, which lie almost entirely within this ( N − 1) -dimensional subspace, align with the simplex’s vertices.
Mirroring the 3-class case, dynamics occur near a manifold of ﬁxed points which also shaped like an ( N − 1) -simplex.
This simplex geometry reﬂects the fact that to classify between N classes, the network must track N − 1 scalars: the relative scores for each class.
As a natural example of this simplex, the full, 4-class AG News dataset results in networks whose trajectories explore an approximate 3-simplex, or tetrahedron.
The ﬁxed points also form a 3dimensional tetrahedral attractor (Fig. 3c).
Additional results for 4-class natural datasets, which also yield tetrahedron attractors, are shown in Appendix E.
3. O RDERED CLASSIFICATION YIELDS PLANE ATTRACTORS
Having seen networks employ simplex attractors to integrate evidence in categorical classiﬁcation, we turn to ordered datasets, with Yelp and Amazon review star prediction as natural examples.
Star prediction is a more ﬁnely-grained version of binary sentiment prediction that RNNs solve by integrating valence along a one-dimensional line attractor (Maheswaranathan et al., 2019).
This turns out not to be the case for either 3-class or 5-class star prediction.
For a network trained on the 5-class Yelp dataset, we plot the two-dimensional projection of RNN trajectories while processing a test batch of reviews as well as the readout vectors for each class (Fig. 4d).
Similar results for 3-class Yelp and Amazon networks are in
Appendix E.
The top two dimensions capture more than 98% of the variance in the explored states: as with categorical classiﬁcation tasks, the dynamics here are largely low-dimensional.
A manifold of ﬁxed points that is also planar exists nearby (Fig. 4f).
The label predicted by the network is determined almost entirely by the position within the plane.
Additionally, eigenmodes of the linearized dynamics around these ﬁxed points show two slow modes with timescales comparable to document length, separated by a clear gap from the other, much faster, modes (Fig. 2c, d).
These two integration modes lie almost entirely within the ﬁxed-point plane, while the others are nearly orthogonal to it.
These facts suggest that — in contrast to binary sentiment analysis — 5-class (and 3-class) ordered networks are two-dimensional , tracking two scalars associated with each token rather than simply a single sentiment score.
As an initial clue to understanding what these two dimensions represent, we examine the deﬂections in the plane caused by particular words (Fig. 4f).
These deﬂections span two dimensions—in contrast to a one-dimensional integrator, the word ‘horrible’ has a different effect than multiple instances of a weaker word like “poor.”
These two dimensions of the deﬂection vector seem to roughly correspond to a word’s “sentiment” (e.g. good vs. bad) and “intensity” (strong vs. neutral).
In this two-dimensional integration, a word like ‘okay’ is treated by the network as evidence of a neutral (e.g., 3-star) review.
Inspired by these observations, we build a synthetic ordered dataset with a word bank { amazing , good , okay , bad , awful , neutral } , in which each word now has a separate sentiment and intensity score.
4 Labels are assigned to phrases based on both its total sentiment and intensity; e.g., phrases with low intensity and sentiment scores are classiﬁed as “3 stars”, while phrases with high positive sentiment and high intensity are “5 stars” (see Appendix D.2 for full details).
Networks trained on this dataset correspond very well to the networks trained on Amazon and Yelp datasets (Fig.
4a-c).
Dynamics are largely two-dimensional, with readout vectors fanning out in the plane from ﬁve stars to one.
Deﬂections from individual words correspond roughly to the sentiment and intensity scores, and the underlying ﬁxed-point manifold is two-dimensional.
More generally, the appearance of a plane attractor in both 3-class and 5-class ordered classiﬁcation shows that in integration models, relationships (such as order) between classes can change the dimensionality of the network’s integration manifold.
These relationships cause the LSA evidence vectors for each word to lie in a low-dimensional space.
As in the previous section, we can see this low-dimensional in the dataset statistics itself using LSA, showing that two singular values explain more than 95% of the variance (Fig. 4f).
Thus, the planar structure of these networks, with dimensions tracking both (roughly) sentiment and intensity, is a consequence of correlations present in the dataset itself. 3. M ULTI - LABELED CLASSIFICATION YIELDS INDEPENDENT ATTRACTORS So far, we have studied classiﬁcation datasets where there is only a single label per example. This only requires networks to keep track of the relative evidence for each class, as the overall evidence does not affect the classiﬁcation.
Put another way, the softmax activation used in the ﬁnal layer will normalize out the total evidence accumulated for a given example. This results in networks that, for an N -way classiﬁcation task, need to integrate (or remember) at most N − 1 quantities as we have seen above. However, this is not true in multi-label classiﬁcation.
Here, individual class labels are assigned independently to each example (the task involves N independent binary decisions). Networks trained on this task do need to keep track of the overall evidence level. To study how this changes the geometry of integration, we trained RNNs on a multi-label classiﬁcation dataset, GoEmotions (Demszky et al., 2020). Here, the labels are emotions and a particular text may be labeled with multiple emotions.
We trained networks on two reduced variants of the full dataset, only keeping two or three labels. The results for three labels are detailed in Appendix E.5. For the two-class version, we only kept the labels “admiration” and “approval”, and additionally resampled the dataset so each of the 2 2 = 4 possible label combinations were equally likely. We found that RNNs learned a two-dimensional integration manifold where the readout vectors span a two-dimensional subspace (Fig. 5d), rather than a one-dimensional line as in binary classiﬁcation.
Across the ﬁxed point manifold, there were consistently two slow eigenvalues (Fig. 5e), corresponding to the two integration modes. Similar to the previous datasets, increasing (cid:96) 2 regularization would (eventually) compress the dimensionality, again measured using the participation ratio (Fig. 5f). Notably, GoEmotions is a highly-imbalanced dataset; we found that balancing the number of examples per class was important to observe a match between the synthetic and the natural dynamics.
[Figure]
The synthetic version of this task classiﬁes a phrase as if it were composed of two independent sentiment analyses (detailed in Appendix D.3).
This is meant to represent the presence/absence of a given emotion in a given phrase, but ignores the possibility of correlations between certain emotions. After training a network on this data, we ﬁnd a low-dimensional hidden-state and ﬁxed-point space that both take on the shape of a square (Fig. 5a, c).
The deﬂections of words affecting independent labels act along orthogonal directions (Fig. 5b). These results suggest that integration manifolds are also found in RNNs trained on multi-labeled classiﬁcation datasets. Moreover, the geometry of the corresponding ﬁxed points and readouts is different from the exclusive case; instead of an ( N − 1) -dimensional simplex we get an N -dimensional hypercube. Again, this makes intuitive sense given that the networks must keep track of N independent quantities in order to solve these tasks.
D ISCUSSION
In this work we have studied text classiﬁcation RNNs using dynamical systems analysis. We found integration via attractor manifolds to underlie these tasks, and showed how the dimension and geometry of the manifolds were determined by statistics of the training dataset. As speciﬁc examples, we see ( N − 1) -dimensional simplexes in N -class categorical classiﬁcation where the network needs to track relative class scores; 2 -dimensional attractors in ordered classiﬁcation, reﬂecting the need to track sentiment and intensity; and N -dimensional hypercubes in N -class multi-label classiﬁcation. We hope this line of analysis — using dynamical systems tools to understand RNNs — builds toward a deeper understanding of how neural language models perform more involved tasks in NLP, including language modeling or translation.
These tasks cannot be solved by a pure integration mechanism, but it is plausible that integration serves as a useful computational primitive in RNNs more generally, similar to how line attractor dynamics serve as a computational primitive on top of which contextual processing occurs (Maheswaranathan & Sussillo, 2020).
M ETHODS A. F IXED - POINTS AND L INEARIZATION
We study several RNN architectures and we will generically denote their n -dimensional hidden state and d -dimensional input at time t as h t and x t , respectively.
The function that applies hidden state update for these networks will be denoted by F , so that h t =
F ( h t − 1 , x t ) .
The N output logits are a readout of the ﬁnal hidden state, y =
Wh T + b .
We will denote the readout corresponding to the i th neuron by r i , for i = 1 , . . .
, N . We deﬁne a ﬁxed point of the hidden-state space to satisfy the expression h ∗ = F ( h ∗ , x ) . This deﬁnitions of ﬁxed-points is inherently x -dependent.
In this text, we focus on ﬁxed points of the network in for zero input, i.e. when x = 0 . We will also be interested in ﬁnding points in hidden state space that only satisfy this ﬁxed point relation approximately, i.e. h ∗ ≈ F ( h ∗ , x ) . The slowness of the approximate ﬁxed points can be characterized by deﬁning a loss function q := 1 n (cid:107) h − F ( h , x ) (cid:107) 22 . Throughout this text we use the term ﬁxed point to include these approximate ﬁxed points as well.
Expanding around a given hidden state and input, ( h e , x e ) , the ﬁrst-order approximation of F is
[Equation]
where we have deﬁned the recurrent and input Jacobians as J rec ij ( h , x ) :
= ∂F ( h , x ) i ∂h j and J inp ij ( h , x ) :
= ∂F ( h , x ) i ∂x j , respectively.
If we expand about a ﬁxed point h ∗ ≈ F ( h ∗ , x = 0 ) , the effect of an input x t on the hidden state h T ≥ t can be approximated by ( J rec ) T − t J inp x t .
Writing the eigendecomposition,
J rec =
RΛL , with L = R − 1 , we have
[Equation]
where Λ is the diagonal matrix containing the (complex) eigenvalues, λ 1 ≥ λ 2 ≥ · · · ≥ λ n that are sorted in order of decreasing magnitude; r a are the columns of R ; and (cid:96) (cid:62) a are the rows of L .
The magnitude of the eigenvalues of J rec correspond to a time constant τ a = (cid:12)(cid:12)(cid:12) 1 log | λ a | (cid:12)(cid:12)(cid:12) .
The time constants, τ a , approximately determine how long and what information the system remembers from a given input.
We ﬁnd ﬁxed points by minimizing a function which computes the magnitude of the displacement
F ( h , x = 0 ) − h resulting from applying the update rule at point h .
That is, we numerically solve
[Equation]
We seed the minimization procedure with hidden states visited by the network while processing test examples.
To better sample the region, we also add some isotropic Gaussian noise to the initial points.
A. D IMENSIONALITY M EASURES
Here we provide details regarding the measures used to determine both the dimensionality of our hidden-state and ﬁxed-point manifolds.
When we discuss the dimensionality of a set of points, we will mean their intrinsic dimensionality .
Roughly, this is the dimensionality of a manifold that summarizes the discrete data points, accounting for the fact said manifold could be embedded in a higher-dimensional space in a non-linear fashion.
For example, if the discrete points lie along a onedimensional line that is non-linearly embedded in some two-dimensional space, then the measure of intrinsic dimensionality should be close to 1 .
Let X =
{ X 1 , . . . , X M } be the set of M points X I for I = 1 , . .
, M for which we wish to measure the dimensionality.
In this text, X is either a set of hidden-states or a set of ﬁxed points and each
X I ∈ R n is a point in hidden-state space.
To determine an accurate measure of dimensionality, we use the following measures:
• Variance explained threshold .
Let µ 1 ≥ µ 2 ≥ . .
≥ µ n be the eigenvalues from PCA (i.e. the variances) on X .
A simple measure of dimensionality is to threshold the number of PCA dimensions needed to reach a certain percentage of variance explained.
For low number of classes, this threshold can simply be set at ﬁxed values 90% or 95%.
However, we would expect such threshold to breakdown as the number of classes increase, so we also use an N -dependent threshold of N/ ( N + 1) %.
• Global participation ratio .
Again using PCA on X as above, the participation ratio (PR) is deﬁned to be a scalar function of the eigenvalues:
[Equation] Intuitively, this is a scalar measure of the number of “important” PCA dimensions. • Local participation ratio . Since PCA is a linear mapping, both of the above measures will fail if the manifold is highly non-linear.
We thus implement a local PCA as follows: we choose a random point and compute its k nearest neighbors, then perform PCA on this subset of k + 1 points.
We then calculate the participation ratio on the eigenvalues of the local PCA using equation 5.
We repeat the process over several random points, and then average the results.
This measure is dependent upon the hyperparameter k .
• MLE measure of intrinsic dimension (Levina & Bickel, 2005). This is a nearest-neighbor based measure of dimension. For a point X I , let T k ( X I ) be the Euclidean distance to its k th nearest neighbor.
Deﬁne the scalar quantities
[Equation]
This measure is also dependent upon the number of nearest neighbors k .
• Correlation Dimension (Procaccia & Grassberger, 1983; Camastra & Vinciarelli, 2002).
Deﬁne the
[Figure] [Equation]
Then, plotting log C N ( r ) as a function of log r , the correlation dimension is found by estimating the slope of the linear part of the plot.
We plot these dimensionality measures used on synthetic categorical data for class sizes N =to 10 in Figure 6. Despite their simplicity, we ﬁnd the 95% variance explained threshold and the global participation ratio to be the best match to what is theoretically predicted, hence we use these measures in the main text and in what follows. B M ODELS AND T RAINING The three architectures we study are speciﬁed below, with W and b respectively representing trainable weight matrices and bias parameters, and h t denoting the hidden state at timestep t .
All other vectors ( c , g , r , i , f ) represent intermediate quantities; σ ( · ) represents a pointwise sigmoid nonlinearity; and f ( · ) is the tanh nonlinearity.
Update-Gate
[Equation]
Gated Recurrent Unit (GRU)
[Equation]
Long-Short-Term-Memory (LSTM)
[Equation]
With the natural datasets, we form the input vectors x t by using a (learned) 128-dimensional embedding layer.
These UGRNNs and GRUs have hidden-state dimension n = 256 , while in the LSTMs, both the hidden-state (cid:101) h t and the memory (cid:101)
c t are 256-dimensional, yielding a total hidden-state dimension n = 512 .
For the synthetic datasets, due to their small vocabulary size, we simply pass one-hot encoded inputs in the RNN architectures, i.e. we use no embedding layer.
For UGRNNs and GRU, we use a hidden-state dimension of n = 128 , while for LSTMs we again use the same dimension for both (cid:101) h t and (cid:101) c t , resulting in a total hidden-state dimension of n = 256 .
The model’s predictions (logits for each class) are computed by passing the ﬁnal hidden state h T through a linear layer.
In the synthetic experiments, we do not add a bias term to this linear readout layer, chosen for simplicity and ease of interpretation.
We train the networks using the ADAM optimizer (Kingma & Ba, 2014) with an exponentiallydecaying learning rate schedule. We train using cross-entropy loss with added (cid:96) 2 regularization, penalizing the squared (cid:96) 2 norm of the network parameters. Natural experiments use a batch size of 64 with initial learning rate η = 0 . 01 , clipping gradients to a maximum value of 30; the learning rate decays by 0 .
9984 every step.
Synthetic experiments use a batch size of 128, initial learning rate η = 0 . 1 , and a gradient clip of 10 ; the learning rate decays by 0 .
9997 every step.
C N ATURAL D ATASET D ETAILS
We use the following text classiﬁcation datasets in this study:
• The Yelp reviews dataset (Zhang et al., 2015) consists of Yelp reviews, labeled by the corresponding star rating (1 through 5).
Each of the ﬁve classes features 130,000 training examples and 10,000 test examples.
The mean length of a review is 143 words.
• The Amazon reviews dataset (Zhang et al., 2015) consists of reviews of products bought on Amazon.com over an 18-year period.
As with the Yelp dataset, these reviews are labeled by the corresponding star rating (1 through 5).
Each of the ﬁve classes features 600, training examples and 130,000 test examples.
The mean length of a review is 86 words.
• The DBPedia ontology dataset (Zhang et al., 2015) consists of titles and abstracts of Wikipedia articles in one of 14 non-overlapping categories, from DBPedia 2014.
Categories include: company, educational institution, artist, athlete, ofﬁce holder, mean of transportation, building, natural place, village, animal, plant, album, ﬁlm, and written work.
Each class contains 40,000 training examples and 5,000 testing examples.
We use the abstract only for classiﬁcation; mean abstract length is 56 words.
• The AG’s news corpus (Zhang et al., 2015) contains titles and descriptions of news articles from the web, in the categories: world, sports, business, sci/tech.
Each category features 30,000 training examples and 1,900 testing examples.
We use only the descriptions for classiﬁcation; the mean length of a description is 35 words.
• The GoEmotions dataset (Demszky et al., 2020) contains text from 58,000 Reddit comments collected between 2005 and 2019.
These comments are labeled with the followingemotions: admiration, approval, annoyance, gratitude, disapproval, amusement, curiosity, love, optimism, disappointment, joy, realization, anger, sadness, confusion, caring, excitement, surprise, disgust, desire, fear, remorse, embarrassment, nervousness, pride, relief, grief.
The mean length of a comment is 16 words.
Two main characteristics distinguish these datasets: (i) whether there is a notion of order among the class labels, and (ii) whether labels are exclusive.
The reviews datasets, Amazon and Yelp, are naturally ordered, while the labels in the other datasets are unordered.
All of the datasets besides
GoEmotions feature exclusive labels; only in GoEmotions can two or more labels (e.g., the emotions anger and disappointment ) characterize the same example.
In addition to the standard ﬁve-class versions of the ordered datasets, we form three-class subsets by collecting reviews with 1, 3, andstars (excluding reviews with 2 and 4 stars).
We build a vocabulary for each dataset by converting all characters to lowercase and extracting the 32,768 most common words in the training corpus.
Tokenization is done by TensorFlow TF.Text
WordpieceTokenizer.
D S YNTHETIC D ATASET D ETAILS
In this appendix we provide many additional details and results from our synthetic datasets.
Although these datasets represent signiﬁcantly simpliﬁed settings compared to their realistic counterparts, often the results from training RNNs on the synthetic and natural datasets are strikingly similar.
D. C ATEGORICAL D ATASET
For the categorical synthetic dataset used in Section 3.1, we generate phrases of L words, drawing from a word bank consisting of N + 1 words,
W = { evid 1 , . .
, evid N , neutral } .
Each word
W ∈ W has an N -dimensional vector of integers associated with it, w W = { w W 1 , . . .
, w WN } with w Wi ∈ Z for all i = 1 , . . .
, N .
The word “evid i ” has score deﬁned by w evid
i
{+i = 1 and w evid i j = 
+}
for i (cid:54) =
j .
Meanwhile, the word “neutral” has w neutral j = 0 for all j .
Additionally, each phrase has a corresponding score s that also takes the form of an N -dimensional vector of integers, s = { s 1 , . . .
, s N } .
A phrase’s score is equal to the sum of scores of the words contained in said phrase, s = (cid:80) W ∈ phrase w W .
The phrase is then assigned a label corresponding to the class with the maximum score, y = argmax ( s ) .
In the main text we analyze synthetic datasets where phrases are drawn from a uniform distribution over all possible scores, s .
To do so, we enumerate all possible scores a phrase of length L can produce as well as all possible word combinations that can generate a given score.
It is also possible to build phrases by drawing each word from a uniform distribution over all words in W .
In practice, we ﬁnd all results on synthetic datasets have minor quantitative differences when comparing these two methods, but qualitatively the results are the same.
As highlighted in the main text, after training on this synthetic data we ﬁnd the explored hiddenstate space to resemble a regular ( N − 1) -simplex.
This holds for a large range (cid:96) 2 values relative to the natural datasets.
In Figure 7, we plot the (global) participation ratio, deﬁned in equation 5, as a function of the number of classes, N .
In addition to the hidden states forming a simplex, we observe the N readout vectors are approximately equal magnitude and are aligned along the N vertices of said ( N − 1) -simplex.
In Figure 8, we plot several measures on the readout vectors that support this claim that we now discuss.
We ﬁnd the readout vectors to have very close to the same magnitude (Fig. 8, left panel).
The angle (in degrees) between a pair of vectors that point from the center of a ( N − 1) -simplex to two of its vertices is
[Equation]
For example, for a regular 2 -simplex, i.e. an equilateral triangle, this predicts an angle between readout vectors of 120 degrees.
The distribution of pairwise angles between readout vectors is plotted the center panel of Figure 8.
Lastly, if the readouts lie entirely within the ( N − 1) -simplex, all N of them should live in the same R N − 1 subspace. To measure this, deﬁne r (cid:48) i to be projection of r i into the subspace formed by the other N − 1 readout vectors, i.e. the span of the set { r j | j = 1 , . . . , N ; j (cid:54) = i } . We then deﬁne the subspace percentage, Λ , as follows,
[Equation]
If all the readouts lie within the same R N − 1 subspace, then Λ = 1 . The right panel of Figureshows that in practice Λ ≈ 1 for the synthetic data with (cid:96) 2 regularization parameter of 5 × 10 − 4 . Why a Regular ( N − 1) -Simplex?
Here we propose an intuitive scenario that leads the network’s hidden states to form a regular ( N − 1) -simplex. To classify a given phrase correctly, the network must learn to keep track of the value of the N -dimensional score vector s . One way this can be done is follows: Let the network’s hidden state live in some R N dimensional subspace. Within this subspace, let the N readout vectors be orthogonal and have equal magnitude.
Furthermore, deﬁne a Cartesian coordinate system to have basis vectors aligned with the N readouts, with z i the coordinate along the direction of readout r i .
Then, the coordinates within this subspace encodes the components of the N -dimensional score vector s : the evidence word ‘evid i moves you along the coordinate direction i some ﬁxed amount and so s i ∝ z i . Note the subspace of R N explored by hidden states has a ﬁnite extent, since the phrases are of ﬁnite length. This subspace can be further subdivided into regions corresponding to different class labels: if z i > z j for all j (cid:54) =
i then h · r i > h · r j and the phrase is classiﬁed as Class i . The left panel of Figure 9 shows an example of the 3 -dimensional subspace for N = 3 . The important step that gets us from a subspace of R N to the regular ( N − 1) -simplex is the presence of the softmax layer used when calculating loss.
Since this function normalizes the scores, it is only the relative size of the components of s that matters.
Removing the dependence on the absolute score values corresponds to projecting onto the R N − 1 subspace orthogonal to the N -dimensional ones vector, (1 , 1 , . . This projection results in an ( N − 1) -simplex with the readouts aligned with the vertices. A demonstration of this procedure for N = 3 is shown in Figure 9. D. O RDERED D ATASET
As alluded to in the main text, we try two renditions of ordered synthetic data. The details of both are given below. The ﬁrst relies on a ground truth of only a sentiment score, while the second classiﬁes based on both sentiment and neutrality. Although the ﬁrst is simpler and still bears many resemblances to natural data (i.e. Yelp and Amazon), we ﬁnd the second to be a better match overall.
Sentiment Only Synthetic Data
The ﬁrst synthetic data for ordered datasets is very similar into that of the categorical sets with a minor difference in the word bank and how phrases are assigned labels.
For N -class ordered datasets, the word bank always consists of only three words W = { good , bad , neutral } .
We now take the word and phrase scores to be 1-dimensional and w good= +1 , w bad 1 = − 1 , and w neutral= 0 .
We then subdivide the range of possible scores s into N equal regions, and a phrase is labeled by the region which its score fall into.
Given the above deﬁnitions, the range of scores is [ − L, L ] , and so for the N = 3 , with labels { Positive , Negative , Neutral } , we deﬁne some threshold s N = L/ 3 .
Then a label y is assigned as follows:
[Equation]
[Equation]
Meanwhile, for N = 5 , one could draw the region divisions at the score values {− 3 L/ 5 , − L/ 5 , L/ 5 , 3 L/ 5 } .
Similar to the categorical data above, in the main text we draw phrases from a uniform distribution over all possible scores.
The N = 2 case corresponds to sentiment analysis, and its hidden state space, word deﬂections, and ﬁxed point manifold are plotted in Figure 10.
The dynamics of this system qualitatively match the natural dataset analyzed in Maheswaranathan et al.
Brieﬂy, the sentiment score is encoded in the hidden state’s position along a one-dimensional line, aligned with the readouts which point in opposite directions.
The word ‘good‘ (’bad’) moves you along this line along the ’Positive’ (’Negative’) readout, increasing the corresponding logit value.
The simplest ordered dataset beyond binary sentiment analysis is that of N = 3 , and a plot showing the ﬁnal hidden states, deﬂections, and ﬁxed-point manifold is shown in top row of Figure 11.
In the bottom row, we show the same plots for N = 5 .
In both cases, the hidden-state trajectories move away from h 0 onto a curve embedded in 2d plane, with the curve bent around the origin of said plane.
The N readout vectors are evenly fanned out in the 2d plane, which subdivides the curve into N regions corresponding to each of the N {+classes. The curve subdivisions reﬂect the ordering of the score subdivisions, for N = 
 we see ‘Neutral’ lying in between ‘Positive’ and ‘Negative’ and for+}
N = 5 the stars are ordered from 1 to 5.
In contrast to categorical data, the word deﬂection ∆ h t are highly varied and have a strong dependence on a state’s location in hidden-state space.
On average, the words ‘good’ and ‘bad’ move the hidden state further left/right along the curve.
Although ∆ h t for the word ‘neutral’ is on average smaller, it tends to move the hidden state along the ‘Neutral’ or ‘3 Star‘ readout.
These dynamics are how the network encodes the relative count of ‘good’ and ‘bad’ words in a phrase that ultimately determines the phrase’s classiﬁcation.
We show the ﬁxed points in the far right panel of Figure 11.
For N = 3 , the ﬁxed point manifold mostly resembles that of a one-dimensional bent line attractor, with a small region that is two-dimensional along the ‘Neutral‘ readout.
For N = 5 , the ﬁxed point manifold is much more planar.
Thus, the N = 3 case exhibits very similar dynamics to that of the line attractor studied in Maheswaranathan et al.
(2019), the attractor is now simply subdivided into three regions due to the readout vector alignments.
Sentiment and Neutrality Synthetic Data Instead of classifying a phrase based off a single sentiment score, our second ordered synthetic model classiﬁes a phrase based off of two scores that track the sentiment and intensity of a given phrase.
We draw from an enhanced word bank consisting of
W = { awesome , good , okay , bad , awful , neutral } .
We take the two-dimensional word score to have components corresponding to ( sentiment , intensity ) where positive (negative) sentiment scores correspond to positive (negative) sentiment and positive (negative) intensity scores correspond to high
(low) emotion.
The word score values we use are
[Equation]
As with the other synthetic models, we sum all word scores across a phrase to arrive at a phrase’s sentiment and intensity score, ( s, i ) .
We then assign the phrase a label y based off the following criterion:
[Equation]
Thus we see that scores with negative (low) intensity where the intensity magnitude is greater than the sentiment magnitude are classiﬁed as ‘Three Star’, i.e. it is a neutral phrase. Otherwise, phrases with low intensity that are the less extreme reviews are classiﬁed as either ‘Two Star’ or ‘Four Star’ based on their sentiment. Finally, phrases with high intensity are labeled either ‘One Star’ or ‘Five Star’, again based on their sentiment.
D. M ULTI -L ABELED D ATASET
Here we provide details of the synthetic multi-labeled dataset, that corresponds to natural dataset GoEmotions in Section 3.3 of the main text. Let us introduce this by taking the N = 2 as an explicit example, where each phrase can have up to two labels. We draw from a word bank consisting of
W = { good 1 , bad 1 , good 2 , bad 2 , neutral } , where
[Equation]
We then classify each phrase with two labels, individually based on the score vector components sand s 2 . Namely, [Equation] Thus there are four possible combinations of labels.
For this synthetic datasets, we generate phrases by uniformly drawing words one-by-one from W .
Generalization of the above construction to an arbitrary number of possible labels N is straightforward: one simply adds additional N -dimension score vectors w good i and w bad
i for each possible label i = 1 , . . .
, N and then uses the N components of the score to assign the N labels, y i , individually.
The results after training a network on the N = 2 dataset are shown in the main text in Figure 5, and results for N
= 3 are shown in Figure 12.
Again, we see the explored hidden-state space to be low-dimensional, but notably it now resembles a three-dimensional cube.
This is certainly a large departure from the N = 8 categorical dataset, from which we expect a (seven-dimensional) regular 7 -simplex.
Instead, what we see here is the “outer product” of three N = 2 ordered datasets.
That is, we expect a single N = 2 ordered dataset (i.e. binary sentiment analysis) to have a hidden-state space that resembles a line attractor.
As one might expect, tasking the network with analyzing three such sentiments at once leads to three line attractors that are orthogonal to one another, forming a cube.
This is supported in the center panel of Figure 5, where we see the various sentiment evidences are orthogonal from one another.
E A DDITIONAL RESULTS ON NATURAL DATASETS E. AG N EWS
This subsection contains two ﬁgures: Figures 13 and 14 complement Figure 1 in the main text; the main text ﬁgure showed the manifolds learned by an LSTM on both 3- and 4-class AG News datasets; the ﬁgures in this appendix show corresponding manifolds learned by a GRU and UGRNN.
E. DBP EDIA 3- CLASS AND 4- CLASS CATEGORICAL PREDICTION
Like AG News, DBPedia Ontology is a categorical classiﬁcation dataset.
We show results for networks trained on 3- and 4-class subsets of this dataset in Figures 15, 16, and 17. E. Y ELP 5- CLASS STAR PREDICTION Figures 19, 18, and 20 show the ﬁxed-point manifolds associated with a GRU, LSTM and UGRNN, respectively, trained on 5-class and 3-class Yelp dataset. These reviews are naturally ﬁve star; we create a 3-class subset by removing examples labeled with 2 and 4 stars.
These ﬁgures complement Figure 1 in the main text. E. A MAZON 5- CLASS AND 3- CLASS STAR PREDICTION As another example of an ordered dataset, Figures 21, 22, and 23 show results for networks trained on a 3-class and 5-class subsets of Amazon reviews.
These reviews are naturally ﬁve star; we create a 3-class subset by removing examples labeled with 2 and 4 stars. E. 3- CLASS G O E MOTIONS In addition to the 2 class variant presented in the main text, we also trained a 3 class version of the
GoEmotions dataset.
We ﬁltered the dataset to just include the following three classes: “admiration”, “approval”, and “annoyance” (these were selected as they were the classes with the largest number of examples). These results are presented in Figure 24.
For this network, despite having three classes, we ﬁnd that the ﬁxed points are largely two dimensional (Fig. 24a). The timescales of the eigenvalues of the Jacobian computed at these ﬁxed points have two slow modes (Fig. 24b), which overlap with the two modes (Fig. 24c); thus we have a roughly 2D plane attractor.
However, the participation ratio (Fig. 24d) indicates that the dimensionality of this attractor is slightly higher than the 2D case shown in Fig. We suspect that these differences are due to the strong degree of class imbalance present in the GoEmotions dataset. There are very few examples with multiple labels, for any particular combination of labels.
In synthetic multi-labeled data (which is class balanced), we see much clearer 3D structure when training a 3 class network (Fig. 12). F T HE EFFECT OF (cid:96) 2 REGULARIZATION : COLLAPSE , CONTEXT , AND CORRELATIONS Regularizing the parameters of the network during training can have a strong effect on the dimension of the resulting dynamics.
We describe this effect ﬁrst for the datasets with ordered labels, Yelp and Amazon reviews. We penalize the squared (cid:96) 2 -norm of the parameters, adding the term λ
|| θ
|| 22 to the cross-entropy prediction loss; λ is the (cid:96) 2 penalty and θ are the network parameters. Collapse : Figure 25 shows the performance of the LSTM, GRU, and UGRNN as a function of the (cid:96) 2 penalty. As the (cid:96) 2 penalty is varied, the test accuracy usually decreases gradually; however, at a few values, the accuracy takes a large hit. The ﬁrst two of these jumps correspond to a decrease in the dimension of the integration manifold from 2D and 1D and then 1D to 0D.
The resulting 1D manifold is shown, for the example of a GRU on the Amazon dataset in Figures 26. The effects of collapse on the other architectures for the ordered datasets are identical. [Figure] [Figure]
v e r l ap [Figure] [Figure] P C #[-−  2 ΔPC # −   Δ P C #  
-]
[Figure] [Figure]
V a r .
E x p . [Figure]
When the regularization is sufﬁcient to collapse the manifold to a 1D line, the dynamics are quite similar to the 1D line attractors studied in Maheswaranathan et al. A single accumulated valence score is tracked by the network as it moves along the line; this tracking occurs via a single eigenmode with a time constant comparable to the average document length, aligned with the ﬁxedpoint manifold.
The difference between the binary- and 5-class line-attractor networks are largely in the way the ﬁnal states are classiﬁed; in the 5-class case, the line attractor is divided into sections based largely on the angle the line makes with the readout vector of each class.
The collapse to a 0D manifold with a higher (cid:96) 2 penalty is most strikingly seen in the recurrent
Jacobian spectra at the ﬁxed points (Figure 27).
Here there are no modes which remember activity on the timescale of the mean document length. Given this lack of integration, it is unclear how these networks are achieving accuracies above random chance. Context : While the focus of this study has been on how networks perform integration, it is clear from the plots in Figure 25 that the best-performing models are doing more than just bag-of-words style integration. When the order of words in the sentence is shufﬂed, these models take a hit in accuracy.
Interestingly, when the (cid:96) 2 coefﬁcient is increased from the smallest values we use, the contextual effects are the ﬁrst to be lost: the model’s accuracy on shufﬂed and ordered examples becomes the same. Understanding precisely how contextual processing is carried out by the network is an interesting direction for future work. It is important to show, however, that the basic two-dimensional integration mechanism we have presented in the main text still underlies the dynamics of the networks which are capable of handling context. To show this, we plot in Figure 28 the ﬁxed point manifold, colored by the predicted class.
As with the models which are not order-sensitive, the classiﬁcation of the ﬁxed points depends largely on their top two coordinates (after PCA projection). This is the case even though the PCA explained variance clearly shows extension of the dynamics into higher dimensions. It is thus likely that, similarly to how Maheswaranathan & Sussillo (2020) found that the contextual-processing mechanism was a perturbation on top of the integration dynamics for binary sentiment classiﬁcation, the same is true for more ﬁnely-grained sentiment classiﬁcation. Correlations : As might be expected, increasing (cid:96) 2 regularization also causes collapse in models trained on categorical classiﬁcation tasks.
For example, as shown in Figure 29, the tetrahedral manifold seen in 4-class AG News networks becomes a square at higher values of (cid:96) 2 , collapsing from three dimensions to two. That is, instead of class labels corresponding to vertices of a tetrahedron, when the (cid:96) 2 regularization is increased, these labels correspond to the vertices of a square. Interestingly, in the the collapse to a square, we ﬁnd that — regardless of architecture and across 10 random seeds per architecture — the ordering of vertices around the square appears to reﬂect correlations between classes. Up to symmetries, the only possible ordering of vertices around the square are: (i) World → Sci/Tech → Business → Sports, (ii) World → Sci/Tech → Sports →
Business, and (iii) World → Sports → Sci/Tech → Business. In practice, we observe that most of the time (26 out of 30 trials), order (iii) appears; otherwise, order (i) appears. We never observe order (ii). To show how this ordering arises from correlations between class labels, we train a bag-of-words model on the full 4-class dataset.
Taking the most common 5000 words in the vocabulary, we plot, in Figure 30, the changes in each logit due to these words. As the ﬁgure shows, for most pairs of classes there is a weak negative correlation between the evidence for the pair.
However, between the classes “Sports” and “Business”, there is a strong negative correlation ( R =-0.81); between “Sports” and “Sci/Tech”, there is a slightly weaker negative correlation ( R =-0.61).
Stated another way, words which constitute positive evidence for “Sports” are likely to constitute negative evidence for “Business” and/or “Sci/Tech”. This matches with the geometries we observe in practice, where “Sports” and “Business” readouts are ‘repelled’ most often, and otherwise “Sports” and “Sci/Tech” are repelled. G A CLOSER LOOK AT THE SLOW ZONE In the main text, we were interested in approximate ﬁxed points , or points which were slow on the timescale given roughly by the average length of documents T av in the training dataset.
For practical purposes, points which are slow on this time scale can be treated as effectively ﬁxed, since evolving the system for T av will result in little motion. Whether any of the points in the slow zone are exact ﬁxed points of the dynamics, or whether all of them are simply slow points, is not something our numerical experiments are capable of resolving. However, we do ﬁnd that there is structure to the slow zone in that all of the points are not uniformly slow. We deﬁne the speed of a point, S ( h ) , as the distance traveled from that point after a single application of the dynamical map F ( h , 0 ) (Sussillo & Barak, 2013),
[Equation] Note that, as in the main text, we are characterizing the RNN with zero input (the autonomous system). In Figure 31, we plot three orthogonal slices of this function for a GRU network trained on the 3-class AG News dataset. All the points in the roughly triangular region we identiﬁed in the main text have speeds less than a tenth of the inverse document length, and are thus slow.
However, there are a few neighborhoods, roughly located near the vertices of the triangle that are even slower. This structure can be seen further in Figure 32, in which we plot the speed as a function of the PC 0 coordinate for three values of the PC 1 coordinate. Similar structures can be seen in ﬁxed point manifolds for other networks and datasets. Fully exploring these structures and understanding how they arise from the equations governing the RNN is an interesting direction for future research.