Original: M APPING THE T IMESCALE O RGANIZATION OF N EURAL L ANGUAGE M ODELS
Revised: M APPING THE T IMESCALE O RGANIZATION OF N EURAL L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contextual information over longer timescales.
Revised: In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contextual information over longer timescales.

Original: In contrast, in recurrent neural networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized.
Revised: In contrast, in recurrent neural networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized.

Original: Therefore, we applied tools developed in neuroscience to map the “processing timescales” of individual units within a word-level LSTM language model.
Revised: Therefore, we applied tools developed in neuroscience to map the “processing timescales” of individual units within a word-level LSTM language model.

Original: This timescalemapping method recovered a small set of units previously known to track longrangesequence dependencies, such as noun number or syntactic features, and revealed a new cluster of previously unreported long-timescale units.
Revised: This timescalemapping method assigned long timescales to units previously found to track longrange syntactic dependencies, and revealed a new cluster of previously unreported long-timescale units.

Original: Next, we explored the functional role of units by examining the relationship between their processing timescales and network connectivity.
Revised: Next, we explored the functional role of units by examining the relationship between their processing timescales and network connectivity.

Original: We identiﬁed two classes of long-timescale units. “Controller” units composed a densely interconnected subnetwork and strongly projected to the forget and input gates of the rest of the network. “Integrator” units  expressed connection proﬁles closer to the mean connection proﬁle and produced few strong projections.
Revised: We identiﬁed two classes of long-timescale units: “Controller” units composed a densely interconnected subnetwork and strongly projected to the forget and input gates of the rest of the network, while “Integrator” units showed the longest timescales in the network, and expressed projection proﬁles closer to the mean projection proﬁle. Ablating integrator and controller units affected model performance at different position of a sentence, suggesting distinctive functions of these two sets of units.

Original: Finally, we tested the generalization of these resultsfrom word-level models to  character-level models.
Revised: Finally, we tested the generalization of these results to a character-level LSTM model.

Original: In summary, we demonstrated a model-free technique for mapping the timescale organization in neural network models, and we applied this method to reveal a hierarchical timescale  organization within LSTM language models, including a ”core” of units with strong gating projections and long processing timescales.
Revised: In summary, we demonstrated a model-free technique for mapping the timescale organization in neural network models, and we applied this method to reveal the timescale and functional organization of LSTM language models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Language processing requires tracking information over multiple timescales: to be able to predict the ﬁnal word ”timescales” in the previous sentence, one must consider both the short-range context (e.g. the adjective ”multiple”) and the long-range context (e.g. the subject ”language processing”) .
Revised: Language processing requires tracking information over multiple timescales. To be able to predict the ﬁnal word “timescales” in the previous sentence, one must consider both the short-range context (e.g. the adjective “multiple”) and the long-range context (e.g. the subject “language processing”).

Original: How do humans and neural language models encode such multi-scale context information?
Revised: How do humans and neural language models encode such multi-scale context information?

Original: Neuroscientists have developed methods to study how the human brain encodes information over multiple timescales during sequence processing.
Revised: Neuroscientists have developed methods to study how the human brain encodes information over multiple timescales during sequence processing.

Original: By parametrically varying the timescale of intact context, and measuring changes in the neural response, a series of studies Lerner et al. Xu et al. ; Honey et al. (2012) showed that higher-order regions are more sensitive to long-range context change than lower-order sensory regions.
Revised: By parametrically varying the timescale of intact context, and measuring changes in the neural response, a series of studies (Lerner et al., 2011; Xu et al., 2005; Honey et al., 2012) showed that higher-order regions are more sensitive to long-range context change than lower-order sensory regions.

Original: These studies indicate the existence of a ”hierarchy of processing timescales” in the human brain.
Revised: These studies indicate the existence of a “hierarchy of processing timescales” in the human brain.

Original: More recently, Chien & Honey (2020) used a timeresolved method to investigate how the brain builds shared context, when two groups of people are processing the same sentence preceded by different contexts.
Revised: More recently, Chien & Honey (2020) used a time-resolved method to investigate how the brain builds shared context, when two groups of people processed the same sentence preceded by different contexts.

Original: By directly mapping the time required for individual brain regions to converge on a shared representation in response to shared input, they again found that higher-order regions take longertime to construct this ”shared context”.
Revised: By directly mapping the time required for individual brain regions to converge on a shared representation in response to shared input, they conﬁrmed that higher-order regions take longer to converge to a shared context representation.

Original: Altogether, these and other lines of investigation suggest that sequence processing in the brain is supported by a distributed and hierarchical structure: sensory regions have short processing timescales and are primarily inﬂuenced by the current input and its short-range context, while higher-order cortical regions have longer timescales and track longer-range dependencies (Hasson et al., 2015; Honey et al., 2012; Chien & Honey, 2020; Lerner et al., 2011; Baldassano et al., 2017; Runyan et al., 2017;
Revised: Altogether, these and other lines of investigation suggest that sequence processing in the brain is supported by a distributed and hierarchical structure: sensory regions have short processing timescales and are primarily inﬂuenced by the current input and its short-range context, while higher-order cortical regions have longer timescales and track longer-range dependencies (Hasson et al., 2015; Honey et al., 2012; Chien & Honey, 2020; Lerner et al., 2011; Baldassano et al., 2017; Runyan et al., 2017;

Original: Fuster, 1997).
Revised: Fuster, 1997).

Original: How are processing timescales organized within recurrent neural networks (RNNs) trained to perform natural language processing?
Revised: How are processing timescales organized within recurrent neural networks (RNNs) trained to perform natural language processing?

Original: Long short-term memory networks (LSTMs) (Hochreiter &
Revised: Long short-term memory networks (LSTMs) (Hochreiter &

Original: Schmidhuber, 1997) have been widely investigated in terms of their ability to successfully solve sequential prediction tasks.
Revised: Schmidhuber, 1997) have been widely investigated in terms of their ability to successfully solve sequential prediction tasks.

Original: However, long-range dependencies have usually been studied with respect to a particular linguistic function (e.g. subject-verb number agreement (Linzen et al., 2016; Gulordava et al., 2018; Lakretz et al., 2019), and there has been less attention on the broader question of how sensitivity to prior context – broadly construed – is functionally organized within these recurrent networks.
Revised: However, long-range dependencies have usually been studied with respect to a particular linguistic function (e.g. subject-verb number agreement (Linzen et al., 2016; Gulordava et al., 2018; Lakretz et al., 2019), and there has been less attention on the broader question of how sensitivity to prior context – broadly construed – is functionally organized within these recurrent networks.

Original: Therefore, drawing on prior work in the neuroscience literature,  we demonstrate a model-free approach to mapping processing timescale in recurrent neural networks.
Revised: Therefore, drawing on prior work in the neuroscience literature, here we demonstrate a model-free approach to mapping processing timescale in recurrent neural networks.

Original: We focus on existing language models that were trained to predict upcoming tokens at the word level Gulordava et al. (2018) and at the character level Hahn & Baroni (2019).
Revised: We focused on existing language models that were trained to predict upcoming tokens at the word level (Gulordava et al., 2018) and at the character level (Hahn & Baroni, 2019).

Original: We found that the higher layers of LSTM language models contained a small subset of units which exhibit long-range sequence dependencies; this subset includes previously reported units (e.g. a ”syntax” unit (Lakretz et al., 2019)) as well as previously unreported units.
Revised: The timescale organization of these two models both revealed that the higher layers of LSTM language models contained a small subset of units which exhibit long-range sequence dependencies; this subset includes previously reported units (e.g. a “syntax” unit, Lakretz et al., 2019) as well as previously unreported units.

Original: After mapping the timescale organization, we asked: how does processing timescale relate to the functional role of each unit in the network?
Revised: After mapping the timescales of individual units, we asked: do the processing timescales of each unit in the network relate to its functional role, as measured by its connectivity?

Revised: The question is motivated by neuroscience studies which have shown that in the human brain, higher-degree nodes tend to exhibit slower dynamics and longer context dependence than lower-degree nodes (Baria et al., 2013).

Revised: More generally, the primate brain exhibits a core periphery structure in which a relatively small number of “higher order” and high-degree regions (in the prefrontal cortex, in default-mode regions and in so-called “limbic” zones) maintain a large number of connections with one another, and exert a powerful inﬂuence over large-scale cortical dynamics (Hagmann et al., 2008; Mesulam, 1998; Gu et al., 2015).

Original: Using a  network-theoretic approach, we distinguished two types of longtimescale units: (1) ”controller” units which have distinct and strong projections toaffect the gating behavior of other units and (2) ”integrator” units which have few strong projections and whose connection patterns are closer to the mean pattern.
Revised: Inspired by the relationships between timescales and network structure in neuroscience, we set out to test corresponding hypotheses in neural nets: (1) Do units with longertimescales tend to have higher degree in neural language models? and (2) Do neural language models also exhibit a “core network” composed of functionally inﬂuential high-degree units? Using an exploratory network-theoretic approach, we found that units with longer timescales tend to have more projections to other units. Furthermore, we identiﬁed a set of medium-to-long timescale “controller” units which exhibit distinct and strong projections to the gating nodes that control the state of other units, and a set of long-timescale “integrator units” which showed inﬂuence on predicting words where the long context is relevant.

Original: These ﬁndings advance our understanding of the timescale distribution and functional organization of LSTM language models, and provide a method for identifying the subset of important nodes for representingand manipulating long-range contextual information.
Revised: In summary, these ﬁndings advance our understanding of the timescale distribution and functional organization of LSTM language models, and provide a method for identifying the subset of important nodes for representing long-range contextual information.

Original:  R ELATED W ORK Linguistic Context in LSTMs How do LSTMs encode linguistic context at multiple timescales?
Revised:  R ELATED W ORK Linguistic Context in LSTMs How do LSTMs encode linguistic context at multiple timescales?

Original: Although LSTMs are usually not designed with an explicit multi-scale architecture as observed in the human brain, it is possible that long-range and short-range information may be encoded in distinct subsets of LSTM cell state units.

Original: Prior work, however, suggested that the units sensitive to longrange dependencies  are sparse.
Revised: Prior work suggested that the units sensitive to information that requires long-range dependencies (such as number agreement) are sparse.

Original: By ablating one unit at a time, Lakretz et al.
Revised: By ablating one unit at a time, Lakretz et al.

Original: (2019) found two units that encode information required for processing long-range subject-verb number agreement (one for singular and one for plural, respectively), and several other units for short-range number agreement.
Revised: (2019) found two units that encode information required for processing long-range subject-verb number agreement (one for singular and one for plural information encoding). They further identiﬁed several long-range “syntax units” whose activation was associated with syntactic tree-depth. Overall,

Original: They further identiﬁed a ”syntax unit”, whose activation tracks the subject and its corresponding verb, regardless of the intervening distance; this ”syntax” unit was signiﬁcantly connected to both of the long-range number units, suggesting that information from the ”syntax” unit may support the number units in their functions.
Revised: Lakretz et al.

Original: Overall, prior work suggests that a small subset of units track long-range dependencies.
Revised: (2019) suggests that a sparse subset of units tracks long-range dependencies related to subject-verb agreement and syntax.

Original: If  there are very few such dependency-tracking nodes, this may limit the capacity of the models to process long sentences with high complexity, for reasons similar to those thought to limit human sentence processing (Lakretz et al., 2020). 
Revised: If this pattern is general – i.e. if there are very few nodes tracking long-range dependencies in general – this may limit the capacity of the models to process long sentences with high complexity, for reasons similar to those that may limit human sentence processing (Lakretz et al., 2020). To test whether long-range nodes are sparse in general, we require a model-free approach for mapping the context dependencies of every unit in the language network.

Original: Whole-network context dependence .
Revised: Whole-network context dependence .

Original: Previous work by Khandelwal et al.
Revised: Previous work by Khandelwal et al.

Original: (2018) investigated the duration of prior context that LSTM language models use to achieve word prediction.
Revised: (2018) investigated the duration of prior context that LSTM language models use to support word prediction.

Original: Contextdependence was measured by permuting the order of words preceding the preserved context, and observing the increase in model perplexity when the preserved context gets shorter.
Revised: Contextdependence was measured by permuting the order of words preceding the preserved context, and observing the increase in model perplexity when the preserved context gets shorter.

Original: Khandelwal et al.
Revised: Khandelwal et al.

Original: (2018) found that up to 200 word-tokens of prior context were relevant to the model perplexity, but that the precise ordering of words only mattered within the most recent 50 tokens.
Revised: (2018) found that up to 200 word-tokens of prior context were relevant to the model perplexity, but that the precise ordering of words only mattered within the most recent 50 tokens.

Original: The context-permutation method employed in this study was analogous to the approach used to measure context-dependence in human brain responses to visual movies (Hasson et al., 2008) and to auditory language Lerner et al.
Revised: The context-permutation method employed in this study was analogous to the approach used to measure context-dependence in human brain responses to visual movies (Hasson et al., 2008) and to auditory language Lerner et al.

Original: In the present study, we take a slightly different approach: Instead of entirely permuting the prior context (and preserve a variety of scales of context), we substitute alternative contexts (which is, by itself, a meaningful sentence), and we measure the time-taken for the effect of this prior context to ”decay” (see Methods). In addition, we set out to measure the context-dependence at the level of single units in the model, rather than at the level of overall model perplexity.
Revised: Inspired by the ﬁndings of (Khandelwal et al., 2018) and (Lakretz et al., 2019), in the present study we set out to map the context-dependence across all of the individual units in the LSTM model.

Original: Here, we build on these prior investigations of context-dependence by: (i) using a model-free approach to map multiple types of short- and long-range context-dependence (Chien & Honey, 2020); (ii) examining theprocessing timescales of individual units and (iii) by examining the relationship between processing timescales and the network connectivity of each unit. These investigations aim toward the ultimate goal of developing model-free tools for mapping theﬂow of information from prior context to present processing within neural language models .
Revised: This enabled us to relate the timescales to the effects of node-speciﬁc ablation and the network architecture itself. In addition, our context manipulations included both context-swapping (substituting alternative meaningful contexts) and context-shufﬂing (permuting the words in the prior context to disrupt inter-word structure), which allowed us to better understand how individual words and syntactically structured word-sequences contribute to the the context representation of individual hidden units.

Original:  M ETHODS 3. L ANGUAGE M ODELS AND C ORPUS
Revised:  M ETHODS 3. L ANGUAGE M ODELS AND C ORPUS

Original: We evaluated the internal representations generated by a pre-trained word-level LSTM language model (WLSTM Gulordava et al. (2018)) as well as a pre-trained character-level LSTM model (CLSTM, Hahn & Baroni (2019))
Revised: We evaluated the internal representations generated by a pre-trained word-level LSTM language model (WLSTM Gulordava et al., 2018) as well as a pre-trained character-level LSTM model (CLSTM, Hahn & Baroni, 2019) as they processed sentences sampled from the 427804word/1965719-character novel corpus: Anna Karenina by Leo Tolstoy 2 3 .

Original: as they processed sentences sampled from the 427804word/1965719-character novel corpus: Anna Karenina by Leo Tolstoy 1 2 .
Revised: Constance Garnett

Original:  For WLSTM, we use the model made available by Gulordava et al.
Revised: retrieved from http://www For WLSTM, we used the model made available by Gulordava et al.

Original: The model has an 650-dimensional embedding layer, two 650-dimensional hidden layers and an output layer with vocabulary size 5,0000.
Revised: The model has a 650-dimensional embedding layer, two 650-dimensional hidden layers and an output layer with vocabulary size 50,000.

Original: The model was trained  on Wikipedia data, without ﬁne-tuning to the novel.
Revised: The model was trained and tested on Wikipedia sentences and was not ﬁnetuned to the novel corpus.

Original: Considering that the model was trained and tested on Wikipedia sentences and may not be familiar with some sentences in the novel, we selected sentences with low perplexity comparable to sentences with similar structure sampled in the testing data in  Gulordava et al. 
Revised: Therefore, we only used sentences with low perplexity from the novel in our main timescale analysis. We did the same analysis using the Wikipedia test set from Gulordava et al. and yielded similar results (See Section 5.3, Figure A.4A, Appendix A.2.1).

Original: For CLSTM, we use the model made available by Hahn & Baroni (2019).
Revised: For CLSTM, we used the model made available by Hahn & Baroni (2019).

Original: The model has an 200dimensional embedding layer, three 1024-dimensional hidden layers and an output layer with vocabulary size 63.
Revised: The model has a 200dimensional embedding layer, three 1024-dimensional hidden layers and an output layer with vocabulary size 63.

Original: The model was trained on Wikipedia data with all character lower-cased and whitespace removed.
Revised: The model was trained on Wikipedia data with all characters lower-cased and whitespace removed.

Original: We tested the model with sentences sampled from Anna Karenina as the WLSTM model, and we got bits-per-character (BPC) close to  Hahn & Baroni (2019) reported in the original paper.
Revised: We tested the model with sentences sampled from Anna Karenina as the WLSTM model, and we got bits-per-character (BPC) close to what Hahn & Baroni (2019) reported in the original paper.

Original: 3. T EMPORAL C ONTEXT C ONSTRUCTION P ARADIGM
Revised: 3. T EMPORAL C ONTEXT C ONSTRUCTION P ARADIGM

Original: In order to determine the processing timescales of cell state vectors and individual units, we modiﬁed the ”temporal context construction” method developed by Chien & Honey (2020) .
Revised: In order to determine the processing timescales of cell state vectors and individual units, we modiﬁed the “temporal context construction” method developed by Chien & Honey (2020).

Original: Thus, the internal representations of the model were compared across two conditions: (1) the intact context condition and (2) the random context condition.
Revised: Thus, the internal representations of the model were compared across two conditions: (1) the Intact Context condition and (2) the Random Context condition.

Original: In both conditions, the model was processing the same shared sequence of words (for example, segment B), but the preceding sentence differed across the two conditions (Figure 1A, Appendix A.1).
Revised: In both conditions, the model was processing the same shared sequence of words (for example, segment B), but the preceding sentence differed across the two conditions.

Original: In the Intact Context condition, the model processed segment B (the shared segment) preceded by segment A, which was the actual preceding context from the original text. 
Revised: In the Intact Context condition, the model processed segment B (the shared segment) preceded by segment A, which was the actual preceding context from the original text. In the current study, for example, segment A and B are connected by “, and” within long sentences from the novel corpus (Figure 1A), to ensure the temporal dependencies between A and B.

Original: In the Random Context condition, however, the model processed the same shared input (segment B), but it was preceded bya randomly sampled context, segment X, which was  sampled  from the rest of the corpus.
Revised: In the Random Context condition, however, the model processed the same shared input (segment B), but the context was replaced by segment X, which was a randomly sampled segment from the rest of the corpus.

Original: Segment X was therefore not usually coherently related to segment B.
Revised: Segment X was therefore not usually coherently related to segment B.

Revised: For the WLSTM timescale analysis, we chose long sentences in the Intact Context condition that satisﬁed the following constraints: (1) mean perplexity across all words in the sentence < 200, (2) the shared segment was longer than 25 words, and (3) the context segment was longer than 10 words.

Revised: 77 sentences are included as trials in our analyses.

Revised: In the Random Context condition, we preserved the same shared segments and randomly sampled 30 context segments (each longer than 10 words) from other parts of the novel.

Revised: For the CLSTM timescale analysis, we used the same 77 long sentences in the Intact

Original:  In these two conditions, the model is processing the same input (the shared segment) with different  context (the intact vs. random context).
Revised: Context condition, and randomly sampled 25 context segments (with length > 33 characters) for the Random Context condition. In brief, the model is processing the same input (the shared segment) with different preceding context (the intact vs. random context).

Original: We can then examine how the cell state activation differ between the two conditions, while the network is processing the shared segments with identical input.
Revised: We can now measure the context dependence of individual units by examining how the cell state activations differ between the two conditions, while the network is processing the shared segments with identical input.

Original: Any difference in internal representations(i.e. hidden state activations) must arise from the context manipulation, since the current input is the same (i.e. Sentence B).
Revised: Any difference in internal representations must arise from the context manipulation, since the current input is the same.

Original: Decrease in activation difference over time means that the cell state activation is less and less affected by the context.
Revised: Decrease in activation difference over time means that the cell state activation is less and less affected by the context.

Original: For a long-timescale unit, whose current state is dependent on information in the far-preceding context, we will see an activation difference is preserved across contexts (Figure 1B, green curve), even while the unit is processing the shared input.
Revised: For a long-timescale unit, whose current state is dependent on information in the far-preceding context, we will see that the activation difference is preserved across contexts (Figure 1B, green curve), even while the unit is processing the shared input.

Original: On the other hand, for a short-timescale unit, which is driven largely by the current input, we will see that the activation difference drop quickly (Figure 1B, red curve) as the unit processes the shared input.
Revised: On the other hand, for a short-timescale unit whose activation is driven largely by the current input, we will see that the activation difference drops quickly (Figure 1B, red curve) as the unit processes the shared input.

Original:  H IERARCHICAL O RGANIZATION OF T IMESCALES A CROSS L AYERS
Revised:  H IERARCHICAL O RGANIZATION OF T IMESCALES A CROSS L AYERS

Original: Do higher levels of the LSTM model exhibit greater context-dependence?
Revised: Do higher levels of the LSTM model exhibit greater context-dependence?

Original: Lakretz et al.
Revised: Lakretz et al.

Original: (2019) observed that long-range functional units were more common in higher layers, and in general higherlevels of hierarchical language model exhibit longer range context-dependence (Jain et al., 2019;
Revised: (2019) observed that long-range functional units were more common in higher layers, and in general, higherlevels of hierarchical language model exhibit longer range context-dependence (Jain et al., 2019;

Original: Jain & Huth, 2018).
Revised: Jain & Huth, 2018).

Original: Therefore, to validate our stimuli and the sensitivity of our methods, we ﬁrst compared the processing timescales of different hidden layers in both of the LSTMs. The higher layers of cell state activation patterns exhibited longer context-dependence scales, relative to the lower layers, in both the CLSTM and WLSTM. We correlated the cell state vectors, column by column, between the Intact condition and Random condition.
Revised: Therefore, to validate our stimuli and the sensitivity of our methods, we ﬁrst compared the processing timescales of different hidden layers in both of the LSTMs, by correlating the cell state vectors, column by column, between the Intact condition and Random condition.

Original: We found that both layers showed near-zero correlation when processing the different context, and the correlation increase as they began to process the shared input.
Revised: We found that both layers showed near-zero correlation when processing the different context and the correlation increased as they began to process the shared input.

Original: In the WLSTM, the correlation increased more slowly for second-level cell state vectors than for ﬁrst-level cell state vectors.
Revised: In the WLSTM, the correlation increased more slowly for second-level cell state vectors than for ﬁrst-level cell state vectors.

Original: Thus, the representation of second-level cell state is more sensitive to the different context than the ﬁrst level.
Revised: Thus, the representation of second-level cell state is more sensitive to the different context than the ﬁrst level.

Original: Similarly, for the CLSTM model, the third-level cell state exhibited longer-lasting context sensitivity than lower levels (Figure A.1).
Revised: Similarly, for the CLSTM model, the third-level cell state exhibited longer-lasting context sensitivity than lower levels (Figure 2).

Original: This observation of longer context-dependence in higher stages of processing is consistent with prior machine learning analyses (Lakretz et al., 2019; Jain & Huth, 2018) and is also analogous to what is seen in the human brain (Hasson et al., 2015; Chien & Honey, 2020; Lerner et al., 2011; Jain et al., 2019).
Revised: This observation of longer context-dependence in higher stages of processing is consistent with prior machine learning analyses (Lakretz et al., 2019; Jain & Huth, 2018) and is also analogous to what is seen in the human brain (Hasson et al., 2015; Chien & Honey, 2020; Lerner et al., 2011; Jain et al., 2019).

Original: However, our goal in this study was to examine the functional organization at a ﬁner scale, testing the temporal dependence of individual units within individual layers of the model.

Original: Based on the ﬁnding of longer processing timescales in higher layers, we examined single units in the highest level hidden units, i.e. the second level of WLSTM (n=650) and the third level of CLSTM (n=1024).
Revised: Based on the ﬁnding of longer context dependence in higher layers, we examined single units in the highest level hidden units, i e. the second level of WLSTM (n=650) and the third level of CLSTM (n=1024).

Original:  P ROCESSING T IMESCALES OF I NDIVIDUAL U NITS W ITHIN LSTM L AYERS 5. Q UANTIFYING  T IMESCALESVIA THE R ATE OF C ROSS -C ONTEXT A CTIVATION A LIGNMENT
Revised:  P ROCESSING T IMESCALES OF I NDIVIDUAL U NITS W ITHIN LSTM L AYERS 5. Q UANTIFYING S INGLE U NIT T IMESCALES

Original: We examined the absolute single unit activation difference when processing the shared segments preceded by different context.
Revised: We examined the absolute single unit activation difference when processing the shared segments preceded by different context.

Original: As expected, most of the hidden units showed different activation when the input tokens were different (i.e. while processing the non-shared context in the Intact
Revised: As expected, most of the hidden units showed different activation when the input tokens were different (i.e. while processing the non-shared context in the Intact

Original: Context or Random Context conditions).
Revised: Context and Random Context conditions).

Original: However, once the shared input tokens begin (at t = 0 ) the Intact-Random activation differences drop (Figure 1C, 1D).
Revised: However, once the shared input tokens begin (at t = 0 ) the Intact-Random activation differences drop (Figure A.1A, A.1B).

Original: We use the rate at which the curves drop to quantify the processing timescale, as this is a measure of how quickly the responses align across differing contexts (Figure 1C, 1D).
Revised: We used the rate at which the curves drop to quantify the processing timescale, as this is a measure of how quickly the responses align across different context conditions.

Original: To quantify the timescale of individual units, we ﬁt the activation difference curves with a logistic function:
Revised: To quantify the timescale of individual units, we ﬁt the activation difference curves with a logistic function:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: As shown in Figure 1C and Figure 1D, the logistic function ﬁt the raw activation difference curves.
Revised: As shown in Figure A.1A and Figure A.1B, the logistic function ﬁt the raw activation difference curves.

Original: We then computed the ”timescale” of each unit as the time-to-half-maximum of the logistic decay.
Revised: We then computed the ”timescale” of each unit as the time-to-half-maximum of the logistic decay.

Original: In particular, for the WLSTM we used the activation difference Y (0) at the beginning of the shared segment, and at the end of the shared segment Y (24) ( Y (79) for the CSLTM) to calculate the timeto-half-maximum of unit i as:
Revised: In particular, for the WLSTM we used the activation difference Y (0) at the beginning of the shared segment, and at the end of the shared segment Y (24) ( Y (79) for the CSLTM) to calculate the time-to-half-maximum of unit i as:

Original: [Equation]
Revised: [Equation]

Original: where the inverse function Y − 1 ( y ) identiﬁes the largest integer t , for which Y ( t ) <
Revised: where the inverse function Y − 1 ( y ) identiﬁes the largest integer t , for which Y ( t ) <

Original: y .
Revised: y .

Original: We included 635 units in WLSTM and 1012 units in CLSTM for further analysis after excluding the units which cannot be properly ﬁtted by  logistic functionto quantify their timescales (See Appendix A.2).
Revised: We included 635 units in WLSTM and 1012 units in CLSTM for further analysis after excluding the units which could not be accurately ﬁt by a logistic function (See Appendix A.1).

Original: 5. I DENTIFYING L ONG - TIMESCALE F UNCTIONAL U NITS

Original: As we were testing the same word-level LSTM previously studied by Lakretz et al.

Original: , we began by examining the timescales of hidden-state units that were already known to be involved in processing context-dependence language information: a ”singular number unit” 776, a ”plural number unit” 988, and a ”syntax unit” 1150.

Original: Previously, the ”number” units (singular/plural) were shown to increase their activation when encountering the singular/plural nouns, ignoring in-congruent attractors and reacting to the corresponding singular or plural verbs (Lakretz et al., 2020).

Original: Most importantly, both ”number” units received information from the ”syntax” unit, which tracked long-range subjectverb correspondence.

Original: We found that compared to other units, the 2 ”number” units have similar, medium timescales (3 words), while ”syntax” unit has a longer timescale (7 words).

Original: This is consistent with the ﬁndings by Lakretz et al.

Original: (2019) that ”syntax” unit, which requires a long timescale to track subject-verb dependency, provides information to ”number” units (via strong hidden to input and forget gate projections), which do not necessarily have long timescales by themselves to keep the context.

Original: 5. D ISTRIBUTION OF P ROCESSING T IMESCALES IN A W ORD - LEVEL LSTM
Revised: 5. D ISTRIBUTION OF P ROCESSING T IMESCALES IN A W ORD - LEVEL LSTM

Original: Most units were insensitive to long-range context, but we identiﬁed a substantial subset with long processing timescales. Of the 635 WLSTM units whose processing timescale we mapped, approximately 70% of the units were insensitive to long-range context (processing timescale < 3 words): their activation difference dropped immediately at onset of the shared segment.
Revised: The results showed that of the 635 WLSTM units whose processing timescale we mapped, approximately 70% of the units were insensitive to long-range context (processing timescale < 3 words): their activation difference dropped immediately at onset of the shared segment.

Original: However, approximately 13% of the units had a timescalesequal to or longer than the syntax units (processing timescale > 7 words)(Figure A.2A).
Revised: In contrast, only approximately 13% of the units had a timescales > 7 words (Figure A.2A).

Original: Figure 2A shows the absolute activation difference of all units in WLSTM sorted by timescale (long to short).
Revised: Figure 3A shows the absolute activation difference of all units in WLSTM sorted by timescale (long to short).

Original: Some of the longer-timescale units continued to exhibit a large activation difference even when processing the shared segments for more than 20 tokens.
Revised: Some of the longer-timescale units continued to exhibit a large activation difference even when processing the shared segments for more than 20 tokens.

Original: This suggests that some of the units have timescales longer than we measured here, consistent with previous ﬁndings that LSTM overall perplexity can be affected by up to 50 ordered context tokens (Khandelwal et al., 2018). 
Revised: This suggests that some of the units have timescales longer than we measured here, consistent with previous ﬁndings that LSTM overall perplexity can be affected by up to 50 ordered context tokens (Khandelwal et al., 2018). As we were testing the same word-level LSTM previously studied by Lakretz et al.

Revised: , we began by examining the timescales of hidden-state units that were already known to be involved in processing context-dependence language information: a “singular number unit” 988, a “plural number unit” 776, and a “syntax unit” 1150.

Revised: We found that, compared to other units, both “number” units had medium timescales ( ∼ 3 words, ranked 129/650 units), while the “syntax” unit had a long timescale ( ∼ 7 words, ranked 64/650 units) (Figure A.1).

Original: 5.D ISTRIBUTION OF P ROCESSING T IMESCALES IN A C HARACTER - LEVEL LSTM
Revised: 5. T IMESCALE V ARIANCE ACROSS D IFFERENT D ATASETS AND C ONTEXT C ONDITIONS

Revised: To ensure that the timescales we measured are robust across datasets, we conducted the same analysis using the Wikipedia testing dataset used in Gulordava et al.

Revised: The mapped timescales were highly correlated (r=0.82, p < 0.001) across the Anna Karenina dataset and the Wikipedia dataset (Appendix A.2.1, Figure A.4A).

Revised: Similarly, to conﬁrm that the timescales measured are not speciﬁc to our testing using the “, and” conjunction point, we also measured timescales at an alternative segmentation point, and found the timescales largely preserved (r=0.83, p < 0.001), except for a subset of units (Appendix A.2.2, Figure

Revised: A.4B).

Revised: Although the timescales of context dependence we measured were based on the “token distance”, they may vary under different context conditions, following the “syntactic distance”.

Revised: For example, one would expect that the units should “reset” at the beginning of a new sentence, and show little context dependence under that condition.

Revised: Indeed, we found that most units showed little context dependence when the context segments ended with a “full stop”, which served as a clear signal for the end of a sentence (Appendix A.2.3, Figure A.4C).

Revised: Finally, we examined the role of individual words in shaping context representation and affecting the timescales measurement.

Revised: We did this by shufﬂing the words in the Random Context condition instead of replacing it with whole segments from other parts of the corpus.

Revised: We found that while individual words play an important role in shaping the context dependence of unit timescale, several units showed a longer timescale when that prior context is composed of coherently structured language (Appendix A.2.4, Figure A.4D).

Original: We repeated the timescale mapping in the CLSTM model, and again identiﬁed a small subset of long-timescale units (Figure 2B, Figure A.2B).
Revised: We repeated the timescale mapping in the CLSTM model, and again identiﬁed a small subset of long-timescale units (Figure 3B, Figure A.2B).

Original: Although there are overall more units in CLSTM, over 63% of the units are insensitive to the context (timescale < 3 characters).
Revised: Although there are overall more units in CLSTM, over 63% of the units are insensitive to the context (timescale < 3 characters).

Original: Fewer than 15% of the units exhibited timescale > 10 characters, and the unit with the longest timescale only dropped to its half-maximum activation-difference after 50 characters of shared input.
Revised: Fewer than 15% of the units exhibited timescale > 10 characters, and the unit with the longest timescale only dropped to its half-maximum activation-difference after 50 characters of shared input.

Original:  C ONNECTIVITY OF U NITS WITH L ONG AND M EDIUM P ROCESSING T IMESCALES
Revised:  C ONNECTIVITY OF U NITS WITH L ONG AND M EDIUM P ROCESSING T IMESCALES

Original: Having mapped the timescales of each processing units, we next asked: do units with longer timescales exhibit different connection patterns from units with shorter processing timescales?
Revised: Having mapped the timescales of each processing units, we next asked: Does the processing timescale relate to the functional role of each unit in the network? More speciﬁcally, are units with longer timescales also units with high degree in the connectivity network?

Original: To answer this question, we analyzed  the direct weight projections from one hidden unit at time t to the input and forget gate of other hidden units at time t + 1 .We set out to measure both (i) the similarity connection proﬁle (
Revised: To answer these questions, we analyzed (1) the projection strength of each unit and (2) the similarity of the overall projection pattern (hidden-to-gates) across different units, using the direct weight projections from one hidden unit at time t to the input and forget gate of other hidden units at time t + 1 .

Original: In LSTMs, the amount of contextual ( c t − 1 ) and input ( ˜ c t ) information stored in the cell state ( c t ) is determined by the forget gate ( f t ) and input gate ( i t ) activation (Eq. 3); and the activation of the gates i t and f t are determined by the current input at time t and the hidden units at time t +1 through weight matrices W i and Wh (Eq. 4, 5).
Revised: In LSTMs, the amount of contextual ( c t − 1 ) and input ( ˜ c t ) information stored in the cell state ( c t ) is determined by the forget gate ( f t ) and input gate ( i t ) activation (Eq. 3); and the activation of the gates i t and f t are determined by the current input at time t and the hidden units at time t − 1 through weight matrices U and W (Eq. 4, 5).

Original: [Equation]
Revised: [Equation]

Original: Thus, to understand how the contextual information over different timescales is projected from the hidden units to the input and forget gates of other units, and how this inﬂuences subsequent cell states, wenext analyzed the network connectivity, focusing on the weight matrices
Revised: Here, we are interested in understanding how the contextual information over different timescales is projected from the hidden units to the input and forget gates of other units, and further inﬂuence the update of cell states. Thus, we analyzed the network connectivity focusing on the weight matrices

Original: W hi and W hf within the highest layer of the WLSTM or CLSTM.
Revised: W i and W f within the highest layer of the WLSTM or CLSTM.

Original: 6. S TRONGH IDDEN -G ATE P ROJECTIONS FROM L ONG -T IMESCALE  U NITS
Revised: 6. S TRONG P ROJECTIONS FROM L ONG -T IMESCALE H IDDEN U NITS TO G ATE U NITS

Original: Units with longer processing timescales made a larger number of strong projections ( | z-score | > 5,
Revised: Units with longer processing timescales made a larger number of strong projections ( | z-score | > 5,

Original: Appendix A.3) to the input and forget gates of other units in both WLSTM (r=0.31, p < 0.001, Figure 3A) and CLSTM models (r=0.24, p < 0.001, Figure A.3A)
Revised: Appendix A.3) to the input and forget gates of other units in both WLSTM (r=0.31, p < 0.001, Figure 4A) and CLSTM models (r=0.24, p < 0.001, Figure A.5A).

Original: In WLSTM, the ”syntax” unit (Unit 1150) reported by Lakretz et al.
Revised: Further, we found that the “syntax” unit (Unit 1150) reported by Lakretz et al.

Original: (2019)  possessed the most strong projections to  input and forget gates of all other units. Furthermore, the target units that Unit 1150 strongly project to are units with medium- to long-timescale units.
Revised: (2019) in WLSTM possessed the largest number of strong projections to the input and forget gates of all other units, and the target units that Unit 1150 strongly project to are units with medium- to long-timescale units (Figure 4B).

Original: Figure 3B illustrates all the z-scored weight projections from Unit 1150 to the forget gate and input gate, and the projected units are sorted by timescales.

Original: 6.1. C ONTROLLERAND I NTEGRATOR U NITS IN A W ORD - LEVEL LSTM 
Revised: 6. C ONTROLLER U NITS IN A W ORD - LEVEL LSTM AND A C HARACTER - LEVEL LSTM

Original: The presence of strong projections from the ”syntax” unit(Unit 1150) to other long-timescale units implies that the long-timescale units with strong projections may also densely interconnect.
Revised: The presence of strong projections from the “syntax” unit to other long-timescale units motivated us to further explore whether high-degree, long-timescale units in the LSTM also densely interconnect to form a “core network”, as seen in the brain (Hagmann et al., 2008; Mesulam, 1998; Baria et al., 2013).

Original: If so, this set of units may have an especially important role in determining how prior context is updated and how it is used to gate current processing.
Revised: If so, this set of units may have an especially important role in determining how prior context is updated and how it is used to gate current processing, analogous to the controller system in the brain (Gu et al., 2015).

Original: To identify such ”controller units”, we binarized the network by identifying a set of 258 ”strong” projections (see Appendix A.3), which provided the edges for a network analysis.
Revised: To identify such “controller units”, we binarized the network by identifying the top 258 projection weights from the weight matrices (see Appendix A.3), which provided the edges for a network analysis.

Original: We then used k-core analysis (Batagelj & Zaversnik, 2003) to identify the ”main network core” (the core with the largest degree) amongst these nodes (Figure 3C).
Revised: We then used k-core analysis (Batagelj & Zaversnik, 2003) to identify the “main network core” (the core with the largest degree) of the network (Figure A.3).

Original: At the maximal k = 5 , the k-core analysis yielded a set of densely interconnected nodes, composed of many long-timescale and medium-timescale units (labeled in red in Figure 3A).
Revised: At the maximal k = 5 , the k-core analysis yielded a set of densely interconnected nodes, composed of many long-timescale and medium-timescale units (Figure A.3), also labeled in red in Figure 4A).

Original: We  refer to this set as the ”controller” set of the network.
Revised: We (tentatively) refer to this set as the “controller” set of the network.

Revised: We performed the same k-core analyses on the CLSTM and observed similar results that the main core network was composed of disproportionately many medium and long-timescale “controller” units (Figure A.5A).

Original: Finally, we used multi-dimensional scaling (MDS) to visualize the similarity of connection pattern among units in LSTM.
Revised: We used multi-dimensional scaling (MDS) to visualize the similarity of projection patterns across LSTM units.

Revised: We recovered a 2-dimensional MDS embedding, where the inter-unit distances was deﬁned based on the similarity of their hidden-to-gate projection patterns (i.e., values in the unthresholded LSTM weight matrices W i and W f ).

Original: Figure 3D shows the2-dimensional MDS space of the unit connection patterns of all units, color-coded by timescales.
Revised: We visualized the MDS solution as a graph structure, in which each node is a unit, and the edges reﬂect connection properties of that unit. Figure 4D shows the resulting 2-D space, with units color-coded by their timescale.

Original: The size of the nodes indicate the number of strong projections from the node, and the lines indicate their input gate/forget gate projections.
Revised: The size of the nodes indicates the number of strong projections from the node, and the lines indicate those projections (to forget gate and input gate, based on the color of the line).

Original: We observed that the ”controller units” (labeled on Figure 3D) are positionedat different locations around the edge of the MDS space, suggesting that these units have distinct connection pattern from other ”controller” units and compared to the rest of the network. 
Revised: “Controller units” (labeled on Figure 4D) were positioned around the periphery of the MDS space, suggesting that these units have distinct projection patterns, both from other “controller” units and from the rest of the network. In contrast, we observed several long-timescale units positioned in the center of the MDS space, suggesting that the projection patterns of these units are close to the mean projection pattern.

Original: We also found that several otherlongtimescale units which have few strong projections, are positioned in the center of the MDS space, suggesting that these ”integrator” units have connection proﬁles that are close to the mean connection pattern.
Revised: We refer to this more MDS-central set as the “integrator units” (labeled in green in Figure 4A). Similar to the WLSTM, the projection patterns of the “controller units” in CLSTM were distinct from other units in the network, according to the MDS results (Figure A.5C). However, we did not observe the “integrator units” positioned in the center of the MDS space of the CLSTM as was observed in the WLSTM.

Original: 6.1. C ONTROLLER  U NITSIN A C HARACTER - LEVEL LSTM
Revised: 6. R OLES OF C ONTROLLER AND I NTEGRATOR U NITS

Original: Performing the same network analyses on the CLSTM yielded similar results, insofar as (1) the main core network was composed of disproportionately many long-timescale and medium-timescale units (Figure A.3A, A.3B) and (2) the connection pattern of these  controller units was distinct from other units in the network, according to the MDS results (Figure A.3C). However, we did not observe the ”integrator units” positioned in the center of  the MDS space of the CLSTM as was observed in the
Revised: Are the “controller” and “integrator” units important for the model’s ability to predict the next token? To test the functional importance of these subsets of units, we conducted group ablation analyses (See Appendix A.5). Ablating controller units reduced the accuracy of token prediction overall, while ablating integrator units only reduced prediction accuracy for the last words of the sentences (Figure 4C).

Original: WLSTM.
Revised: The results conﬁrm that the putative controller and integrator nodes are functionally signiﬁcant, with distinctive roles in the WLSTM language model. Finally, to test the generalization of the timescale and connectivity analyses to a different model architecture, we conducted a preliminary analysis of a Gated Recurrent Unit (GRU) language model (Cho et al., 2014). These GRU results are preliminary, because the model was not fully trained, and its perplexity was an order of magnitude inferior to the LSTM models. We found similar sparsity of long-timescale units in the GRU, but did not observe the same relationship between timescales and connectivity (Appendix A.4; Figure A.6; A.7).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We demonstrated a new method for mapping the timescale organization in recurrent neural language models.
Revised: We demonstrated a new method for mapping the timescale organization in recurrent neural language models.

Original: We quantiﬁed processing timescales by measuring the number of shared tokens (words or characters) that are required for an individual unit to ”align” its activation patterns, when the shared tokens were preceded by distinct contexts.

Original: Using this method, we mapped the timescale distributions of units within word-level and character-level neural language models and identiﬁed subsets of 15% of units with long timescales.
Revised: Using this method, we mapped the timescale distributions of units within word-level and character-level LSTM language models and identiﬁed a small set of units with long timescales.

Original: Furthermore, in both models, we identiﬁed a ”controller” subnetwork, in which the units (i) possess unusually signiﬁcantly strong projections to  the forget and input gates of one another and the rest of the network; and (ii) have unusually long processing timescale units on average.
Revised: We then used network analyses to understand the relationship between the timescale of a unit and its connectivity proﬁle and we distinguished two subsets of long-timescale units with seemingly different functions.

Original: Altogether, we proposed a method for discovering timescaleorganization in RNN and applied it to reveal a hierarchical timescale organization within LSTM language models.
Revised: Altogether, we proposed methods combining timescale and connectivity analyses for discovering timescale and functional organization in language models.

Original: Theset of units with long processing timescales included  units whose role in long-range language dependencies was already  established (Lakretz et al., 2019), but almost all  the long timescale units are of unknown function.
Revised: The units with long processing timescales included some units whose role in long-range language dependencies had already been established (Lakretz et al., 2019) but almost all of the long timescale units are of unknown function.

Original: The timescale mapping procedure described here provides a modelfree method for identifying nodes necessary for long-range linguistic and discursive processes (e.g. tracking whether a series of words constitutes an assertion or a question).
Revised: The timescale mapping procedure described here provides a modelfree method for identifying nodes necessary for long-range linguistic and discursive processes (e.g. tracking whether a series of words constitutes an assertion or a question).

Original: Future studies of these neural language models could focus on the functions of long-timescale units, and especially the controller units, to understandhow and why each unit is tracking long-range dependencies.
Revised: Future studies of these neural language models could focus on the functions of long-timescale units, and especially the “controller” units, to understand why each unit is tracking long-range information, and how they control the information ﬂow of other units in the network.

Original: Processing timescales may exhibit an analogous hierarchical organization in LSTMs and in the human cerebral cortex: in both cases, a subset of nodes with high degree and high inter-connectivity express unusually long timescales. More detailed testing of this correspondence is required, however, because units within an LSTM layer are not spatially embedded, and thus may not show a spatially graded timescale topography. Finally, we note that our analyses relied on a relatively small set of long sentences from a single novel; future applications using larger and more diverse text corpora will be required to generalize these ﬁndings. A A PPENDIX A. D ETAILS OF TRAINING AND TESTING
Revised: Moreover, it will be important to investigate whether the processing timescales characterized via token distance (as demonstrated in the current study) are comparable to those measured using functional measures, such as syntactic distance.

Original: The context and shared segments in the Intact Context condition are connected by ”, and” in long sentences from the novel. In this way, we ensured that the  context segments and the shared segments have temporal dependencies which can be tracked by long-timescale units.
Revised: Relatedly, while we explored the timescale variance under several context conditions, a more thorough investigation will be needed to examine how the timescales of individual units may vary at different positions within a sentence, both in terms of token location and syntactic location.

Original: For the WLSTM timescale analysis, we chose long sentences in the Intact Context condition that satisﬁed the following constraints: (1) mean perplexity across all words in the sentence < 200, (2) the shared segment was longer than 25 words, and (3) the context segment was longer thanwords. 77 sentences are included as trials in our analyses. To generate a paired ”sentence” for the Random Context condition, we preserved the same shared segments and randomly sampledcontext segments (each longer than 10 words) from other parts of the novel. To compute the absolute activation difference at a given token position, (i) for each Intact Context sentence we computed the mean absolute activation-difference at that position with each of its 30 associated Random Context segments; and then (ii) we computed the mean of these absolute difference values across all 77 of the Intact Context sentences.
Revised: Processing timescales may exhibit an analogous hierarchical organization in LSTMs and in the human cerebral cortex: in both cases, a subset of nodes with high degree and high inter-connectivity express unusually long timescales.

Original: For the CLSTM timescale analysis, we used the same 77 long sentences in the Intact Context condition, and randomly sampled 25 context segments (with length ¿ 33 characters) for the Random Context condition. The mean bits-per-character (BPC) = 1.49, was close to the BPC reported in Hahn & Baroni (2019).
Revised: More detailed testing of this apparent correspondence is required, however, because units within an LSTM layer are not spatially embedded, and thus may not show a spatially graded timescale topography.

Original:  A. U NITS EXCLUDED FROM TIMESCALE ANALYSIS
Revised: A A PPENDIX A. U NITS EXCLUDED FROM TIMESCALE ANALYSIS

Original: We excluded 1 unit in  WLSTM model and 5 units in CLSTM model which failed to be properly ﬁtted using the logistic function; we further excluded 14 units in the WLSTM mode and 7 units in the CLSTM model which either did not show a near-zero activation difference before the shared segment started, or show an increasing activation difference when started to process the shared segment.
Revised: We excluded 1 unit in the WLSTM model and 5 units in CLSTM model which were not properly ﬁt using the logistic function; we further excluded 14 units in the WLSTM model and 7 units in the CLSTM model which either did not show a non-zero activation difference before the shared segment started, or whose activation differences increased when started to process the shared segment.

Original: After these exclusions,there were 635 units remaining in the WLSTM and 1012 units remaining in the CLSTM for further analysis. 
Revised: After these exclusions, 635 units remained in the WLSTM and 1012 units remained in the CLSTM for further analysis. A. T IMESCALE ANALYSES ACROSS DIFFERENT DATASETS AND CONTEXT CONDITIONS A.2. W IKIPEDIA TEST DATASET

Revised: The Anna Karenina corpus used in the current study has a different linguistic structure from the

Revised: Wikipedia corpus on which the WLSTM and CLSTM models were trained.

Original: A. I DENTIFYING STRONG HIDDEN - TO - GATE PROJECTIONS
Revised: Although we analyzed only the Anna Karenina sentences with low perplexity, it wsa important to test the robustness of our results across datasets. Thus, we mapped the timescale of each unit using the Wikipedia test set, as used by Gulordava et al. Speciﬁcally, we sampled 500 long sentences containing “, and” for the Intact Context condition. As before, we generated sentences by preceding the “shared input” segment (after the conjunction) with either the original prior context segment, or a randomly chosen prior context segment.

Revised: Same as the original analysis, we then replaced the context segment with 30 context segments randomly sampled from other parts of the test set for generating the Random

Revised: Context condition.

Revised: The mapped timescales using the Wikipedia test set were highly correlated with the novel corpus, suggesting the robustness of unit timescales (Figure A.4A).

Revised: A.2. T IMESCALES MEASURED IN THE MIDDLE OF A SENTENCE

Revised: To examine how the timescales of individual units may vary across different positions in a sentence, we varied the location of the segmentation point.

Revised: Instead of using the conjunction (“, and”) as the segmentation point, we chose an arbitrary segmentation point: the 15th token of a long sentence, to separate context segment and shared input segment.

Revised: In the Random Context condition, we replaced the context segment with the ﬁrst 15 tokens from other sentences of the corpus.

Revised: We found that the unit timescales were highly correlated with the condition where we used the conjunction as the segmentation point with several units shift their timescales to either directions (Figure A.4B).

Revised: This analysis was conducted using Wikipedia test set.

Revised: A.2. T IMESCALE RESET AT THE BEGINNING OF A SENTENCE

Revised: To examine if the timescales of individual units can ﬂexibly reset at the beginning of a sentence, we conducted the same timescale analysis but using a “full stop” as the segmentation point instead of the conjunction “, and”.

Revised: Thus, if the original test string was “The girl kicked the call, and the boy caught it”, then the full-stop test string would be “The girl kicked the ball.

Revised: The boy caught it.”

Revised: In this setting, the context segment and shared input segment in the Intact Context condition are two consecutive sentences.

Revised: To ensure the temporal dependence between the context segment and shared input segment, we sampled 100 consecutive sentence pairs from the Anna Karenina corpus.

Revised: Note that This is not possible using the Wikipedia test set from Gulordava et al.

Revised: (2018), which is composed of unrelated sentences.

Revised: The Random Context condition was generated by replacing the ﬁrst sentence with randomly sampled sentences from other parts of the novel.

Revised: We found that when using “full stop” to segment context and shared input, most units in the network showed timescale near 0, indicating near-zero dependence on the linguistic context from the text preceding the full stop (Figure A.4C).

Revised: This suggests that the units in LSTM tend to “reset” their context representation at the beginning of a sentence.

Revised: A.2. C ONTEXT REPRESENTATION SHAPED BY INDIVIDUAL WORDS

Revised: Inspired by the token-shufﬂing procedure of Khandelwal et al.

Revised: , we explored whether the context representations of individual units in the LSTM were shaped by individual words, rather than coherent sequences of words.

Revised: For this analysis, instead of replacing the context with syntactically structured segments from other part of the corpus, we generated the “random context” by shufﬂing the order of words within the context segment.

Revised: We then mapped the unit timescales as before, by examining the unit activation difference as a function of the distance from the onset of shared input.

Revised: Intriguingly, we found that most of the units showed similar timescales across the context-replacement and context-shufﬂing procedures (Figure A.4D).

Revised: This suggests that the context representations in LSTMs largely depend on the presence of individual words in the context, rather than their appearance within coherent linguistic sequences.

Revised: However, we did observe a subset of units (labeled in the Figure, and almost all long-timescale units) whose timescales were longer when context was replaced rather than shufﬂed.

Revised: For this subset of units, the ability to maintain a representation of prior context over many tokens depends on that prior context being a coherent linguistic sequence.

Revised: This subset of units are a promising target for future studies of syntactic representations in LSTMs.

Original: First, for each  unit, we concatenated its corresponding rows in the W hi and W hf matrices, to generate a single ”hidden-to-gate” projection vector for that  unit.
Revised: First, for each hidden unit, we concatenated the corresponding rows in the W hi and W hf matrices, to generate a single “hidden-to-gate” projection vector for that hidden unit.

Original: Next we we z-scored the vector to get standardized projection values from that unit to all other units in the network.
Revised: Next we we z-scored the vector to get standardized projection values from that unit to all other units in the network.

Original: Using | z-score | >as criterion, we identiﬁed a total of 258 ”strong projections” from all hidden units to the input gate and forget gate in the WLSTM.
Revised: Using | z-score | > 5 as criterion, we identiﬁed a total of 258 “strong projections” from all hidden units to the input gate and forget gate in the WLSTM.

Original: The projection strength of each unit was then calculated based on its number of ”strong projections” (Figure 3A).
Revised: The projection strength of each unit was then calculated based on its number of ”strong projections” (Figure 4A).

Revised: Although the criterion | zscore | > was selected to better visualize the results in Figure 4, different criteria did not change the results that units with longer timescales have more strong projections.

Revised: For example, using | z-score | > 3 as threshold we obtained corr(timescale, projections) = 0.30, p < 0.001; | z-score | > 4 we obtained corr(timescale, projections) = 0.35, p < 0.001.

Original: Next, we identiﬁed the edges corresponding to the top 258 magnitude weight values within the combined W hi and W hf matrices.
Revised: Next, we identiﬁed the edges corresponding to the top 258 magnitude weight-values within the combined W hi and W hf matrices.

Original: Together, these edges formed a ”strong-projection network”.
Revised: Together, these edges formed a ”strong-projection network”.

Original: Finally, we used k-core analysis to identify the main core of the strong-projection network.
Revised: Finally, we used k-core analysis to identify the main core of the strong-projection network.

Original: This main core composed our ”controller units” (Figure 3C).
Revised: This main core composed our ”controller units” (Figure A.3).

Original: Using the same criteria and method, we identiﬁed a total of 390 ”strong projections” from all hidden units to the input gate and forget gate in the CLSTM.
Revised: Using the same criteria and method, we identiﬁed a total of 390 “strong projections” from all hidden units to the input gate and forget gate in the CLSTM.

Original: We then extracted the top 390 weight values from the weight matrices to construct a ”strong-projection network” and again identiﬁed the main core network, composed the ”controller units” for the CLSTM model (Figure A.3A, A.3B)
Revised: We then extracted the top 390 weight values from the weight matrices to construct a “strong-projection network” and again identiﬁed the main core network, composed the “controller units” for the CLSTM model (Figure A.5A, A.5B) A. M APPING TIMESCALE ORGANIZATION IN A GRU LANGUAGE MODEL A.4. T RAINING To explore whether the timescale mapping methods, and our ﬁndsings, may generalize to other model architectures, we trained and studied a word-level GRU language model (Cho et al., 2014). As far as possible, we applied the same parameters in the GRU as were used for the LSTM by

Revised: Gulordava et al.

Revised: (2018): the same Wikipedia training corpus, the same loss function (i.e. crossentropy loss), and the same hyperparameters.

Revised: Thus, the GRU model also had two layers, withhidden units in each layer.

Revised: Due to limitations of time and computational resources, we had to stop training the GRU model after 48 hours (10.5 epochs), at which point the GRU achieved a test perplexity of 349.39.

Revised: This perplexity is much higher than that of the LSTM model reported in Gulordava et al.

Revised: (perplexity = 52.1 in the English corpora, after training for 40 epochs and selecting the model with the lowest perplexity out of 68 combinations of different hyperparamters).

Revised: In spite of the fact that the GRU model was sub-optimal, we analyzed the timescale of its hidden units using the same method as was used for analyzing the LSTMs, and using the test data derived from the training Wikipedia corpus.

Revised: A.4. T IMESCALE ORGANIZATION OF A GRU MODEL

Revised: Similar to the LSTM model of Gulordova et al, the majority of the units in the GRU also showed shorter timescales.

Revised: More speciﬁcally, we found: (1) the second layer of the GRU model was more sensitive to prior context than the ﬁrst layer, as in the LSTM (A.6A); (2) the distribution of timescales across units was similar in the GRU and LSTM, although the GRU showed a more right-skewed distribution with a larger proportion of short-timescale units (A.6B, C).

Revised: A.4. T IMESCALE VERSUS NETWORK CONNECTIVITY IN A GRU MODEL

Revised: We also performed the timescale vs. network connectivity analyses on the GRU model.

Revised: Because the update of hidden states in GRU are controlled by the update gate, we measured the projection patterns of hidden units by analyzing the matrix of hidden-to-update-gate weights.

Revised: In contrast to the

Revised: LSTM models, hidden units in the GRU that we trained did not show a relationship between longer timescales and stronger hidden-to-gate projections (A.7A).

Revised: Moreover, when using k-core analysis to identify subunits of interconnected high-degree units, the core network in the GRU contained many units with long to short timescales.

Revised: Interestingly, when we visualized the position of the k-core units in the MDS space, they tended to locate at the edge of the space, similar to what we found in LSTM.

Revised: This indicates that, as in the LSTM, the core units in the GRU have distinctive proﬁles, distant from one another and from other units in the network (A.7B).

Revised: These apparent similarities and differences between LSTM and GRU are intriguing, but we emphasize that (1) the perplexity of this GRU model is much higher than the LSTM, due to limitations in training time and tuning, and that (2) comparing the LSTM and GRU connection patterns is not straightforward, as the overall distribution of weights is different.

Revised: Further work will be required to determine comparable thresholds for “strong” projections and “high-degree units” in each case.

Revised: As we noted in the manuscript and above, the connectivity results are exploratory; however, we believe that the GRU analysis demonstrates how these methods can be extended to map and compare the functional organization of language models of different architectures.

Revised: A. A BLATION ANALYSES ON PUTATIVE CONTROLLER AND INTEGRATOR UNITS

Revised: To examine the non-trivial roles of the controller and integrator units identiﬁed in the word-level

Revised: LSTM model, we performed a preliminary group ablation analysis to look at how ablating the controller units inﬂuences model performance on predicting the next token, relative to the ablation of a random set of units.

Revised: Speciﬁcally, since long-timescale integrator units should have most effect pre-

Revised: dicting tokens at the later part of the sentences (i.e., when more context is integrated), we examined the model performance on predicting tokens at two different positions: (1) all the tokens regardless of their positions in the sentences (“All tokens” condition), and (2) the last tokens of sentences (“Final tokens” condition).

Revised: We evaluated the effects of ablation on model performance by measuring the differences of probabilities ( ∆ P ) assigned to the target words ( ∆ P = probability of target word in ablated model minus probability of target word in original model).

Revised: Ablation effects for controller units ( N =9) and integrator units ( N =10) were compared against a baseline of ablating the same number of randomlyselected units from layer 2 of the LSTM (Figure 4C).

Revised: We used the test corpus used by Gulordava et al.

Revised: and measured the average performance of each model across 100 text-batches, randomly sampled from the Wikipedia test dataset.

Revised: Each text-batch was composed of 1000 tokens that start at the beginning of a sentence.

Revised: In the “All tokens” condition, we calculated the ∆ P for every token in the tested text, while in the “Final tokens” condition, we calculated ∆ P only at the last token of every sentence (i.e. the token right before the full stop“.” of each sentence).

Revised: We then average the ∆ P in both conditions across text-batches to get a mean performance difference between the ablated model and the intact model.

Revised: Ablating controller units reduced the probabilities assigned to the target words, more so than ablating random units (Figure 4C, controller vs. random across 100 text batches: Cohen’s d

Revised: = -4.85, t = -34.28, p < 0.001).

Revised: In contrast, ablating integrator units showed reduced the probabilities less than ablating random units (integrator vs. random:

Revised: Cohen’s d = 2.50, t = 17.67, p < 0.001).

Revised: We hypothesized that that the integrator units mostly inﬂuence the model performance on predicting tokens in cases where long-range information is especially relevant, such as in the later portions of clauses and sentences.

Revised: Consistent with this, we found that, when we examined the ablation effects only for tokens in the ﬁnal position of a sentence, ablating integrator units reduced the probabilities more than ablating random units (Cohen’s d = -0.34, t = -2.41, p = 0.017).

Revised: Interestingly, ablating controller units reduced the probability of sentence-ﬁnal targets less than random units (Cohen’s d = 0.67, t = 4.74, p < 0.001).

Revised: In summary, these ablation results indicate a non-trivial functional role for the controller and integrator units, despite the fact that each subset of units is composed of only 10 amongst 650 total hidden units.

Revised: Also, the putative controller and integrator sets appear to have distinctive roles within the WLSTM, with the controllers supporting accurate predictions overall, while the integrator units appear to boost accurate predictions at the end of sentences.