Original: M APPING THE T IMESCALE O RGANIZATION OF N EURAL L ANGUAGE M ODELS
Revised: M APPING THE T IMESCALE O RGANIZATION OF N EURAL L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contextual information over longer timescales.
Revised: In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contex tual information over longer timescales.

Original: In contrast, in recurrent neural networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized.
Revised: In contrast in recurrent neural networks which perform natural language processing we know little about how the multiple timescales of contextual information are functionally organized.

Original: Therefore, we applied tools developed in neuroscience to map the “processing timescales” of individual units within a word-level LSTM language model.
Revised: Therefore , we applied tools developed in neuroscience to map the “processing timescales” of individual units within a word -level LSTM language model.

Original: This timescalemapping method assigned long timescales to units previously found to track longrange syntactic dependencies, and revealed a new cluster of  previously unreported long-timescale units.
Revised: This timescalemapping method assigned long timescales to units previously found to track longrange syntactic dependencies and revealed a small subset (less than 15%) of units with long timescales, whose function had not previously been explored.

Original: Next, we explored the functional role of units by examining the relationship between their processing timescales and  network connectivity.
Revised: We next probed the functional organization of the network by examining the relationship between the processing timescale of units and their network connectivity.

Original: We identiﬁed two classes of long-timescale units: “Controller” units composed a densely interconnected subnetwork and strongly projected to theforget and input gates of the rest of the network, while “Integrator” units showed the longest timescales in the network, and expressed projection proﬁles closer to the mean projection proﬁle.
Revised: We identiﬁed two classes of long-timescale units: “controller” units composed a densely interconnected subnetwork and strongly projected to the rest of the network while “integrator” units showed the longest timescales in the network, and expressed projection proﬁles closer to the mean projection proﬁle.

Original: Ablating integrator and controller units affected model performance at different position of a sentence, suggesting distinctive functions of these two sets of units.
Revised: Ablating integrator and controller units affected model performance at different positions within a sentence, suggesting distinctive functions of these two sets of units.

Original: Finally, we tested the generalization of these results to a character-level LSTM model.
Revised: Finally we tested the generalization of these results to a character-level LSTM model and models with different architectures.

Original: In summary, we demonstrated a model-free technique for mapping the timescale organization in  neural network models, and we applied this method to reveal the timescale and functional organization of LSTM language models.
Revised: In summary, we demonstrated a model-free technique for mapping the timescale organization in recurrent neural networks and we applied this method to reveal the timescale and functional orga nization of neural language models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Language processing requires tracking information over multiple timescales.
Revised: Language processing requires tracking information over multiple timescales.

Original: To be able to predict the ﬁnal word “timescales” in the previous sentence, one must consider both the short-range context (e.g. the adjective “multiple”) and the long-range context (e.g. the subject “language processing”).
Revised: To be able to predict the ﬁnal word “timescales” in the previous sentence, one must consider both the short-range context (e.g. the adjective “multiple”) and the long-range context (e.g. the subject “language processing”).

Original: How do humans and neural language models encode such multi-scale context information?
Revised: How do humans and neural language models encode such multi-scale context information?

Original: Neuroscientists have developed methods to study how the human brain encodes information over multiple timescales during sequence processing.
Revised: Neuroscientists have developed methods to study how the human brain encodes information over multiple timescales during sequence processing.

Original: By parametrically varying the timescale of intact context, and measuring  changes in the neural response, a series of studies (Lerner et al., 2011; Xu et al., 2005; Honey et al., 2012) showed that higher-order regions are more sensitive to long-range context change than lower-order sensory regions.
Revised: By parametrically varying the timescale of intact context, and measuring the resultant changes in the neural response, a series of studies (Lerner et al., 2011; Xu et al., 2005; Honey et al., 2012) showed that higher-order regions are more sensitive to longrange context change than lower-order sensory regions.

Original: These studies indicate the existence of a “hierarchy of processing timescales” in the human brain.
Revised: These studies indicate the existence of a “hierarchy of processing timescales” in the human brain.

Original: More recently, Chien & Honey (2020) used a time-resolved method to investigate how the brain builds  shared context, when two groups of people processed the same sentence preceded by different contexts.
Revised: More recently, Chien & Honey (2020) used a time-resolved method to investigate how the brain builds a shared representation, when two groups of people processed the same narrative segment preceded by different contexts.

Original: By directly mapping the time required for individual brain regions to converge on a shared representation in response to shared input, they conﬁrmed that higher-order regions take longer to converge to a sharedcontext representation.
Revised: By directly mapping the time required for individual brain regions to converge on a shared representation in response to shared input, we conﬁrmed that higher-order regions take longer to build a shared representation.

Original: Altogether, these and other lines of investigation suggest that sequence processing in the brain is supported by a distributed and hierarchical structure: sensory regions have short processing timescales and are primarily inﬂuenced by the current input and its short-range context, while higher-order cortical regions have longer timescales and track longer-range dependencies (Hasson et al., 2015; Honey et al., 2012; Chien & Honey, 2020; Lerner et al., 2011; Baldassano et al., 2017;Runyan et al., 2017;
Revised: Altogether, these and other lines of investigation suggest that sequence processing in the brain is supported by a distributed and hierarchical structure: sensory regions have short processing timescales and are primarily inﬂuenced by the current input and its short-range context, while higher-order cortical regions have longer timescales and track longer-range dependencies (Hasson et al., 2015; Honey et al., 2012; Chien & Honey, 2020; Lerner et al., 2011; Baldassano et al., 2017;

Original:  Fuster, 1997).
Revised: Runyan et al., 2017; Fuster, 1997).

Original: How are processing timescales organized within recurrent neural networks (RNNs) trained to perform natural language processing?
Revised: How are processing timescales organized within recurrent neural networks (RNNs) trained to perform natural language processing?

Original: Long short-term memory networks (LSTMs) (Hochreiter &
Revised: Long short-term memory networks (LSTMs) (Hochreiter &

Original: Schmidhuber, 1997) have been widely investigated in terms of their ability to successfully solve sequential prediction tasks.
Revised: Schmidhuber, 1997) have been widely investigated in terms of their ability to successfully solve sequential prediction tasks.

Original: However, long-range dependencies have usually been studied with respect to a particular linguistic function (e.g. subject-verb number agreement (Linzen et al., 2016; Gulordava et al., 2018; Lakretz et al., 2019), and there has been less attention on the broader question of how sensitivity to prior context – broadly construed – is functionally organized within these recurrent networks.
Revised: However, long-range dependencies have usually been studied with respect to a particular linguistic function (e.g. subject-verb number agreement, Linzen et al. 2016; Gulordava et al. 2018; Lakretz et al. 2019), and there has been less attention on the broader question of how sensitivity to prior context – broadly construed – is functionally organized within these RNNs.

Original: Therefore, drawing on prior work in the neuroscience literature, here we demonstrate a model-free approach to mapping processing timescale in recurrent neural networks.
Revised: Therefore, drawing on prior work in the neuroscience literature, here we demonstrate a model-free approach to mapping processing timescale in RNNs.

Original: We focused on existing language models that were trained to predict upcoming tokens at the word level (Gulordava et al., 2018) and at the character level (Hahn & Baroni, 2019).
Revised: We focused on existing language models that were trained to predict upcoming tokens at the word level (Gulordava et al., 2018) and at the character level (Hahn & Baroni, 2019).

Original: The timescale organization of these two models both revealed that the higher layers of LSTM language models contained a small subset of units which exhibit long-range sequence dependencies; this subset includes previously reported units (e.g. a “syntax” unit, Lakretz et al., 2019) as well as previously unreported units.
Revised: The timescale organization of these two models both revealed that the higher layers of LSTM language models contained a small subset of units which exhibit long-range sequence dependencies; this subset includes previously reported units (e.g. a “syntax” unit, Lakretz et al., 2019) as well as previously unreported units.

Original: After mapping the timescales of individual units, we asked: do the processing timescales of each unit in the network relate to its functional role, as measured by its connectivity?
Revised: After mapping the timescales of individual units, we asked: does the processing timescales of each unit in the network relate to its functional role, as measured by its connectivity?

Original: The question is motivated by neuroscience studies which have shown that in the human brain, higher-degree nodes tend to exhibit slower dynamics and longer context dependence than lower-degree nodes (Baria et al., 2013).
Revised: The question is motivated by neuroscience studies which have shown that in the human brain, higher-degree nodes tend to exhibit slower dynamics and longer context dependence than lower-degree nodes (Baria et al., 2013).

Original: More generally, the primate brain exhibits a core periphery structure in which a relatively small number of “higher order” and high-degree regions (in the prefrontal cortex, in default-mode regions and in so-called “limbic” zones) maintain a large number of connections with one another, and exert a powerful inﬂuence over large-scale cortical dynamics (Hagmann et al., 2008; Mesulam, 1998; Gu et al., 2015).
Revised: More generally, the primate brain exhibits a core periphery structure in which a relatively small number of “higher order” and high-degree regions (in the prefrontal cortex, in default-mode regions and in so-called “limbic” zones) maintain a large number of connections with one another, and exert a powerful inﬂuence over large-scale cortical dynamics (Hagmann et al., 2008; Mesulam, 1998; Gu et al., 2015).

Original: Inspired by the relationships between timescales and network structure in neuroscience, we set out to test corresponding hypotheses in neural nets: (1) Do units with longertimescales tend to have higher degree in neural language models?
Revised: Inspired by the relationships between timescales and network structure in the brain, we set out to test corresponding hypotheses in RNNs: (1) Do units with longer-timescales tend to have higher degree in neural language models?

Original: and (2) Do neural language models also exhibit a “core network” composed of functionally inﬂuential high-degree units?
Revised: and (2) Do neural language models also exhibit a “core network” composed of functionally inﬂuential high-degree units?

Original: Using an exploratory network-theoretic approach, we found that units with longer timescales tend to have more projections to other units.
Revised: Using an exploratory network-theoretic approach, we found that units with longer timescales tend to have more projections to other units.

Original: Furthermore, we identiﬁed a set of medium-to-long timescale “controller” units which exhibit distinct and strong projections tothe gating nodes that control the state of other units, and a set of long-timescale “integrator units” which showed inﬂuence on predicting words where the long context is relevant.
Revised: Furthermore, we identiﬁed a set of medium-to-long timescale “controller” units which exhibit distinct and strong projections to control the state of other units, and a set of longtimescale “integrator units” which showed inﬂuence on predicting words where the long context is relevant.

Original: In summary, these ﬁndings advance our understanding of the timescale distribution and functional organization of LSTM language models, and provide a method for identifyingthe subset of important nodes for representing long-range contextual information.
Revised: In summary, these ﬁndings advance our understanding of the timescale distribution and functional organization of LSTM language models, and provide a method for identifying important units representing long-range contextual information in RNNs.

Original:  R ELATED W ORKLinguistic Context in LSTMs How do LSTMs encode linguistic context at multiple timescales?
Revised:  R ELATED W ORK

Revised: Linguistic Context in LSTMs . How do LSTMs encode linguistic context at multiple timescales?

Original: Prior work suggested that the units sensitive to information that requires long-range dependencies(such as number agreement) are sparse.
Revised: Prior work suggested that the units sensitive to information that requires long-range dependencies are sparse.

Original: By ablating one unit at a time, Lakretz et al.
Revised: By ablating one unit at a time, Lakretz et al.

Original: (2019) found two units that encode information required for processing long-range subject-verb number agreement (one for singular and one for plural information encoding).
Revised: (2019) found two units that encode information required for processing long-range subject-verb number agreement (one for singular and one for plural information encoding).

Original: They further identiﬁed several long-range “syntax units” whose activation was associated with syntactic tree-depth.
Revised: They further identiﬁed several long-range “syntax units” whose activation was associated with syntactic tree-depth.

Original: Overall, Lakretz et al.
Revised: Overall, Lakretz et al.

Original: (2019) suggests that a sparse subset of units tracks long-range dependencies related to subject-verb agreement and syntax.
Revised: (2019) suggests that a sparse subset of units tracks long-range dependencies related to subject-verb agreement and syntax.

Original: If this pattern is general – i.e. if there are very few nodes tracking long-range dependencies in general – this may limit the capacity of the models to process long sentences with high complexity, for reasons similar to those that may limit human sentence processing (Lakretz et al., 2020).
Revised: If this pattern is general – i.e. if there are very few nodes tracking long-range dependencies in general – this may limit the capacity of the models to process long sentences with high complexity, for reasons similar to those that may limit human sentence processing (Lakretz et al., 2020).

Original: To test whether long-range nodes are sparse in general, we require a model-free approach for mapping the context dependencies of every unit in the language network.
Revised: To test whether long-range nodes are sparse in general, we require a model-free approach for mapping the context dependencies of every unit in the language network.

Original: Whole-network context dependence .
Revised: Whole-network context dependence .

Original: Previous work by Khandelwal et al.
Revised: Previous work by Khandelwal et al.

Original: (2018) investigated the duration of prior context that LSTM language models use to support word prediction.
Revised: (2018) investigated the duration of prior context that LSTM language models use to support word prediction.

Original: Contextdependence was measured by permuting the order of words preceding the preserved context, and observing the increase in model perplexity when the preserved context gets shorter.
Revised: Contextdependence was measured by permuting the order of words preceding the preserved context, and observing the increase in model perplexity when the preserved context gets shorter.

Original: Khandelwal et al.
Revised: Khandelwal et al.

Original: (2018) found that up to 200 word-tokens of prior context were relevant to the model perplexity, but that the precise ordering of words only mattered within the most recent 50 tokens.
Revised: (2018) found that up to 200 word-tokens of prior context were relevant to the model perplexity, but that the precise ordering of words only mattered within the most recent 50 tokens.

Original: The  context-permutation method employed in this study was analogous to the approach used to measure context-dependence in human brain responses tovisual movies (Hasson et al., 2008) and to auditory language Lerner et al.
Revised: The tokenbased context-permutation method employed in this study was analogous to the approach used to measure context-dependence in human brain responses to movies (Hasson et al., 2008) and to auditory narratives (Lerner et al., 2011).

Original: Inspired by the ﬁndings of (Khandelwal et al., 2018) and (Lakretz et al., 2019), in the present study we set out to map the context-dependence across all of the individual units in the LSTM model.
Revised: Inspired by the ﬁndings of Khandelwal et al. (2018) and Lakretz et al. (2019), in the present study we set out to map the context-dependence across all of the individual units in the LSTM model.

Original: This enabled us to relate the timescales to the effects of node-speciﬁc ablation and the network architecture itself.
Revised: This enabled us to relate the timescales to the effects of node-speciﬁc ablation and the network architecture itself.

Original: In addition, our context manipulations included both context-swapping (substituting alternative meaningful contexts) and context-shufﬂing (permuting the words in the prior context to disrupt inter-word structure), which allowed us to better understand how individual words and syntactically structured word-sequences contribute to the the context representation of individual hidden units.
Revised: In addition, our context manipulations included both context-swapping (substituting alternative meaningful contexts) and context-shufﬂing (permuting the words in the prior context to disrupt inter-word structure), which allowed us to better understand how individual words and syntactically structured word-sequences contribute to the the context representation of individual hidden units.

Original:  M ETHODS 3. L ANGUAGE M ODELS AND C ORPUS
Revised:  M ETHODS 3. L ANGUAGE M ODELS AND C ORPUS

Original: We evaluated the internal representations generated by a pre-trained word-level LSTM language model (WLSTM Gulordava et al., 2018) as well as a pre-trained character-level LSTM model (CLSTM, Hahn & Baroni, 2019) as they processed sentences sampled from the 427804word/1965719-character novel corpus: Anna Karenina by Leo Tolstoy 2 3 .
Revised: We evaluated the internal representations generated by a pre-trained word-level LSTM language model (WLSTM, Gulordava et al., 2018) as well as a pre-trained character-level LSTM model (CLSTM, Hahn & Baroni, 2019) as they processed sentences sampled from the 427804-word (1965719-character) novel corpus: Anna Karenina by Leo Tolstoy (Tolstoy, 2016), translated from

Original:  Constance Garnett retrieved from http://www
Revised: Russian to English by Constance Garnett.

Original: For  WLSTM, we used the model made available by Gulordava et al.
Revised: For the WLSTM, we used the model made available by Gulordava et al.

Original: The model has a 650-dimensional embedding layer, two 650-dimensional hidden layers and an output layer with vocabulary size 50,000.
Revised: The WLSTM has a 650-dimensional embedding layer, two 650-dimensional hidden layers and an output layer with vocabulary size 50,000.

Original: The model was trained and tested on Wikipedia sentences and was not ﬁnetuned to the novel corpus.
Revised: The model was trained and tested on Wikipedia sentences and was not ﬁne-tuned to the novel corpus.

Original: Therefore, we only used sentences with low perplexity from the novel in our main timescale analysis.
Revised: Therefore, we only used sentences with low perplexity from the novel in our main timescale analysis.

Original: We did the same analysis using the Wikipedia test set from Gulordava et al.
Revised: We performed the same analysis using the Wikipedia test set from Gulordava et al.

Original: and yielded similar results (See Section 5.3, Figure A.4A, Appendix A.2.1).
Revised: and obtained similar results (See Section 5.3, Figure A.4A, Appendix A.2.1).

Original: For  CLSTM, we used the model made available by Hahn & Baroni (2019).
Revised: For the CLSTM, we used the model made available by Hahn & Baroni (2019).

Original: The model has a 200dimensional embedding layer, three 1024-dimensional hidden layers and an output layer with vocabulary size 63.
Revised: The CLSTM has a 200-dimensional embedding layer, three 1024-dimensional hidden layers and an output layer with vocabulary size 63.

Original: The model was trained on Wikipedia data with all characters lower-cased and whitespace removed.
Revised: The model was trained on Wikipedia data with all characters lower-cased and whitespace removed.

Original: We tested the model with sentences sampled from Anna Karenina as the WLSTM model, and we got bits-per-character (BPC) close to what Hahn & Baroni (2019) reported in the original paper.
Revised: We tested the model with sentences sampled from Anna Karenina as the WLSTM model, and we obtained bits-per-character (BPC) similar to what Hahn & Baroni (2019) reported in their original work.

Original: 3. T EMPORAL C ONTEXT C ONSTRUCTION P ARADIGM
Revised: 3. T EMPORAL C ONTEXT C ONSTRUCTION P ARADIGM

Original: In order to determine the processing timescales of cell state vectors and individual units, we modiﬁed the “temporal context construction” method developed by Chien & Honey (2020).
Revised: In order to determine the processing timescales of cell state vectors and individual units, we modiﬁed the “temporal context construction” method developed by Chien & Honey (2020).

Original: Thus, the internal representations of the model were compared across two conditions: (1) the Intact Context condition and (2) the Random Context condition.
Revised: Thus, the internal representations of the model were compared across two conditions: (1) the Intact Context condition and (2) the Random Context condition.

Original: In both conditions, the model was processing the same shared sequence of words (for example, segment B), but the preceding sentence differed across the two conditions.
Revised: In both conditions, the model was processing the same shared sequence of words (for example, segment B), but the preceding sentence differed across the two conditions.

Original: In the Intact Context condition, the model processed segment B (the shared segment) preceded by segment A, which was the actual preceding context from the original text.
Revised: In the Intact Context condition, the model processed segment B (the shared segment) preceded by segment A, which was the actual preceding context from the original text.

Original: In the current study, for example, segment A and B are connected by “, and” within long sentences from the novel corpus (Figure 1A), to ensure the temporal dependencies between A and B.
Revised: In the current study, for example, segment A and B are connected by “, and” within long sentences from the novel corpus (Figure 1A), to ensure the temporal dependencies between A and B.

Original: In the 
Revised: In the Random

Original: Random Context condition, however, the model processed the same shared input (segment B), but the context was replaced by segment X, which was a randomly sampled  segment from the rest of the corpus.
Revised: Context condition, however, the model processed the same shared input (segment B), but the context was replaced by segment X, which was a randomly sampled context segment from the rest of the corpus.

Original: Segment X was therefore not usually coherently related to segment B.
Revised: Segment X was therefore not usually coherently related to segment B.

Original: For the WLSTM timescale analysis, we chose long sentences in the Intact Context condition that satisﬁed the following constraints: (1) mean perplexity across all words in the sentence < 200, (2) the shared segment was longer than 25 words, and (3) the context segment was longer than 10 words.
Revised: For the WLSTM timescale analysis, we chose long sentences in the Intact Context condition that satisﬁed the following constraints: (1) mean perplexity across all words in the sentence < 200, (2) the shared segment was longer than 25 words, and (3) the context segment was longer than 10 words.

Original: 77 sentences are included as trials in our analyses.
Revised: 77 sentences are included as trials in our analyses.

Original: In the Random Context condition, we preserved the same shared segments and randomly sampled 30 context segments (each longer than 10 words) from other parts of the novel.
Revised: In the Random Context condition, we preserved the same shared segments and randomly sampled 30 context segments (each longer than 10 words) from other parts of the novel.

Original: For the CLSTM timescale analysis, we used the same 77 long sentences in the Intact
Revised: For the CLSTM timescale analysis, we used the same 77 long sentences in the Intact

Original: Context condition, and randomly sampled 25 context segments (with length > 33 characters) for the

Original: Random Context condition.

Original: In brief, the model is processing the same input (the shared segment) with different preceding context (the intact vs. random context).
Revised: In brief, the model is processing the same input (the shared segment) with different preceding context (the intact vs. random context).

Original: We can now measure the context dependence of individual units by examining how the cell state activations differ between the two conditions, while the network is processing the shared segments with identical input.
Revised: We can now measure the context dependence of individual units by examining how the cell state activations differ between the two conditions, while the network is processing the shared segments with identical input.

Original: Any difference in internal representations must arise from the context manipulation, since the current input is the same.
Revised: Any difference in internal representations must arise from the context manipulation, since the current input is the same.

Original: Decrease in activation difference over time means that the cell state activation is less and less affected by the context.
Revised: A decrease in activation difference over time implies that the units exposed in the Intact context and Random context start to build a similar representation as they process the shared input.

Original: For a long-timescale unit, whose current state is dependent on information in the far-preceding context, we will see that the activation difference is preserved across contexts (Figure 1B, green curve), even while the unit is processing the shared input.
Revised: For a long-timescale unit whose current state is dependent on information in the far-preceding context, we will see that the activation difference is preserved across contexts (Figure 1B, green curve), even while the unit is processing the shared input.

Original: On the other hand, for a short-timescale unit whose activation is driven largely by the current input, we will see that the activation difference drops quickly (Figure 1B, red curve) as the unit processes the shared input.
Revised: On the other hand, for a short-timescale unit whose activation is driven largely by the current input, we will see that the activation difference drops quickly (Figure 1B, red curve) as the unit processes the shared input.

Original:  H IERARCHICAL O RGANIZATION OF T IMESCALES A CROSS L AYERS
Revised:  H IERARCHICAL O RGANIZATION OF T IMESCALES A CROSS L AYERS

Original: Do higher levels of the LSTM model exhibit greater context-dependence?
Revised: Do higher levels of the LSTM model exhibit greater context-dependence?

Original: Lakretz et al.
Revised: Lakretz et al.

Original: (2019) observed that long-range functional units were more common in higher layers, and in general, higherlevels of hierarchical language model exhibit longer range context-dependence (Jain et al., 2019;
Revised: (2019) observed that long-range functional units were more common in higher layers, and in general, higherlevels of hierarchical language model exhibit longer range context-dependence (Jain et al., 2019;

Original: Jain & Huth, 2018).
Revised: Jain & Huth, 2018).

Original: Therefore, to validate our stimuli and the sensitivity of our methods, we ﬁrst compared the processing timescales of different hidden layers in both of the LSTMs, by correlating the cell state vectors, column by column, between the Intact condition and Random condition.
Revised: Therefore, to validate our stimuli and the sensitivity of our methods, we ﬁrst compared the processing timescales of different hidden layers in both of the LSTMs, by correlating the cell state vectors, column by column, between the Intact condition and Random condition.

Original: We found that both layers showed near-zero correlation when processing the different context and the correlation increased as they began to process the shared input.
Revised: We found that both layers showed near-zero correlation when processing the different context, and the correlation increased as they began to process the shared input.

Original: In the WLSTM, the correlation increased more slowly for second-level cell state vectors than for ﬁrst-level cell state vectors.
Revised: In the WLSTM, the correlation increased more slowly for second-level cell state vectors than for ﬁrst-level cell state vectors.

Original: Thus, the representation of second-level cell state is more sensitive to the different context than the ﬁrst level.
Revised: Thus, the representation of second-level cell state is more sensitive to the different context than the ﬁrst level.

Original: Similarly, for the CLSTM model, the third-level cell state exhibited longer-lasting context sensitivity than lower levels (Figure 2).
Revised: Similarly, for the CLSTM model, the third-level cell state exhibited longer-lasting context sensitivity than lower levels (Figure 2).

Original: This observation of longer context-dependence in higher stages of processing is consistent with prior machine learning analyses (Lakretz et al., 2019; Jain &
Revised: This observation of longer context-dependence in higher stages of processing is consistent with prior machine learning analyses (Lakretz et al., 2019; Jain &

Original: Huth, 2018) and is also analogous to what is seen in the human brain (Hasson et al., 2015; Chien & Honey, 2020; Lerner et al., 2011; Jain et al., 2019).
Revised: Huth, 2018) and is also analogous to what is seen in the human brain (Hasson et al., 2015; Chien & Honey, 2020; Lerner et al., 2011; Jain et al., 2019).

Original: Based on the ﬁnding of longer context dependence in higher layers, we examined single units in the highest level hidden units, i e. the second level of WLSTM (n=650) and the third level of CLSTM (n=1024). 
Revised: Based on the ﬁnding of longer context dependence in higher layers, we examined single units in the highest level hidden units, i.e. the second level of WLSTM (n=650) and the third level of CLSTM (n=1024). segment.

Revised: Higher-level cell states exhibited a slower increase in correlation, compared to lower-level cell states, indicating that the higher-levels retain more of the prior context information for longer.

Revised: B. As for A, but applied to the three levels of CLSTM.

Revised: Similar to the WLSTM, higher-level cell state of the CLSTM showed more context sensitivity than the lower-level cell state.

Original:  P ROCESSING T IMESCALES OF I NDIVIDUAL U NITS W ITHIN LSTM L AYERS 5. Q UANTIFYING S INGLE U NIT T IMESCALES
Revised:  P ROCESSING T IMESCALES OF I NDIVIDUAL U NITS W ITHIN LSTM L AYERS 5. Q UANTIFYING S INGLE U NIT T IMESCALES

Original: We examined the absolute single unit activation difference when processing the shared segments preceded by different context.
Revised: We examined the absolute single unit activation difference when processing the shared segments preceded by different context.

Original: As expected, most of the hidden units showed different activation when the input tokens were different (i.e. while processing the non-shared context in the Intact
Revised: As expected, most of the hidden units showed different activation when the input tokens were different (i.e. while processing the non-shared context in the Intact

Original: Context and Random Context conditions).
Revised: Context and Random Context conditions).

Original: However, once the shared input tokens begin (at t = 0 ) the Intact-Random activation differences drop (Figure A.1A, A.1B).
Revised: However, once the shared input tokens begin (at t = 0 ) the Intact-Random activation differences drop (Figure A.1A, A.1B).

Original: We used the rate at which the curves drop to quantify the processing timescale, as this is a measure of how quickly the responses align across different context conditions.
Revised: We used the rate at which the curves drop to quantify the processing timescale, as this is a measure of how quickly the responses align across different context conditions.

Original: To quantify the timescale of individual units, we ﬁt the activation difference curves with a logistic function:
Revised: To quantify the timescale of individual units, we ﬁt the activation difference curves with a logistic function:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As shown in Figure A.1A and Figure A.1B, the logistic function ﬁt the raw activation difference curves.
Revised: As shown in Figure A.1A and Figure A.1B, the logistic function ﬁt the raw activation difference curves.

Original: We then computed the ”timescale” of each unit as the time-to-half-maximum of the logistic decay.
Revised: We then computed the ”timescale” of each unit as the time-to-half-maximum of the logistic decay.

Original: In particular, for the WLSTM we used the activation difference Y (0) at the beginning of the shared segment, and at the end of the shared segment Y (24) ( Y (79) for the CSLTM) to calculate the time-to-half-maximum of unit i as:
Revised: In particular, for the WLSTM we used the activation difference Y (0) at the beginning of the shared segment, and at the end of the shared segment Y (24) ( Y (79) for the CSLTM) to calculate the time-to-half-maximum of unit i as:

Original: [Equation]
Revised: [Equation]

Original: where the inverse function Y − 1 ( y ) identiﬁes the largest integer t , for which Y ( t ) <
Revised: where the inverse function Y − 1 ( y ) identiﬁes the largest integer t , for which Y ( t ) <

Original: y .
Revised: y .

Original: We included 635 units in WLSTM and 1012 units in CLSTM for further analysis after excluding the units which could not be accurately ﬁt by a logistic function (See Appendix A.1).
Revised: We included 635 units in WLSTM and 1012 units in CLSTM for further analysis after excluding the units which could not be accurately ﬁt by a logistic function (See Appendix A.1).

Original: 5. D ISTRIBUTION OF P ROCESSING T IMESCALES INA W ORD - LEVEL LSTM 
Revised: 5. D ISTRIBUTION OF U NIT T IMESCALES IN LSTM L ANGUAGE M ODELS

Original: The results showed that of the 635 WLSTM units whose processing timescale we mapped, approximately 70% of the units were insensitive to long-range context (processing timescale < 3 words): their activation difference dropped immediately at onset of the shared segment.
Revised: The results showed that of the 635 WLSTM units whose processing timescale we mapped, approximately 70% of the units were insensitive to long-range context (processing timescale < 3 words): their activation difference dropped immediately at onset of the shared segment.

Original: In contrast, only approximately 13% of the units had a timescales > 7 words (Figure A.2A).
Revised: In contrast, only approximately 13% of the units had a timescales > 7 words (Figure A.2A).

Original: Figure 3A shows the absolute activation difference of all units in WLSTM sorted by timescale (long to short).
Revised: Figure 3A shows the absolute activation difference of all units in WLSTM sorted by timescale (long to short).

Original: Some of the longer-timescale units continued to exhibit a large activation difference even when processing the shared segments for more than 20 tokens.
Revised: Some of the longer-timescale units continued to exhibit a large activation difference even when processing the shared segments for more than 20 tokens.

Original: This suggests that some of the units have timescales longer than we measured here, consistent with previous ﬁndings that LSTM overall perplexity can be affected by up to 50 ordered context tokens (Khandelwal et al., 2018).

Original: As we were testing the same word-level LSTM previously studied by Lakretz et al.
Revised: As we were testing the same word-level LSTM previously studied by Lakretz et al.

Original: , we began by examining the timescales of hidden-state units that were already known to be involved in processing context-dependence language information: a “singular number unit” 988, a “plural number unit” 776, and a “syntax unit” 1150.
Revised: , we began by examining the timescales of hidden-state units that were already known to be involved in processing context-dependence language information: a “singular number unit” 988, a “plural number unit” 776, and a “syntax unit” 1150.

Original: We found that, compared to other units, both “number” units had medium timescales ( ∼ 3 words, ranked 129/650 units), while the “syntax” unit had a long timescale ( ∼ 7 words, ranked 64/650 units) (Figure A.1).
Revised: We found that, compared to other units, both “number” units had medium timescales ( ∼ 3 words, ranked 129 of 635 units), while the “syntax” unit had a long timescale ( ∼ 7 words, ranked 64 of 635 units) (Figure A.1).

Revised: We repeated the timescale mapping in the CLSTM model, and again identiﬁed a small subset of long-timescale units (Figure 3B, Figure A.2B).

Revised: Although there were overall more units in CLSTM, over 63% of the units were insensitive to the context (timescale < 3 characters).

Revised: Fewer than 15% of the units exhibited timescale > 10 characters, and the unit with the longest timescale only dropped to its half-maximum activation-difference after 50 characters of shared input.

Original: 5. T IMESCALE V ARIANCE ACROSS DIFFERENT D ATASETS AND C ONTEXT C ONDITIONS
Revised: 5. T IMESCALE V ARIATION ACROSS D ATASETS AND C ONTEXT C ONDITIONS

Original: To ensure that the timescales we measured are robust across datasets, we conducted the same analysis  using the Wikipedia testing dataset used in Gulordava et al.
Revised: To ensure that the timescales we measured were robust across datasets, we conducted the same analysis on WLSTM using the Wikipedia testing dataset used in Gulordava et al.

Original: The mapped timescales were highly correlated (r=0.82, p < 0.001) across the Anna Karenina dataset and the Wikipedia dataset (Appendix A.2.1, Figure A.4A).
Revised: The mapped timescales were highly correlated (r=0.82, p < 0.001) across the Anna Karenina dataset and the Wikipedia dataset (Appendix A.2.1, Figure A.4A).

Original: Similarly, to conﬁrm that the timescales measured are not speciﬁc to our testing using the “, and” conjunction point, we also measured timescales at an alternative segmentation point, and found  the timescales  largely preserved (r=0.83, p < 0.001), except for a subset of units (Appendix A.2.2, Figure A.4B).
Revised: Similarly, to conﬁrm that the timescales measured were not speciﬁc to our testing using the “, and” conjunction point, we also measured timescales at an alternative segmentation point, and found that the timescales were largely preserved (r=0.83, p < 0.001), notwithstanding there were a small set of notable exceptions (Appendix A.2.2, Figure A.4B).

Original: Although  the timescales of context dependence we measured were based on the “token distance”, they may vary under different context conditions, following the “syntactic distance”.
Revised: Although we measured the timescales of context dependence using “token distance”, these measures are not invariant to changes in the the “syntactic distance”.

Original: For example,  one would expect that the units should “reset” at the beginning of a new sentence, and show little context dependence under that condition.
Revised: For example, if one were to replace a comma with a ”full stop”, then the token distance would be unaltered but the syntactic distance could be greatly altered.

Original: Indeed, we found that most units showed little context dependence when the  context segments ended with a “full stop”, which served as a clear signal for the end of a sentence (Appendix A.2.3, Figure A.4C).
Revised: Indeed, we found that most units showed little context dependence when the preceding context segment ended with a “full stop”, which served as a clear signal for the end of a sentence (Appendix A.2.3, Figure A.4C).

Original: Finally, we examined  the role of individual words in shaping context representation and affecting the  timescales measurement.
Revised: Finally, we examined whether the contextual information retained by the language models (and the associated timescales measurement) was sensitive to linguistic structure in the context, or whether it was primarily driven simply by the presence or absence of individual words.

Original: We did this by shufﬂing the words in the Random Context condition instead of replacing it with whole segments fromother parts of the corpus.
Revised: To this end, we generated text for the Random Context condition by shufﬂing the order of words from the Intact segment.

Original: We found that while  individual words  play an important role in shaping the context dependence of unit timescale, several units showed a longer timescale when that prior context is composed of coherently structured language (Appendix A.2.4, Figure A.4D).
Revised: We found that while the presence of individual words did play an important role in determining the context representations (and thus the timescales), several units showed a longer timescale when the prior context was composed of coherently structured language (Appendix A.2.4, Figure A.4D).

Original: 5. D ISTRIBUTION OF P ROCESSING T IMESCALES IN A C HARACTER - LEVEL LSTM

Original: We repeated the timescale mapping in the CLSTM model, and again identiﬁed a small subset of long-timescale units (Figure 3B, Figure A.2B).

Original: Although there are overall more units in CLSTM, over 63% of the units are insensitive to the context (timescale < 3 characters).

Original: Fewer than 15% of the units exhibited timescale > 10 characters, and the unit with the longest timescale only dropped to its half-maximum activation-difference after 50 characters of shared input.

Original:  C ONNECTIVITY OFU NITS WITH L ONG AND M EDIUM P ROCESSING T IMESCALES 
Revised:  C ONNECTIVITY OF M EDIUM - TO L ONG -T IMESCALES U NITS IN LSTM S

Original: Having mapped the timescales of each processing units, we next asked: Does the processing timescale  relate to the functional role of each unit in the network?
Revised: Having mapped the timescales of each processing unit, we next asked: how does the processing timescale of a unit relate to its functional role within the network?

Original: More speciﬁcally, are units with longer timescales also units with high degree in the connectivity network?
Revised: More speciﬁcally, are units with longer timescales also units with high degree in the connectivity network?

Original: To answer these questions, we analyzed (1) the projection strength of each unit and (2) the similarity of the overall projection pattern (hidden-to-gates) across different units, using the direct weight projections from one hidden unit at time t to the input and forget gate of other hidden units at time t + 1 .
Revised: To answer these questions, we analyzed (1) the projection strength of each unit and (2) the similarity of the overall projection pattern (hidden-to-gates) across different units. The projection patterns were deﬁned using the direct weight projections from one hidden unit at time t to the input and forget gate of other hidden units at time t + 1 .

Original: In LSTMs, the amount of contextual ( c t − 1 ) and input ( ˜ c t ) information stored in the cell state ( c t ) is determined by the forget gate ( f t ) and input gate ( i t ) activation (Eq. 3); and the activation of the gates i t and f t are determined by the current input at time t and the hidden units at time t − 1 through weight matrices U and W (Eq. 4, 5).
Revised: In LSTMs, the amount of contextual ( c t − 1 ) and input ( ˜ c t ) information stored in the cell state ( c t ) is determined by the forget gate ( f t ) and input gate ( i t ) activation (Eq. 3); and the activation of the gates i t and f t are determined by the current input at time t and the hidden units at time t − 1 through weight matrices U and W (Eq. 4, 5).

Original: [Equation]
Revised: [Equation]

Original: Here, we are interested in understanding how the contextual information over different timescales is projected from the hidden units to the input and forget gates of other units, and further inﬂuence the update of cell states.
Revised: Here, we were interested in understanding how the contextual information over different timescales is projected from the hidden units to the input and forget gates of other units, and further inﬂuence the update of cell states.

Original: Thus, we analyzed the network connectivity focusing on the weight matrices
Revised: Thus, we analyzed the network connectivity focusing on the weight matrices

Original: W i and W f within the highest layer of the WLSTM or CLSTM.
Revised: W i and W f within the highest layer of the WLSTM or CLSTM.

Original: 6. S TRONG P ROJECTIONS FROM L ONG -T IMESCALE H IDDEN U NITS TO G ATE U NITS
Revised: 6. S TRONG P ROJECTIONS FROM L ONG -T IMESCALE H IDDEN U NITS TO G ATE U NITS

Original: Units with longer processing timescales made a larger number of strong projections ( | z-score | > 5, Appendix A.3) to the input and forget gates of other units in both WLSTM (r=0.31, p < 0.001, Figure 4A) and CLSTM models (r=0.24, p < 0.001, Figure A.5A).
Revised: Units with longer processing timescales made a larger number of strong projections ( | z-score | > 5, Appendix A.3) to the input and forget gates of other units in both WLSTM (r=0.31, p < 0.001, Figure 4A) and CLSTM models (r=0.24, p < 0.001, Figure A.5A).

Original: Further, we found that the “syntax” unit (Unit 1150) reported by Lakretz et al.
Revised: Furthermore, we found that the “syntax” unit (Unit 1150) reported by Lakretz et al.

Original: (2019) in  WLSTM  possessed the largest number of strong projections to the input and forget gates of all other units, and the target units that Unit 1150 strongly project to are units with medium- to long-timescale units (Figure 4B).
Revised: (2019) in the WLSTM model possessed the largest number of strong projections to the input and forget gates of all other units, and the major recipients from Unit 1150 were units with medium- to long-timescale units (Figure 4B).

Original: 6.  C ONTROLLER U NITS INA W ORD - LEVEL LSTM AND A C HARACTER - LEVEL LSTM 
Revised: 6. I DENTIFY C ONTROLLER U NITS IN LSTM L ANGUAGE M ODELS

Original: The presence of strong projections from the “syntax” unit to other long-timescale units motivated us to further explore whether high-degree, long-timescale units in the LSTM also densely interconnect to form a “core network”, as seen in the brain (Hagmann et al., 2008; Mesulam, 1998; Baria et al., 2013).
Revised: The presence of strong projections from the “syntax” unit to other long-timescale units motivated us to further explore whether high-degree, long-timescale units in the LSTM also densely interconnect to form a “core network”, perhaps analogous to what is seen in the brain (Hagmann et al., 2008; Mesulam, 1998; Baria et al., 2013).

Original: If so, this set of units may have an especially important role in determining how prior context is updated and how it is used to gate current processing, analogous to the controller system in the brain (Gu et al., 2015).
Revised: If so, this set of units may have an especially important role in controlling how prior context is updated and how it is used to gate current processing, analogous to the controller system in the brain (Gu et al., 2015).

Original: To identify such “controller units”, we binarized the network by identifying the top 258 projection weights from the weight matrices (see Appendix A.3), which provided the edges for a network analysis.
Revised: To identify these putative “controller units”, we binarized the network by identifying the top 258 projection weights from the weight matrices (see Appendix A.3), which provided the edges for a network analysis.

Original: We then used k-core analysis (Batagelj & Zaversnik, 2003) to identify the “main network core” (the core with the largest degree) of the network (Figure A.3).
Revised: We then used k-core analysis (Batagelj & Zaversnik, 2003) to identify the “main network core” (the core with the largest degree) of the network (Figure A.3).

Original: At the maximal k = 5 , the k-core analysis yielded a set of densely interconnected nodes, composed of many long-timescale and medium-timescale units (Figure A.3), also labeled in red in Figure 4A).
Revised: At the maximal k = 5 , the k-core analysis yielded a set of densely interconnected nodes, composed of many long-timescale and medium-timescale units (Figure A.3), also labeled in red in Figure 4A).

Original: We (tentatively) refer to this set as the “controller” set of the network.
Revised: We (tentatively) refer to this set as the “controller” set of the network.

Original: We performed the same k-core analyses on the CLSTM and observedsimilar results that the main core network was  composed of disproportionately many medium and long-timescale “controller” units (Figure A.5A). 
Revised: Performing the same k-core analyses on the CLSTM model, we observed that the main core network was again composed of disproportionately many medium and long-timescale “controller” units (Figure A.5A). results in different impact to the LSTM performance.

Revised: Speciﬁcally, ablating “controller” units impaired overall word prediction (upper panel), while ablating “integrator” units impaired prediction of words in the later part of the sentences (bottom panel).

Revised: D. Multi-dimensional scaling representation of network connectivity.

Original:  6. P ROJECTION SIMILARITY OF  C ONTROLLER AND I NTEGRATOR U NITS
Revised: The distance between two nodes indicates the similarity of their hidden-to-gate connection patterns. The size of each node indicates its degree (the number of strong projections from that node to the gate units). An edge between nodes indicates a signiﬁcant hidden-to-gate projection between them. 6. D ISTINCTIVE R OLES OF L ONG - TIMESCALE C ONTROLLER AND I NTEGRATOR U NITS

Original: We used multi-dimensional scaling (MDS) to visualize the similarity of projection patterns across
Revised: We used multi-dimensional scaling (MDS) to visualize the similarity of projection patterns across

Original: LSTM units.
Revised: LSTM units.

Original: We recovered a 2-dimensional MDS embedding, where the inter-unit distances was deﬁned based on the similarity of their hidden-to-gate projection patterns (i.e.,  values in the unthresholded LSTM weight matrices W i and W f ).
Revised: We recovered a 2-dimensional MDS embedding, in which the inter-unit distances was deﬁned based on the similarity of their hidden-to-gate projection patterns (i.e., similarity of values in the unthresholded LSTM weight matrices W i and W f ).

Original: We visualized the MDS solution as a graph structure, in which each node is a unit, and the edges reﬂect connection properties of that unit.
Revised: We visualized the MDS solution as a graph structure, in which each node is a unit, and the edges reﬂect connection properties of that unit.

Original: Figure 4D shows the resulting 2-D space, with units color-coded by their timescale.
Revised: Figure 4D shows the resulting 2-D space, with units color-coded by their timescale.

Original: The size of the nodes indicates the number of strong projections from the node, and the lines indicate those projections (to forget gate and input gate, based on the color of the line).

Original: “Controller units” (labeled on Figure 4D) were positioned around the periphery of the MDS space, suggesting that these units have distinct projection patterns, both from other “controller” units and  from the rest of the network.
Revised: “Controller units” (labeled on Figure 4D) were positioned around the periphery of the MDS space, suggesting that these units expressed projection patterns that were distinct from other “controller” units and also from the rest of the network.

Original: In contrast, we observed several long-timescale units positioned in the center of the MDS space, suggesting that the projection patterns of these units are close to the mean projection pattern.
Revised: In contrast, we observed several long-timescale units positioned in the center of the MDS space, suggesting that the projection patterns of these units were similar to the mean projection pattern.

Original: We refer to this more MDS-central set as the “integrator units” (labeled in green in Figure 4A).
Revised: We refer to this more MDS-central set as the “integrator units” (labeled in green in Figure 4A).

Original: Similar to the WLSTM, the projection patterns of the “controller units” in  CLSTM were distinct from other units in the network, according to the MDS results (Figure A.5C).
Revised: Similar to the WLSTM, the projection patterns of the “controller units” in the CLSTM were distinct from other units in the network, according to the MDS results (Figure A.5C).

Original: However, we did not observethe “integrator units” positioned in the center of the MDS space of the CLSTM as was observed in the WLSTM.
Revised: However, we did not observe “integrator units” positioned in the center of the MDS space of the CLSTM.

Original: 6. R OLES OF C ONTROLLER AND I NTEGRATOR U NITS

Original: Are the “controller” and “integrator” units  important for the model’s ability to predict the next token?
Revised: Are the “controller” and “integrator” units particularly important for the model’s ability to predict the next token?

Original: To test the functional importance of these subsets of units, we conducted group ablation analyses (See Appendix A.5).
Revised: To test the functional importance of these subsets of units, we conducted group ablation analyses (See Appendix A.4).

Original: Ablating controller units reduced the accuracy of token prediction overall, while ablating integrator units only reduced prediction accuracy for the last words of the sentences (Figure 4C).
Revised: Ablating controller units reduced the accuracy of token prediction overall, while ablating integrator units only reduced prediction accuracy for the last words of the sentences (Figure 4C).

Original: The results conﬁrm that the putative controller and integrator nodes are functionally signiﬁcant, with distinctive roles in the WLSTM language model.
Revised: The results conﬁrm that the putative controller and integrator nodes are functionally signiﬁcant, with distinctive roles in the WLSTM language model.

Original: Finally, to test the generalization of the timescale and connectivity analyses to a different model architecture, we conducteda preliminary analysis of a Gated Recurrent Unit (GRU) language model (Cho et al., 2014).
Revised: Finally, to test the generalization of the timescale and connectivity analyses to a different model architecture, we conducted preliminary analyses on a Gated Recurrent Unit (GRU) language model (Cho et al., 2014) and another word-level LSTM model with a smaller hidden size (100 units) per layer.

Original: These GRU results are preliminary, because the model was not fully trained, and its perplexity was an order of magnitude inferior to the LSTM models.
Revised: The models were trained using similar parameter settings as in Gulordava et al. (2018) until they converged without any model-speciﬁc optimization.

Original: We found similar sparsity of long-timescale units in the GRU, but did not observe the same relationship between timescales and connectivity (Appendix A.4; Figure A.6; A.7).
Revised: We found similar sparsity of longtimescale units in both models, but did not observe the same relationship between timescales and connectivity (Appendix A.5; A.6; Figure A.7; A.8; A.9; A.10).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We demonstrated a new method for mapping the timescale organization in recurrent neural language models.
Revised: We demonstrated a new method for mapping the timescale organization in recurrent neural language models.

Original: Using this method, we mapped the timescale distributions of units within word-level and character-level LSTM language models and identiﬁed a small set of units with long timescales.
Revised: Using this method, we mapped the timescale distributions of units within word-level and character-level LSTM language models, and identiﬁed a small set of units with long timescales.

Original: We then used network analyses to understand the relationship between the timescale of a unit and its connectivity proﬁle and we distinguished two subsets of long-timescale units with seemingly different functions.
Revised: We then used network analyses to understand the relationship between the timescale of a unit and its connectivity proﬁle, and we distinguished two subsets of long-timescale units with seemingly distinctive functions.

Original: Altogether, we proposed methods combining timescale and connectivity analyses for discovering timescale and functional organization in language models.
Revised: Altogether, we proposed methods combining timescale and connectivity analyses for discovering timescale and functional organization in language models.

Original: The units with long processing timescales included some units whose role in long-range language dependencies had already been established (Lakretz et al., 2019) but almost all of the long timescale units are of unknown function.
Revised: The units with longer processing timescales included some units whose role in long-range language dependencies had already been established (Lakretz et al., 2019), but almost all of the long timescale units are of unknown function.

Original: The timescale mapping procedure described here provides a modelfree method for identifying nodes necessary for long-range linguistic and discursive processes (e.g. tracking whether a series of words constitutes an assertion or a question).
Revised: The timescale mapping procedure described here provides a model-free method for identifying nodes necessary for long-range linguistic and discursive processes (e.g. tracking whether a series of words constitutes an assertion or a question).

Original: Future studies of these neural language models could focus on the functions of long-timescale units,and especially the “controller” units, to understand why each unit is tracking long-range information, and how they control the information ﬂow of other units in the network. 
Revised: Future studies of these neural language models could focus on the speciﬁc linguistic information tracked by the long-timescale units, especially the “controller” units which control the information ﬂow of other units in the network. The current study measured unit timescales using a simple token distance, and so the method may be applied to understanding recurrent neural nets beyond language models.

Original: Moreover, it will be important to investigate whether the processing timescales characterized via token distance(as demonstrated in the current study) are comparable to those measured using functional measures, such as syntactic distance.
Revised: It will be insightful for future studies to investigate whether the processing timescales characterized via token distance are comparable to those measured using functional measures, such as syntactic distance.

Original: Relatedly, while we explored the timescale variance under several context conditions, a more thorough investigation will be needed to examine how the timescales of individual units may vary at different positions within a sentence, both in terms of token location and syntactic location.
Revised: Relatedly, while we explored the timescale variance under several context conditions, a more thorough investigation will be needed to examine how the timescales of individual units may vary at different positions within a sentence, both in terms of token location and syntactic location.

Original: Processing timescales may exhibit an analogous hierarchical organization in LSTMs and in the human cerebral cortex: in both cases, a subset of nodes with high degree and high inter-connectivity express unusually long timescales.
Revised: Processing timescales may exhibit an analogous hierarchical organization in LSTMs and in the human cerebral cortex: in both cases, a subset of nodes with high degree and high inter-connectivity express unusually long timescales.

Original: More detailed testing of this apparent correspondence is required, however, because units within an LSTM layer are not spatially embedded, and thus may not show a spatially graded timescale topography. 
Revised: More detailed testing of this apparent correspondence is required, however, because units within an LSTM layer are not spatially embedded and constrained as in biological brains, and thus the LSTM units do not express a spatially graded timescale topography. A CKNOWLEDGMENTS

Revised: C.J.H and H-Y.S.C gratefully acknowledge the support of the National Institutes of Mental Health (grant R01MH119099)

Original: A A PPENDIX A. U NITS EXCLUDED FROM TIMESCALE ANALYSIS
Revised: A A PPENDIX A. U NITS EXCLUDED FROM TIMESCALE ANALYSIS

Original: We excluded 1 unit in the WLSTM model and 5 units in CLSTM model which were not properly ﬁt using the logistic function; we further excluded 14 units in the WLSTM model and 7 units in the
Revised: We excluded 1 unit in the WLSTM model and 5 units in CLSTM model which were not properly ﬁt using the logistic function; we further excluded 14 units in the WLSTM model and 7 units in the

Original: CLSTM model which either did not show a non-zero activation difference before the shared segment started, or whose activation differences increased when started to process the shared segment.
Revised: CLSTM model which either did not show a non-zero activation difference before the shared segment started, or whose activation differences increased when started to process the shared segment.

Original: After these exclusions, 635 units remained in the WLSTM and 1012 units remained in the CLSTM for further analysis.
Revised: After these exclusions, 635 units remained in the WLSTM and 1012 units remained in the CLSTM for further analysis.

Original: A. T IMESCALE ANALYSES ACROSS DIFFERENT DATASETS AND CONTEXT CONDITIONS A.2. W IKIPEDIA TEST DATASET
Revised: A. T IMESCALE ANALYSES ACROSS DIFFERENT DATASETS AND CONTEXT CONDITIONS A.2. W IKIPEDIA TEST DATASET

Original: The Anna Karenina corpus used in the current study has a different linguistic structure from the
Revised: The Anna Karenina corpus used in the current study has a different linguistic structure from the

Original: Wikipedia corpus on which the WLSTM and CLSTM models were trained.
Revised: Wikipedia corpus on which the WLSTM and CLSTM models were trained.

Original: Although we analyzed only the Anna Karenina sentences with low perplexity, it wsa important to test the robustness of our results across datasets.
Revised: Although we analyzed only the Anna Karenina sentences with low perplexity, it was important to test the robustness of our results across datasets.

Original: Thus, we mapped the timescale of each unit using the Wikipedia test set, as used by Gulordava et al.
Revised: Thus, we mapped the timescale of each unit using the Wikipedia test set, as used by Gulordava et al.

Original: Speciﬁcally, we sampled 500 long sentences containing “, and” for the Intact Context condition.
Revised: Speciﬁcally, we sampled 500 long sentences containing “, and” for the Intact Context condition.

Original: As before, we generated sentences by preceding the “shared input” segment (after the conjunction) with either the original prior context segment, or a randomly chosen prior context segment.
Revised: As before, we generated sentences by preceding the “shared input” segment (after the conjunction) with either the original prior context segment, or a randomly chosen prior context segment.

Original: Same as the original analysis, we then replaced the context segment with 30 context segments randomly sampled from other parts of the test set for generating the Random
Revised: Same as the original analysis, we then replaced the context segment with 30 context segments randomly sampled from other parts of the test set for generating the Random

Original: Context condition.
Revised: Context condition.

Original: The mapped timescales using the Wikipedia test set were highly correlated with the novel corpus, suggesting the robustness of unit timescales (Figure A.4A).
Revised: The mapped timescales using the Wikipedia test set were highly correlated with the novel corpus, suggesting the robustness of unit timescales (Figure A.4A).

Original: A.2. T IMESCALES MEASURED IN THE MIDDLE OF A SENTENCE
Revised: A.2. T IMESCALES MEASURED IN THE MIDDLE OF A SENTENCE

Original: To examine how the timescales of individual units may vary across different positions in a sentence, we varied the location of the segmentation point.
Revised: To examine how the timescales of individual units may vary across different positions in a sentence, we varied the location of the segmentation point.

Original: Instead of using the conjunction (“, and”) as the segmentation point, we chose an arbitrary segmentation point: the 15th token of a long sentence, to separate context segment and shared input segment.
Revised: Instead of using the conjunction (“, and”) as the segmentation point, we chose an arbitrary segmentation point: the 15th token of a long sentence, to separate context segment and shared input segment.

Original: In the Random Context condition, we replaced the context segment with the ﬁrst 15 tokens from other sentences of the corpus.
Revised: In the Random Context condition, we replaced the context segment with the ﬁrst 15 tokens from other sentences of the corpus.

Original: We found that the unit timescales were highly correlated with the condition where we used the conjunction as the segmentation point with several units shift their timescales to either directions (Figure A.4B).
Revised: We found that the unit timescales were highly correlated with the condition where we used the conjunction as the segmentation point with several units shift their timescales to either directions (Figure A.4B).

Original: This analysis was conducted using Wikipedia test set.
Revised: This analysis was conducted using Wikipedia test set.

Original: A.2. T IMESCALE RESET AT THE BEGINNING OF A SENTENCE
Revised: A.2. T IMESCALE RESET AT THE BEGINNING OF A SENTENCE

Original: To examine if the timescales of individual units can ﬂexibly reset at the beginning of a sentence, we conducted the same timescale analysis but using a “full stop” as the segmentation point instead of the conjunction “, and”.
Revised: To examine if the timescales of individual units can ﬂexibly reset at the beginning of a sentence, we conducted the same timescale analysis but using a “full stop” as the segmentation point instead of the conjunction “, and”.

Original: Thus, if the original test string was “The girl kicked the call, and the boy caught it”, then the full-stop  test string would be “The girl kicked the ball.
Revised: Thus, if the original test string was “The girl kicked the call, and the boy caught it”, then the full-stop version of the test string would be “The girl kicked the ball.

Original: The boy caught it.”
Revised: The boy caught it.”

Original: In this setting, the context segment and shared input segment in the Intact Context condition are two consecutive sentences.
Revised: In this setting, the context segment and shared input segment in the Intact Context condition are two consecutive sentences.

Original: To ensure the temporal dependence between the context segment and shared input segment, we sampled 100 consecutive sentence pairs from the Anna Karenina corpus.
Revised: To ensure the temporal dependence between the context segment and shared input segment, we sampled 100 consecutive sentence pairs from the Anna Karenina corpus.

Original: Note that This is not possible using the Wikipedia test set from Gulordava et al.
Revised: Note that this is not possible using the Wikipedia test set from Gulordava et al.

Original: (2018), which is composed of unrelated sentences.
Revised: (2018), because that set is composed of unrelated sentences.

Original: The Random Context condition was generated by replacing the ﬁrst sentence with randomly sampled sentences from other parts of the novel.
Revised: The Random Context condition was generated by replacing the ﬁrst sentence with randomly sampled sentences from other parts of the novel.

Original: We found that when using “full stop” to segment context and shared input, most units in the network showed timescale near 0, indicating near-zero dependence on the linguistic context from the text preceding the full stop (Figure A.4C).
Revised: We found that when using “full stop” to segment context and shared input, most units in the network showed timescale near 0, indicating near-zero dependence on the linguistic context from the text preceding the full stop (Figure A.4C).

Original: This suggests that the units in LSTM tend to “reset” their context representation at the beginning of a sentence.
Revised: This suggests that the units in LSTM tend to “reset” their context representation at the beginning of a sentence.

Original: A.2. C ONTEXT REPRESENTATION SHAPED BY INDIVIDUAL WORDS
Revised: A.2. C ONTEXT REPRESENTATION SHAPED BY INDIVIDUAL WORDS

Original: Inspired by the token-shufﬂing procedure of Khandelwal et al.
Revised: Inspired by the token-shufﬂing procedure of Khandelwal et al.

Original: , we explored whether the context representations of individual units in the LSTM were shaped by individual words, rather than coherent sequences of words.
Revised: , we explored whether the context representations of individual units in the LSTM were shaped by individual words, rather than coherent sequences of words.

Original: For this analysis, instead of replacing the context with syntactically structured segments from other part of the corpus, we generated the “random context” by shufﬂing the order of words within the context segment.
Revised: For this analysis, instead of replacing the context with syntactically structured segments from other part of the corpus, we generated the “random context” by shufﬂing the order of words within the context segment.

Original: We then mapped the unit timescales as before, by examining the unit activation difference as a function of the distance from the onset of shared input.
Revised: We then mapped the unit timescales as before, by examining the unit activation difference as a function of the distance from the onset of shared input.

Original: Intriguingly, we found that most of the units showed similar timescales across the context-replacement and context-shufﬂing procedures (Figure A.4D).
Revised: Intriguingly, we found that most of the units showed similar timescales across the context-replacement and context-shufﬂing procedures (Figure A.4D).

Original: This suggests that the context representations in LSTMs largely depend on the presence of individual words in the context, rather than their appearance within coherent linguistic sequences.
Revised: This suggests that the context representations in LSTMs largely depend on the presence of individual words in the context, rather than their appearance within coherent linguistic sequences.

Original: However, we did observe a subset of units (labeled in the Figure, and almost all long-timescale units) whose timescales were longer when context was replaced rather than shufﬂed.
Revised: However, we did observe a subset of units (labeled in the Figure, and almost all long-timescale units) whose timescales were longer when context was replaced rather than shufﬂed.

Original: For this subset of units, the ability to maintain a representation of prior context over many tokens depends on that prior context being a coherent linguistic sequence.
Revised: For this subset of units, the ability to maintain a representation of prior context over many tokens depends on that prior context being a coherent linguistic sequence.

Original: This subset of units are a promising target for future studies of syntactic representations in LSTMs.
Revised: This subset of units are a promising target for future studies of syntactic representations in LSTMs.

Original: A. I DENTIFYING STRONG HIDDEN - TO - GATE PROJECTIONS
Revised: A. I DENTIFYING STRONG HIDDEN - TO - GATE PROJECTIONS

Original: First, for each hidden unit, we concatenated the corresponding rows in the W hi and W hf matrices, to generate a single “hidden-to-gate” projection vector for that hidden unit.
Revised: First, for each hidden unit, we concatenated the corresponding rows in the W hi and W hf matrices, to generate a single “hidden-to-gate” projection vector for that hidden unit.

Original: Next we we z-scored the vector to get standardized projection values from that unit to all other units in the network.
Revised: Next we we z-scored the vector to get standardized projection values from that unit to all other units in the network.

Original: Using | z-score | > 5 as criterion, we identiﬁed a total of 258 “strong projections” from all hidden units to the input gate and forget gate in the WLSTM.
Revised: Using | z-score | > 5 as criterion, we identiﬁed a total of 258 “strong projections” from all hidden units to the input gate and forget gate in the WLSTM.

Original: The projection strength of each unit was then calculated based on its number of ”strong projections” (Figure 4A).
Revised: The projection strength of each unit was then calculated based on its number of ”strong projections” (Figure 4A).

Original: Although the criterion | zscore | > was selected to better visualize the results in Figure 4, different criteria did not change the results that units with longer timescales have more strong projections.
Revised: Although the criterion | zscore | > was selected to better visualize the results in Figure 4, different criteria did not change the results that units with longer timescales have more strong projections.

Original: For example, using | z-score | > 3 as threshold we obtained corr(timescale, projections) = 0.30, p < 0.001; | z-score | > 4 we obtained corr(timescale, projections) = 0.35, p < 0.001.
Revised: For example, using | z-score | > 3 as threshold we obtained corr(timescale, projections) = 0.30, p < 0.001; | z-score | > 4 we obtained corr(timescale, projections) = 0.35, p < 0.001.

Original: Next, we identiﬁed the edges corresponding to the top 258 magnitude weight-values within the combined W hi and W hf matrices.
Revised: Next, we identiﬁed the edges corresponding to the top 258 magnitude weight-values within the combined W hi and W hf matrices.

Original: Together, these edges formed a ”strong-projection network”.
Revised: Together, these edges formed a ”strong-projection network”.

Original: Finally, we used k-core analysis to identify the main core of the strong-projection network.
Revised: Finally, we used k-core analysis to identify the main core of the strong-projection network.

Original: This main core composed our ”controller units” (Figure A.3).
Revised: This main core composed our ”controller units” (Figure A.3).

Original: Using the same criteria and method, we identiﬁed a total of 390 “strong projections” from all hidden units to the input gate and forget gate in the CLSTM.
Revised: Using the same criteria and method, we identiﬁed a total of 390 “strong projections” from all hidden units to the input gate and forget gate in the CLSTM.

Original: We then extracted the top 390 weight values from the weight matrices to construct a “strong-projection network” and again identiﬁed the main core network, composed the “controller units” for the CLSTM model (Figure A.5A, A.5B) 
Revised: We then extracted the top 390 weight values from the weight matrices to construct a “strong-projection network” and again identiﬁed the main core network, composed the “controller units” for the CLSTM model (Figure A.5A, A.5B) A. A BLATION ANALYSES ON PUTATIVE CONTROLLER AND INTEGRATOR UNITS

Revised: To examine the non-trivial roles of the controller and integrator units identiﬁed in the word-level

Original: A. M APPING TIMESCALE ORGANIZATION IN A GRU LANGUAGE MODEL A.4. T RAINING
Revised: LSTM model, we performed a preliminary group ablation analysis to look at how ablating the controller units inﬂuences model performance on predicting the next token, relative to the ablation of a random set of units. Speciﬁcally, since long-timescale integrator units should have most effect predicting tokens at the later part of the sentences (i.e., when more context is integrated), we examined the model performance on predicting tokens at two different positions: (1) all the tokens regardless of their positions in the sentences (“All tokens” condition), and (2) the last tokens of sentences (“Final tokens” condition). We evaluated the effects of ablation on model performance by measuring the differences of probabilities ( ∆ P ) assigned to the target words ( ∆ P = probability of target word in ablated model minus probability of target word in original model). Ablation effects for controller units ( N =9) and integrator units ( N =10) were compared against a baseline of ablating the same number of randomlyselected units from layer 2 of the LSTM (Figure 4C).

Revised: We used the test corpus used by Gulordava et al.

Revised: and measured the average performance of each model across 100 text-batches, randomly sampled from the Wikipedia test dataset.

Revised: Each text-batch was composed of 1000 tokens that start at the beginning of a sentence.

Revised: In the “All tokens” condition, we calculated the ∆ P for every token in the tested text, while in the “Final tokens” condition, we calculated ∆ P only at the last token of every sentence (i.e. the token right before the full stop“.” of each sentence).

Revised: We then average the ∆ P in both conditions across text-batches to get a mean performance difference between the ablated model and the intact model.

Revised: Ablating controller units reduced the probabilities assigned to the target words, more so than ablating random units (Figure 4C, controller vs. random across 100 text batches: Cohen’s d

Revised: = -4.85, t = - 34.28, p < 0.001).

Revised: In contrast, ablating integrator units reduced the probabilities less than ablating random units (integrator vs. random:

Revised: Cohen’s d = 2.50, t = 17.67, p < 0.001).

Revised: We hypothesized that that the integrator units mostly inﬂuence the model’s prediction performance for tokens where long-range information is especially relevant, such as in the later portions of clauses and sentences.

Revised: Consistent with this, we found that, when we examined the ablation effects only for tokens in the ﬁnal position of a sentence, ablating integrator units reduced the probabilities more than ablating random units (Cohen’s d = -0.34, t = -2.41, p = 0.017).

Revised: Interestingly, ablating controller units reduced the probability of sentence-ﬁnal targets less than random units (Cohen’s d = 0.67, t = 4.74, p < 0.001).

Revised: In summary, these ablation results indicate a non-trivial functional role for the controller and integrator units, despite the fact that each subset of units is composed of only 10 amongst 650 total hidden units.

Revised: Also, the putative controller and integrator sets appear to have distinctive roles within the WLSTM, with the controllers supporting accurate predictions overall, while the integrator units appear to boost accurate predictions at the end of sentences.

Revised: A. M APPING THE TIMESCALE ORGANIZATION IN A GRU LANGUAGE MODEL A.5. T RAINING

Original: To explore whether the timescale mapping methods, and our ﬁndsings, may generalize to other model architectures, we trained and studied a word-level GRU language model (Cho et al., 2014).
Revised: To explore whether the timescale mapping methods, and our ﬁndings, may generalize to other model architectures, we trained and studied a word-level GRU language model (Cho et al., 2014).

Original: As far as possible, we applied the same parameters in the GRU as were used for the LSTM by Gulordava et al.
Revised: As far as possible, we applied similar parameters in the GRU as were used for the LSTM by Gulordava et al.

Original: (2018): the same Wikipedia training corpus, the same loss function (i.e. crossentropy loss), and the same hyperparameters.
Revised: (2018): the same Wikipedia training corpus, the same loss function (i.e. cross-entropy loss), and the same hyperparameters except for a learning rate initialized to 0.1, which we found more optimal to train the GRU.

Original: Thus, the GRU model also had two layers, withhidden units in each layer.
Revised: The GRU model also had two layers, with 650 hidden units in each layer.

Original: Due to limitations of time and computational resources, we had to stop training the GRU model after 48 hours (10.5 epochs), at which point the GRU achieved a test perplexity of 349.39.
Revised: We trained the GRU model for 30 epochs, at which point the GRU converged to a validation perplexity of 118.36. Note that since we adapted similar training settings as were used for training the LSTM model by Gulordava et al.

Original: This perplexity ismuch higher than that of the LSTM model reported in Gulordava et al.
Revised: without model-speciﬁc optimization, the perplexity is higher than that of the LSTM model reported in Gulordava et al.

Original: (perplexity = 52.1 in the English corpora, after training for 40 epochs and selecting the model with the lowest perplexity out of 68 combinations of different hyperparamters).
Revised: (perplexity = 52.1 in the English corpora, after training for 40 epochs and selecting the model with the lowest perplexity out ofcombinations of different hyperparamters).

Original: In spite of the fact that the GRU model was sub-optimal, we analyzed the timescale of its hidden units using the same method as was used for analyzing the LSTMs, and using the test data derived from the training Wikipedia corpus.
Revised: We then analyzed the timescale of its hidden units using the same method as was used for analyzing the LSTMs, and using the test data derived from the training Wikipedia corpus.

Original: A.4. T IMESCALE ORGANIZATION OF A GRU MODEL
Revised: A.5. T IMESCALE ORGANIZATION OF A GRU MODEL

Original: Similar to the LSTM model of Gulordova et al, the majority of the units in the GRU also showed shorter timescales.
Revised: Similar to the LSTM model of Gulordova et al, the majority of the units in the GRU also showed shorter timescales.

Original: More speciﬁcally, we found: (1) the second layer of the GRU model was more sensitive to prior context than the ﬁrst layer, as in the LSTM (A.6A); (2) the distribution of timescales across units was similar in the GRU and LSTM, although the GRU showed a more right-skewed distribution with a larger proportion of short-timescale units (A.6B, C).
Revised: More speciﬁcally, we found: (1) the second layer of the GRU model was more sensitive to prior context than the ﬁrst layer, as in the LSTM (A.7A); (2) the distribution of timescales across units was similar in the GRU and LSTM, although the GRU showed a more right-skewed distribution with a larger proportion of short-timescale units (A.7B, C).

Original: A.4. T IMESCALE VERSUS NETWORK CONNECTIVITY IN A GRU MODEL
Revised: A.5. T IMESCALE VERSUS NETWORK CONNECTIVITY IN A GRU MODEL

Original: We also performed the timescale vs. network connectivity analyses on the GRU model.
Revised: We also performed the timescale vs. network connectivity analyses on the GRU model.

Original: Because the update of hidden states in GRU are controlled by the  update gate, we measured the projection patterns of hidden units by analyzing the matrix of  hidden-to-update-gate  weights.
Revised: Because the update of hidden states in GRU are controlled by the reset and update gate, we measured the projection patterns of hidden units by analyzing the matrix of combined hidden-to-update-gate and hiddento-reset-gate weights.

Original: In contrast to the LSTM models, hidden units in the GRU that we trained did not show a relationship between longer timescales and stronger hidden-to-gate projections (A.7A).
Revised: In contrast to the LSTM models, hidden units in the GRU that we trained did not show a relationship between longer timescales and stronger hidden-to-gate projections (A.8A).

Original: Moreover, when using k-core analysis to identify subunits of interconnected high-degree units, the core network in the GRU contained many units with long to short timescales.
Revised: Moreover, when using k-core analysis to identify subunits of interconnected high-degree units, the core network in the GRU contained many units with long to short timescales.

Original: Interestingly, when we visualized the position of the k-core units in the MDS space, they tended to locate at the edge of the space, similar to what we found in LSTM.
Revised: Interestingly, when we visualized the position of the k-core units in the MDS space, they tended to locate at the edge of the space, similar to what we found in LSTM.

Original: This indicates that, as in the LSTM, the core units in the GRU have distinctive proﬁles, distant from one another and from other units in the network (A.7B).
Revised: This indicates that, as in the LSTM, the core units in the GRU have distinctive proﬁles, distant from one another and from other units in the network (A.8B).

Revised: However, we did not observe the pattern of “integrator units” in the GRU as in the LSTM.

Original: These apparent similarities and differences between LSTM and GRU are intriguing, but we emphasize that (1) the perplexity of this GRU model is much higher than the LSTM, due to limitations in training time and tuning, and that (2) comparing the LSTM and GRU connection patterns is not straightforward, as the overall distribution of weights is different.
Revised: These apparent similarities and differences between LSTM and GRU are intriguing, but we emphasize that (1) the perplexity of this GRU model is much higher than the LSTM, due to the sub-optimal parameter settings, and that (2) comparing the LSTM and GRU connection patterns is not straightforward, as the overall distribution of weights is different.

Original: Further work will be required to determine comparable thresholds for “strong” projections and “high-degree units” in each case.
Revised: Further work will be required to determine comparable thresholds for “strong” projections and “high-degree units” in each case.

Original: As we noted in the manuscript and above, the connectivity results are exploratory; however, we believe that the GRU analysis demonstrates how these methods can be extended to map and compare the functional organization of language models of different architectures.
Revised: As we noted in the manuscript and above, the connectivity results are exploratory; however, we believe that the GRU analysis demonstrates how these methods can be extended to map and compare the functional organization of language models of different architecture.

Original: A. A BLATION ANALYSES ON PUTATIVE CONTROLLER AND INTEGRATOR UNITS
Revised: Finally, we note that when conducting the timescale analysis on an incompletely trained GRU model (trained ∼ 10 epochs, validation perplexity ≈ 350), the timescale distribution was more right-skewed (Figure A.6B) than the better-trained GRU (Figure A.7B). Altogether, these results suggest that the long-timescale units in GRU were gradually formed during the training process.

Original: To examine  the non-trivial roles of the controller and integrator unitsidentiﬁed in the  word-level LSTMmodel, we performed a preliminary group ablation analysis to look at how ablating the controller units inﬂuences model performance on predicting the next token, relative to the ablation of a random set of units.
Revised: To examine whether the number of hidden units in the model would affect the timescale organization in an LSTM, we trained another 2-layer word-level LSTM model with the same Wikipedia corpus and similar parameter settings as in Gulordava et al.

Original: Speciﬁcally, since long-timescale integrator unitsshould have most effect pre- dicting tokens at the later part of the sentences (i.e., when more context is integrated), we examined the model performance on predicting tokens at two different positions: (1) all the tokens regardless of their positions in the sentences (“All tokens” condition), and (2) the last tokens of sentences (“Final tokens” condition). We evaluated the effects of ablation on model performance by measuring the differences of probabilities ( ∆ P ) assigned to the target words ( ∆ P = probability of target word in ablated model minus probability of target word in original model). Ablation effects for controller units ( N =9) and integrator units ( N =10) were compared against a baseline of ablating the same number of randomlyselected units from layer 2 of the LSTM (Figure 4C).
Revised: (2018), but with only 100 hidden units in each layer.

Original: We used the test corpus used by Gulordava et al.
Revised: We called this model LSTM-100.

Original: and measured theaverage performance of each model across 100 text-batches, randomly sampled from the Wikipedia test dataset.
Revised: We trained the model for 56 epochs until the model converged to a validation perplexity 98.75, and conducted the same analysis as described in the main text to map the timescales of LSTM-100.

Original: Each text-batch was composed of 1000 tokens that start at the beginning of a sentence. In the “All tokens” condition, we calculated the ∆ P for every token in the tested text, while in the “Final tokens” condition, we calculated ∆ P only at the last token of every sentence (i.e. the token right before the full stop“.” of each sentence). We then average the ∆ P in both conditions across text-batches to get a mean performance difference between the ablated model and the intact model.
Revised: Because LSTM-100 have overall less weight connections, we use | z-score | > 3 as criteria to determine the “strong” hidden-to-gate projections for connectivity analyses. Regarding the timescale distribution in LSTM-100, we found that the results were similar to the 650-unit word-level LSTM model, in that: (1) the second layer of LSTM-100 showed more context sensitivity than the ﬁrst layer, and (2) although it was difﬁcult to quantitatively compare the unitlevel timescale distribution between the LSTM-100 model and the LSTM with 650 units, they both contain a similarly small subset of long-timescale units.

Original: Ablating controller units reduced the probabilities assigned to the target words, more so than ablating random units (Figure 4C, controller vs. random across 100 text batches: Cohen’s d = -4.85, t = -34.28, p < 0.001). In contrast, ablating integrator units showed reduced the probabilities less than ablating random units (integrator vs. random: Cohen’s d = 2.50, t = 17.67, p < 0.001).
Revised: (A.9).

Original: We hypothesized that that the integrator units mostly inﬂuence the model performance on predicting tokens in cases where long-range information is especially relevant, such as in the later portions of clauses and sentences. Consistent with this, we found that, when we examined the ablation effects only for tokens in the ﬁnal position of a sentence, ablating integrator units reduced the probabilities more than ablating random units (Cohen’s d = -0.34, t = -2.41, p = 0.017). Interestingly, ablating controller units reduced the probability of sentence-ﬁnal targets less than random units (Cohen’s d = 0.67, t = 4.74, p < 0.001).
Revised: We did not observe a signiﬁcant correlation between the unit timescale and number of strong projections generated by each unit in the LSTM-100 model: the long-timescale units in the LSTM-100 did not have more connections than short-timescale units.

Original: In summary, these ablation results indicate a non-trivial functional role for the controller and integrator units, despite the fact that each subset of  units is composed ofonly 10 amongst 650 total hidden units. Also, the putative controller and integrator sets appear tohave distinctive roles within the WLSTM, with the controllers supporting accurate predictions overall, while the integrator units appear to boost accurate predictions at the end of sentences.
Revised: When visualizing the MDS space of connectivity similarity of LSTM-100, the “controller units” identiﬁed using the k-core analysis were located in the edge of the space, similar to the 650-unit LSTM model. Interestingly, we observed a subset of long-timescale units in the center of the MDS space, analogous to the “integrator units” found in the 650-unit LSTM model. Altogether, the pattern of “integrator units” might be a commonly evolved feature that is shared between LSTM model architectures, but not with GRU architectures.