Original: Sparsity Emerges Naturally in Neural Language Models
Revised: Sparsity Emerges Naturally in Neural Language Models

Original: Anonymous submission

Original: Abstract
Revised: Abstract

Original: Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks.
Revised: Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks.

Original: If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse?
Revised: If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse?

Original: Using the
Revised: Using the

Original: Taxi-Euclidean norm to measure sparsity, we ﬁnd that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients.
Revised: Taxi-Euclidean norm to measure sparsity, we ﬁnd that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients.

Original: We ﬁnd that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.
Revised: We ﬁnd that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

Original:  Introduction
Revised:  Introduction

Original: Researchers in NLP have long relied on engineering features to reﬂect the sparse structures underlying language.
Revised: Researchers in NLP have long relied on engineering features to reﬂect the sparse structures underlying language.

Original: Modern deep learning methods promised to relegate this practice to history, but have not eliminated the interest in sparse modeling for NLP.
Revised: Modern deep learning methods promised to relegate this practice to history, but have not eliminated the interest in sparse modeling for NLP.

Original: Along with concerns about computational resources (Chen et al., 2016; Narang et al., 2017b) and interpretability (Murphy et al., 2012; Subramanian et al., 2018), human intuitions continue to motivate sparse representations of language.
Revised: Along with concerns about computational resources (Chen et al., 2016; Narang et al., 2017b) and interpretability (Murphy et al., 2012; Subramanian et al., 2018), human intuitions continue to motivate sparse representations of language.

Original: For example, some work applies assumptions of sparsity to model latent hard categories such as syntactic dependencies (Padó and Lapata, 2007) or phonemes (Cotterell and Eisner, 2018).
Revised: For example, some work applies assumptions of sparsity to model latent hard categories such as syntactic dependencies (Padó and Lapata, 2007) or phonemes (Cotterell and Eisner, 2018).

Original: Niculae and Blondel (2017) found that a sparse attention mechanism outperformed dense methods on some NLP tasks;
Revised: Niculae and Blondel (2017) found that a sparse attention mechanism outperformed dense methods on some NLP tasks;

Original: Narang et al.
Revised: Narang et al.

Original: (2017a) found sparsiﬁed versions of LMs that outperform dense originals.
Revised: (2017a) found sparsiﬁed versions of LMs that outperform dense originals.

Original: Attempts to engineer sparsity rest on an unstated assumption that it doesn’t arise naturally when neural models are learned.
Revised: Attempts to engineer sparsity rest on an unstated assumption that it doesn’t arise naturally when neural models are learned.

Original: Is this true?
Revised: Is this true?

Original: Using a simple measure of sparsity, we analyze how it arises in different layers of a neural language model in relation to word frequency.
Revised: Using a simple measure of sparsity, we analyze how it arises in different layers of a neural language model in relation to word frequency.

Original: We show that the sparsity of a word representation increases with exposure to that word during training.
Revised: We show that the sparsity of a word representation increases with exposure to that word during training.

Original: We also ﬁnd evidence of syntactic learning: gradient updates in backpropagation depend on whether a word’s part of speech is open or closed class, even controlling for word frequency.
Revised: We also ﬁnd evidence of syntactic learning: gradient updates in backpropagation depend on whether a word’s part of speech is open or closed class, even controlling for word frequency.

Original:  Methods 
Revised:  Methods Language model.

Original: Language model.

Original: Our LM is trained on a corpus of tokenized, lowercased English Wikipedia (70/10/20 train/dev/test split).
Revised: Our LM is trained on a corpus of tokenized, lowercased English Wikipedia (70/10/20 train/dev/test split).

Original: To reduce the number of unique words (mostly names) in the corpus, we excluded any sentence with a word which appears fewer than 100 times.
Revised: To reduce the number of unique words (mostly names) in the corpus, we excluded any sentence with a word which appears fewer than 100 times.

Original: Those words which still appear fewer than 100 times after this ﬁlter are replaced with <UNK> .
Revised: Those words which still appear fewer than 100 times after this ﬁlter are replaced with <UNK> .

Original: The resulting training set is over 227 million tokens of around 19.5K types.
Revised: The resulting training set is over 227 million tokens of around 19.5K types.

Original: We use a standard 2-layer LSTM LM trained with cross entropy loss for 50 epochs.
Revised: We use a standard 2-layer LSTM LM trained with cross entropy loss for 50 epochs.

Original: The pipeline from input x t − 1 at time step t − 1 to predicted output distribution ˆ x for time t is described in Figure 1, illustrating intermediate activations h et , h 1 t , and h 2 t .
Revised: The pipeline from input x t − 1 at time step t − 1 to predicted output distribution ˆ x for time t is described in Figure 1, illustrating intermediate activations h et , h 1 t , and h 2 t .

Original: At training time, the network observes x t and backpropagates the gradient updates ¯ h et , ¯ h 1 t , ¯ h 2 t , and ¯ x t .
Revised: At training time, the network observes x t and backpropagates the gradient updates ¯ h et , ¯ h 1 t , ¯ h 2 t , and ¯ x t .

Original: The embeddings produced by the encoding layer are 200 units, and the recurrent layers have 200 hidden units each.
Revised: The embeddings produced by the encoding layer are 200 units, and the recurrent layers have 200 hidden units each.

Original: The batch size is set to forty, the maximum sequence length to 35, and the dropout ratio to 0.2.
Revised: The batch size is set to forty, the maximum sequence length to 35, and the dropout ratio to 0.2.

Original: The optimizer is standard SGD with clipped gradients at (cid:96) 2 = 0 .
Revised: The optimizer is standard SGD with clipped gradients at (cid:96) 2 = 0 .

Original: 25 , where the learning rate begins at 20 and is quartered whenever loss fails to improve.
Revised: 25 , where the learning rate begins at 20 and is quartered whenever loss fails to improve.

Original: Measuring sparsity.
Revised: Measuring sparsity.

Original: We measure the sparsity of a vector v using the reciprocal of the TaxicabEuclidean norm ratio (Repetti et al., 2015).
Revised: We measure the sparsity of a vector v using the reciprocal of the TaxicabEuclidean norm ratio (Repetti et al., 2015).

Original: This measurement has a long history as a measurement of sparsity in natural settings (Zibulevsky 
Revised: This measurement has a long history as a measurement of sparsity in natural settings (Zibulevsky embedding

Original: embedding
Revised: [Figure] c [Figure] ¯ x

Original: [Figure]
Revised: [Figure]

Original: and Pearlmutter, 2001; Hoyer, 2004; Pham et al., 2017;
Revised: h [Figure] ¯ h e [Figure]

Revised: h

Original:  Yin et al., 2014) and is formally deﬁned as χ ( v ) =
Revised: [Figure] ¯ h 1 t h [Figure] ¯ h 2 t and Pearlmutter, 2001; Hoyer, 2004; Pham et al., 2017; Yin et al., 2014) and is formally deﬁned as χ ( v ) =

Original: (cid:107) v (cid:107) 2 / (cid:107) v (cid:107) 1 .
Revised: (cid:107) v (cid:107) 2 / (cid:107) v (cid:107) 1 .

Original: The relationship between sparsity and this ratio is illustrated in two dimensions in the image on the right, in which darker blue regions are more concentrated.
Revised: The relationship between sparsity and this ratio is illustrated in two dimensions in the image on the right, in which darker blue regions are more concentrated.

Original: The pink circle shows the area where (cid:96) 

Revised: The pink circle shows the area where (cid:96)

Original: ≤while the yellow diamond depicts (cid:96) 1 ≤ 1 .
Revised: ≤while the yellow diamond depicts (cid:96) 1 ≤ 1 .

Original: For sparse vectors (cid:104) 1 , 0 (cid:105) or (cid:104) 0 , 1 (cid:105) , the norms are identical so χ is 1, its maximum.
Revised: For sparse vectors (cid:104) 1 , 0 (cid:105) or (cid:104) 0 , 1 (cid:105) , the norms are identical so χ is 1, its maximum.

Original: For a uniform vector like (cid:104) 1 , 1 (cid:105) , χ is at its smallest.
Revised: For a uniform vector like (cid:104) 1 , 1 (cid:105) , χ is at its smallest.

Original: In general, χ ( v ) is higher when most elements of v are close to 0; and lower when the elements are all similar in value.
Revised: In general, χ ( v ) is higher when most elements of v are close to 0; and lower when the elements are all similar in value.

Original:  Experiments
Revised:  Experiments

Original: Sparsity is closely related to the behavior of a model: If only a few units hold most of the mass of a representation, the activation vector will be highly concentrated.
Revised: Sparsity is closely related to the behavior of a model: If only a few units hold most of the mass of a representation, the activation vector will be highly concentrated.

Original: If a neural network relies heavily on a small number of units in determining its predictions, the gradient will be highly concentrated.
Revised: If a neural network relies heavily on a small number of units in determining its predictions, the gradient will be highly concentrated.

Original: A highly concentrated gradient is mainly modifying a few speciﬁc pathways.
Revised: A highly concentrated gradient is mainly modifying a few speciﬁc pathways.

Original: For example, it might modify a neuron associated with particular inputs like parentheses (Karpathy et al., 2015), or properties like sentiment (Radford et al., 2017).
Revised: For example, it might modify a neuron associated with particular inputs like parentheses (Karpathy et al., 2015), or properties like sentiment (Radford et al., 2017).

Original: Representations of Target Words.
Revised: Representations of Target Words.

Original: Our ﬁrst experiments look at the relationship of sparsity to target word x t .
Revised: Our ﬁrst experiments look at the relationship of sparsity to target word x t .

Original: Gradient updates triggered by the target are often used to identify units that are relevant to a prediction (Li et al., 2015), and as shown in
Revised: Gradient updates triggered by the target are often used to identify units that are relevant to a prediction (Li et al., 2015), and as shown in

Original: Figure 2, gradient sparsity increases with both the frequency of a word in the corpus and the overall training time.
Revised: Figure 2, gradient sparsity increases with both the frequency of a word in the corpus and the overall training time.

Original: In other words, more exposure leads to sparser relevance.
Revised: In other words, more exposure leads to sparser relevance.

Original: Because the sparsity of ¯ hincreases with target word frequency, we measure not sparsity itself but the Pearson correlation, over all words w , between word frequency and mean χ ( h ) over representations h where w is the target:
Revised: Because the sparsity of ¯ hincreases with target word frequency, we measure not sparsity itself but the Pearson correlation, over all words w , between word frequency and mean χ ( h ) over representations h where w is the target:

Original: [Equation]
Revised: [Equation]

Original: Here (Figure 3a) we conﬁrm that concentrated gradients are not a result of concentrated activations, as activation sparsity χ ( h 2 ) is not correlated with target word frequency.
Revised: Here (Figure 3a) we conﬁrm that concentrated gradients are not a result of concentrated activations, as activation sparsity χ ( h 2 ) is not correlated with target word frequency.

Original: The correlation is strong and increasing only for ρ ← (¯ h 2 ) .
Revised: The correlation is strong and increasing only for ρ ← (¯ h 2 ) .

Original: The sparse structure being applied is therefore particular to the gradient passed from the softmax to the top LSTM layer, related to how a word interacts with its context.
Revised: The sparse structure being applied is therefore particular to the gradient passed from the softmax to the top LSTM layer, related to how a word interacts with its context.

Original: The Role of Part of Speech.
Revised: The Role of Part of Speech.

Original: Figure 4 shows that ρ ← (¯ h 2 ) follows distinctly different trends for open POS classes 1 and closed classes 2 .
Revised: Figure 4 shows that ρ ← (¯ h 2 ) follows distinctly different trends for open POS classes 1 and closed classes 2 .

Original: To associate words to POS, we tagged our training corpus with spacy 3 ; we associate a word to a POS only if the majority (at least 100) of its occurrences are tagged with that POS.
Revised: To associate words to POS, we tagged our training corpus with spacy 3 ; we associate a word to a POS only if the majority (at least 100) of its occurrences are tagged with that POS.

Original: We see that initially, frequent words from closed classes are highly concentrated, but soon stabilize, while frequent words from open classes continue to become more concentrated throughout training.
Revised: We see that initially, frequent words from closed classes are highly concentrated, but soon stabilize, while frequent words from open classes continue to become more concentrated throughout training.

Original: Why?
Revised: Why?

Original: Closed class words clearly signal POS.
Revised: Closed class words clearly signal POS.

Original: But open classes contain many ambiguous words, like
Revised: But open classes contain many ambiguous words, like

Original: [Figure]
Revised: [Figure]

Original: “report”, which can be a noun or verb.
Revised: “report”, which can be a noun or verb.

Original: Open classes also contain many more words in general.
Revised: Open classes also contain many more words in general.

Original: We posit that early in training, closed classes reliably signal syntactic structure, and are essential for shaping network structure.
Revised: We posit that early in training, closed classes reliably signal syntactic structure, and are essential for shaping network structure.

Original: But open classes are essential for predicting speciﬁc words, so their importance in training continues to increase after part of speech tags are effectively learned.
Revised: But open classes are essential for predicting speciﬁc words, so their importance in training continues to increase after part of speech tags are effectively learned.

Original: The high sparsity of function word gradient may be surprising when compared with ﬁndings that content words have a greater inﬂuence on outputs (Kádár et al., 2016).
Revised: The high sparsity of function word gradient may be surprising when compared with ﬁndings that content words have a greater inﬂuence on outputs (Kádár et al., 2016).

Original: However, those ﬁndings were based on the impact on the vector representation of an entire sentence after omitting the word.
Revised: However, those ﬁndings were based on the impact on the vector representation of an entire sentence after omitting the word.

Original: Khandelwal et al.
Revised: Khandelwal et al.

Original: (2018) found that content words have a longer window during which they are relevant, which may explain the results of Kádár et al.
Revised: (2018) found that content words have a longer window during which they are relevant, which may explain the results of Kádár et al.

Original: Neither of these studies controlled for word frequency in their analyses contrasting content and function words, but we believe this oversight is alleviated in our work by measuring correlations rather than raw magnitude.
Revised: Neither of these studies controlled for word frequency in their analyses contrasting content and function words, but we believe this oversight is alleviated in our work by measuring correlations rather than raw magnitude.

Original: Because ρ ← (¯ h 2 ) is higher when evaluated over
Revised: Because ρ ← (¯ h 2 ) is higher when evaluated over

Original: more fre- quent words, which also tend to be function words (see Figure 5), we further control for the effect of frequency by including a measurement of trends in a sample of 120 words each from open and closed classes (Figure 4).
Revised: more fre- quent words, which also tend to be function words (see Figure 5), we further control for the effect of frequency by including a measurement of trends in a sample of 120 words each from open and closed classes (Figure 4).

Original: This sample was selected by sorting all open and closed class words by frequency, then choosing a range of each sorted list with a similar average frequency.
Revised: This sample was selected by sorting all open and closed class words by frequency, then choosing a range of each sorted list with a similar average frequency.

Original: Representations of Input Words.
Revised: Representations of Input Words.

Original: We next looked at the vector representations of each step in the word sequence as a representation of the input word x t − 1 that produced that step.
Revised: We next looked at the vector representations of each step in the word sequence as a representation of the input word x t − 1 that produced that step.

Original: We measure the correlation with input word frequency:
Revised: We measure the correlation with input word frequency:

Original: [Equation]
Revised: [Equation]

Original: Here (Figure 3b) we ﬁnd that the view across training sheds some light on the learning process.
Revised: Here (Figure 3b) we ﬁnd that the view across training sheds some light on the learning process.

Original: While the lower recurrent layer quickly learns sparse representations of common input words, ρ → ( h 1 ) increases more slowly later in training and is eventually surpassed by ρ → ( h e ) , while gradient sparsity never becomes signiﬁcantly correlated with word frequency.
Revised: While the lower recurrent layer quickly learns sparse representations of common input words, ρ → ( h 1 ) increases more slowly later in training and is eventually surpassed by ρ → ( h e ) , while gradient sparsity never becomes signiﬁcantly correlated with word frequency.

Original: Li et al.
Revised: Li et al.

Original: (2016) studied the activations of feedforward networks in terms of the importance of individual units by erasing a particular dimension and measuring the difference in log likelihood of the target class.
Revised: (2016) studied the activations of feedforward networks in terms of the importance of individual units by erasing a particular dimension and measuring the difference in log likelihood of the target class.

Original: They found that importance is concentrated into a small number of units at the lowest layers in a neural network, and is more dispersed at higher layers.
Revised: They found that importance is concentrated into a small number of units at the lowest layers in a neural network, and is more dispersed at higher layers.

Original: Our ﬁndings suggest that this effect may be a natural result of the sparsity of the activations at lower layers.
Revised: Our ﬁndings suggest that this effect may be a natural result of the sparsity of the activations at lower layers.

Original: We relate the trajectory over training to the Information Bottleneck Hypothesis of Shwartz-Ziv and Tishby (2017).
Revised: We relate the trajectory over training to the Information Bottleneck Hypothesis of Shwartz-Ziv and Tishby (2017).

Original: This theory, connected to language model training by Saphra and Lopez (2018), proposes that the earlier stages of training are dedicated to learning to effectively represent inputs, while later in training these representations are compressed and the optimizer removes input information extraneous to the task of predicting outputs.
Revised: This theory, connected to language model training by Saphra and Lopez (2018), proposes that the earlier stages of training are dedicated to learning to effectively represent inputs, while later in training these representations are compressed and the optimizer removes input information extraneous to the task of predicting outputs.

Original: If extraneous information is encoded in speciﬁc units, this compression would lead to the observed effect, in which the ﬁrst time the optimizer rescales the step size, it begins an upward trend in ρ → as extraneous units are mitigated.
Revised: If extraneous information is encoded in speciﬁc units, this compression would lead to the observed effect, in which the ﬁrst time the optimizer rescales the step size, it begins an upward trend in ρ → as extraneous units are mitigated.

Original:  Potential Explanations
Revised:  Potential Explanations

Original: Why do common target words have such concentrated gradients with respect to the ﬁnal LSTM layer?
Revised: Why do common target words have such concentrated gradients with respect to the ﬁnal LSTM layer?

Original: A tempting explanation is that the amount of information we have about common words offers high conﬁdence and stabilizes most of the weights, leading to generally smaller gradients.
Revised: A tempting explanation is that the amount of information we have about common words offers high conﬁdence and stabilizes most of the weights, leading to generally smaller gradients.

Original: If this were true, the denominator of sparsity, gradient (cid:96) 1 , should be strongly anti-correlated with word frequency.
Revised: If this were true, the denominator of sparsity, gradient (cid:96) 1 , should be strongly anti-correlated with word frequency.

Original: In fact, it is only ever slightly anti-correlated (correlation > − . 1 ).
Revised: In fact, it is only ever slightly anti-correlated (correlation > − . 1 ).

Original: Furthermore, the sparsity of the softmax gradient χ (¯ x ) does not exhibit the strong correlation seen in χ (¯ h 2 ) , so sparsity at the LSTM gradient is not a direct effect of sparse logits.
Revised: Furthermore, the sparsity of the softmax gradient χ (¯ x ) does not exhibit the strong correlation seen in χ (¯ h 2 ) , so sparsity at the LSTM gradient is not a direct effect of sparse logits.

Original: However, the model could still be “high conﬁdence” in terms of how it assigns blame for error during common events, even if it is barely more conﬁdent overall in its predictions.
Revised: However, the model could still be “high conﬁdence” in terms of how it assigns blame for error during common events, even if it is barely more conﬁdent overall in its predictions.

Original: According to this hypothesis, a few specialized neurons might be responsible for the handling of such words.
Revised: According to this hypothesis, a few specialized neurons might be responsible for the handling of such words.

Original: Perhaps common words play a prototyping role that deﬁnes clusters of other words, and therefore have a larger impact on these clusters by acting as attractors within the representation space early on.
Revised: Perhaps common words play a prototyping role that deﬁnes clusters of other words, and therefore have a larger impact on these clusters by acting as attractors within the representation space early on.

Original: Such a process would be similar to how humans acquire language by learning to use words like ‘dog’ before similar but less prototypical words like ‘canine’ (Rosch, 1999).
Revised: Such a process would be similar to how humans acquire language by learning to use words like ‘dog’ before similar but less prototypical words like ‘canine’ (Rosch, 1999).

Original: As a possible mechanism for prototyping with individual units, Dalvi et al.
Revised: As a possible mechanism for prototyping with individual units, Dalvi et al.

Original: (2019) found that some neurons in a translation system specialized in particular word forms, such as verb inﬂection or comparative and superlative adjectives.
Revised: (2019) found that some neurons in a translation system specialized in particular word forms, such as verb inﬂection or comparative and superlative adjectives.

Original: For example, a common comparative adjective like ‘better’ might be used as a reliable signal to shape the handling of comparatives by triggering specialized units, while rarer words have representations that are more distributed according to a small collection of speciﬁc contexts.
Revised: For example, a common comparative adjective like ‘better’ might be used as a reliable signal to shape the handling of comparatives by triggering specialized units, while rarer words have representations that are more distributed according to a small collection of speciﬁc contexts.

Original: There may also be some other reason that common words interact more with speciﬁc substructures within the network.
Revised: There may also be some other reason that common words interact more with speciﬁc substructures within the network.

Original: For example, it could be related to the use of context.
Revised: For example, it could be related to the use of context.

Original: Because rare words use more context than common words and content words use more context than function words (Khandelwal et al., 2018), the gradient associated with a common word would be focused on interactions with the most recent words.
Revised: Because rare words use more context than common words and content words use more context than function words (Khandelwal et al., 2018), the gradient associated with a common word would be focused on interactions with the most recent words.

Original: This would lead common word gradients to be more concentrated.
Revised: This would lead common word gradients to be more concentrated.

Original: It is possible that frequent words have sparse activations because frequency is learned as a feature and thus is counted by a few dimensions of proportional magnitude, as posited by Li et al.
Revised: It is possible that frequent words have sparse activations because frequency is learned as a feature and thus is counted by a few dimensions of proportional magnitude, as posited by Li et al.

Original:  Potential Applications
Revised:  Potential Applications

Original: Understanding where natural sparsity emerges in dense networks could be a useful guide in deciding which layers we can apply sparsity constraints to without affecting model performance, for the purpose of interpretability or efﬁciency.
Revised: Understanding where natural sparsity emerges in dense networks could be a useful guide in deciding which layers we can apply sparsity constraints to without affecting model performance, for the purpose of interpretability or efﬁciency.

Original: It might also explain why certain techniques are effective: for example, in some applications, summing representations together works quite well (Hill et al., 2016).
Revised: It might also explain why certain techniques are effective: for example, in some applications, summing representations together works quite well (Hill et al., 2016).

Original: We hypothesize that this occurs when the summed representations are sparse so there is often little overlap.
Revised: We hypothesize that this occurs when the summed representations are sparse so there is often little overlap.

Original: Understanding sparsity could help identify cases where such simple ensembling approaches are likely to be effective.
Revised: Understanding sparsity could help identify cases where such simple ensembling approaches are likely to be effective.

Original: Future work may develop ways of manipulating the training regime, as in curriculum learning, to accelerate the concentration of common words or incorporating concentration into the training objective as a regularizer.
Revised: Future work may develop ways of manipulating the training regime, as in curriculum learning, to accelerate the concentration of common words or incorporating concentration into the training objective as a regularizer.

Original: We would also like to see how sparsity emerges in models designed for speciﬁc end tasks, and to see whether concentration is a useful measure for the information compression predicted by the Information Bottleneck.
Revised: We would also like to see how sparsity emerges in models designed for speciﬁc end tasks, and to see whether concentration is a useful measure for the information compression predicted by the Information Bottleneck.