Original: Identifying and Controlling Important Neurons in Neural Machine Translation
Revised: Identifying and Controlling Important Neurons in Neural Machine Translation

Revised: Anonymous Author(s)

Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: Neural machine translation (NMT) models learn representations containing substantial linguistic information.
Revised: Neural machine translation (NMT) models learn representations containing substantial linguistic information.

Original: However, it is not clear if such information is fullydistributed or if some of it can be attributed to individual neurons.
Revised: However, it is not clear if such information is fullydistributed or if some of it can be attributed to individual neurons.

Original: We developunsupervised methods for discovering important neurons in NMT models.
Revised: We developunsupervised methods for discovering important neurons in NMT models.

Original: Ourmethods rely on the intuition that different models learn similar properties, and donot require any costly external supervision.
Revised: Ourmethods rely on the intuition that different models learn similar properties, and donot require any costly external supervision.

Original: We show experimentally that translation quality depends on the discovered neurons, and ﬁnd that many of themcapture common linguistic phenomena.
Revised: We show experimentally that translation quality depends on the discovered neurons, and ﬁnd that many of themcapture common linguistic phenomena.

Original: Finally, we show how to control NMTtranslations in predictable ways, by modifying activations of individual neurons
Revised: Finally, we show how to control NMTtranslations in predictable ways, by modifying activations of individual neurons

Original:  Introduction
Revised:  Introduction

Original: Neural machine translation (NMT) systems achieve state-of-the-art results by learning from largeamounts of example translations, typically without additional linguistic information.
Revised: Neural machine translation (NMT) systems achieve state-of-the-art results by learning from largeamounts of example translations, typically without additional linguistic information.

Original: Recent studieshave shown that representations learned by NMT models contain a non-trivial amount of linguisticinformation on multiple levels: morphological (Belinkov et al., 2017), syntactic (Shi et al., 2016b),and semantic (Hill et al., 2017).
Revised: Recent studieshave shown that representations learned by NMT models contain a non-trivial amount of linguisticinformation on multiple levels: morphological (Belinkov et al., 2017), syntactic (Shi et al., 2016b),and semantic (Hill et al., 2017).

Original: These studies use trained NMT models to generate feature representations for words, and use these representations to predict certain linguistic properties.
Revised: These studies use trained NMT models to generate feature representations for words, and use these representations to predict certain linguistic properties.

Original: Thisapproach has two main limitations.
Revised: Thisapproach has two main limitations.

Original: First, it targets the whole vector representation and fails toanalyze individual dimensions in the vector space.
Revised: First, it targets the whole vector representation and fails to analyze individual dimensions in the vector space.

Original: In contrast, previous work found meaningfulindividual neurons in computer vision (Zeiler & Fergus, 2014;
Revised: In contrast, previous work found meaningfulindividual neurons in computer vision (Zeiler & Fergus, 2014;

Original: Zhou et al., 2016;
Revised: Zhou et al., 2016;

Original: Bau et al., 2017,among others) and in a few NLP tasks (Karpathy et al., 2015; Radford et al., 2017; Qian et al.,2016a).
Revised: Bau et al., 2017,among others) and in a few NLP tasks (Karpathy et al., 2015; Radford et al., 2017; Qian et al.,2016a).

Original: Second, these methods require external supervision in the form of linguistic annotations.
Revised: Second, these methods require external supervision in the form of linguistic annotations.

Original: They are therefore limited by available annotated data and tools.
Revised: They are therefore limited by available annotated data and tools.

Original: In this work, we make initial progress towards addressing these limitations by developing unsupervised methods for analyzing the contribution of individual neurons to NMT models.
Revised: In this work, we make initial progress towards addressing these limitations by developing unsupervised methods for analyzing the contribution of individual neurons to NMT models.

Original: We aim toanswer the following questions:
Revised: We aim toanswer the following questions:

Original: • How important are individual neurons for obtaining high-quality translations?
Revised: • How important are individual neurons for obtaining high-quality translations?

Original: • Do individual neurons in NMT models contain interpretable linguistic information?
Revised: • Do individual neurons in NMT models contain interpretable linguistic information?

Original: • Can we control MT output by intervening in the representation at the individual neuron level?
Revised: • Can we control MT output by intervening in the representation at the individual neuron level?

Original: To answer these questions, we develop several unsupervised methods for ranking neurons accordingto their importance to an NMT model.
Revised: To answer these questions, we develop several unsupervised methods for ranking neurons accordingto their importance to an NMT model.

Original: Inspired by work in machine vision (Li et al., 2016b), wehypothesize that different NMT models learn similar properties, and therefore similar importantneurons should emerge in different models.
Revised: Inspired by work in machine vision (Li et al., 2016b), wehypothesize that different NMT models learn similar properties, and therefore similar importantneurons should emerge in different models.

Original: To test this hypothesis, we map neurons between pairs oftrained NMT models using several methods: correlation analysis, regression analysis, and SVCCA,a recent method combining singular vectors and canonical correlation analysis (Raghu et al., 2017).
Revised: To test this hypothesis, we map neurons between pairs oftrained NMT models using several methods: correlation analysis, regression analysis, and SVCCA,a recent method combining singular vectors and canonical correlation analysis (Raghu et al., 2017).

Original: Our mappings yield lists of candidate neurons containing shared information across models.
Revised: Our mappings yield lists of candidate neurons containing shared information across models.

Original: Wethen evaluate whether these neurons carry important information to the NMT model by maskingtheir activations during testing.
Revised: Wethen evaluate whether these neurons carry important information to the NMT model by maskingtheir activations during testing.

Original: We ﬁnd that highly-shared neurons impact translation quality muchmore than unshared neurons, afﬁrming our hypothesis that shared information matters .
Revised: We ﬁnd that highly-shared neurons impact translation quality muchmore than unshared neurons, afﬁrming our hypothesis that shared information matters .

Original: Given the list of important neurons, we then investigate what linguistic properties they capture,both qualitatively by visualizing neuron activations and quantitatively by performing supervisedclassiﬁcation experiments.
Revised: Given the list of important neurons, we then investigate what linguistic properties they capture,both qualitatively by visualizing neuron activations and quantitatively by performing supervisedclassiﬁcation experiments.

Original: We were able to identify neurons corresponding to several linguisticphenomena, including morphological and syntactic properties.
Revised: We were able to identify neurons corresponding to several linguisticphenomena, including morphological and syntactic properties.

Original: Finally, we test whether intervening in the representation at the individual neuron level can helpcontrol the translation .
Revised: Finally, we test whether intervening in the representation at the individual neuron level can helpcontrol the translation .

Original: We demonstrate the ability to control NMT translations on three linguisticproperties—tense, number, and gender—to varying degrees of success.
Revised: We demonstrate the ability to control NMT translations on three linguisticproperties—tense, number, and gender—to varying degrees of success.

Original: This sets the ground forcontrolling NMT in desirable ways, potentially reducing system bias to properties like gender.
Revised: This sets the ground forcontrolling NMT in desirable ways, potentially reducing system bias to properties like gender.

Original: Our work indicates that not all information is distributed in NMT models, and that many humaninterpretable grammatical and structural properties are captured by individual neurons.
Revised: Our work indicates that not all information is distributed in NMT models, and that many humaninterpretable grammatical and structural properties are captured by individual neurons.

Original: Moreover,modifying the activations of individual neurons allows controlling the translation output according to speciﬁed linguistic properties.
Revised: Moreover,modifying the activations of individual neurons allows controlling the translation output accordingto speciﬁed linguistic properties.

Original: The methods we develop here are task-independent and canbe used for analyzing neural networks in other tasks.
Revised: The methods we develop here are task-independent and can beused for analyzing neural networks in other tasks.

Original: More broadly, our work contributes to thelocalist/distributed debate in neural cognitive science (Gayler & Levy, 2011) by investigating theimportant case of neural machine translation.
Revised: More broadly, our work contributes to the localist/distributed debate in artiﬁcial intelligence and cognitive science (Gayler & Levy, 2011) byinvestigating the important case of neural machine translation.

Original:  Related Work
Revised:  Related Work

Original: Much recent work has been concerned with analyzing neural representations of linguistic units,such as word embeddings (K¨ohn, 2015;
Revised: Much recent work has been concerned with analyzing neural representations of linguistic units,such as word embeddings (K ¨ ohn, 2015;

Original: Qian et al., 2016b), sentence embeddings (Adi et al., 2016;
Revised: Qian et al., 2016b), sentence embeddings (Adi et al., 2016;

Original: Ganesh et al., 2017; Brunner et al., 2018), and NMT representations at different linguistic levels:morphological (Belinkov et al., 2017), syntactic (Shi et al., 2016b), and semantic (Hill et al., 2017).
Revised: Ganesh et al., 2017; Brunner et al., 2018), and NMT representations at different linguistic levels:morphological (Belinkov et al., 2017), syntactic (Shi et al., 2016b), and semantic (Hill et al., 2017).

Original: These studies follow a common methodology of evaluating learned representations on external supervision by training classiﬁers or measuring other kinds of correlations.
Revised: These studies follow a common methodology of evaluating learned representations on external supervision by training classiﬁers or measuring other kinds of correlations.

Original: Thus they are limited tothe available supervised annotation.
Revised: Thus they are limited tothe available supervised annotation.

Original: In addition, these studies also do not typically consider individual dimensions.
Revised: In addition, these studies also do not typically consider individual dimensions.

Original: In contrast, we propose intrinsic unsupervised methods for detecting importantneurons based on correlations between independently trained models.
Revised: In contrast, we propose intrinsic unsupervised methods for detecting importantneurons based on correlations between independently trained models.

Original: A similar approach was usedto analyze vision networks (Li et al., 2016b), but to the best of our knowledge these ideas were notused to study NMT or other NLP models before.
Revised: A similar approach was usedto analyze vision networks (Li et al., 2016b), but to the best of our knowledge these ideas were notused to study NMT or other NLP models before.

Original: In computer vision, individual neurons were shown to capture meaningful information Zeiler &Fergus (2014);
Revised: In computer vision, individual neurons were shown to capture meaningful information (Zeiler &

Original:  Zhou et al. ; Bau et al.
Revised: Fergus, 2014; Zhou et al., 2016; Bau et al., 2017).

Original: Even though some doubts were cast on the importance of individual units (Morcos et al., 2018), recent work stressed their contribution to predictingspeciﬁc object classes via masking experiments similar to ours (Zhou et al., 2018).
Revised: Even though some doubts were cast on the importance of individual units (Morcos et al., 2018), recent work stressed their contribution to predictingspeciﬁc object classes via masking experiments similar to ours (Zhou et al., 2018).

Original: A few studiesanalyzed individual neurons in NLP.
Revised: A few studiesanalyzed individual neurons in NLP.

Original: For instance, neural language models learn speciﬁc neuronsthat activate on brackets (Karpathy et al., 2015), sentiment (Radford et al., 2017), and length (Qianet al., 2016a).
Revised: For instance, neural language models learn speciﬁc neuronsthat activate on brackets (Karpathy et al., 2015), sentiment (Radford et al., 2017), and length (Qianet al., 2016a).

Original: Length-speciﬁc neurons were also found in NMT (Shi et al., 2016a), but generally notmuch work has been devoted to analyzing individual neurons in NMT.
Revised: Length-speciﬁc neurons were also found in NMT (Shi et al., 2016a), but generally notmuch work has been devoted to analyzing individual neurons in NMT.

Original: We aim to address this gap.
Revised: We aim to address this gap.

Original:  Methodology
Revised:  Methodology

Original: Much recent work on analyzing NMT relies on supervised learning, where NMT representationsare used as features for predicting linguistic annotations (see Section 2).
Revised: Much recent work on analyzing NMT relies on supervised learning, where NMT representationsare used as features for predicting linguistic annotations (see Section 2).

Original: However, such annotationsmay not be available, or constrain the analysis to a particular scheme.
Revised: However, such annotationsmay not be available, or constrain the analysis to a particular scheme.

Original: Instead, we propose to use different kinds of correlations between neurons from different models asa measure of their importance.
Revised: Instead, we propose to use different kinds of correlations between neurons from different models asa measure of their importance.

Original: Suppose we have M such models and let h mt
Revised: Suppose we have M such models and let h mt

Original: [ i ] denote the activationof the i -th neuron in the encoder of the m -th model for the t -th word.
Revised: [ i ] denote the activationof the i -th neuron in the encoder of the m -th model for the t -th word.

Original: 1 These may be modelsfrom different training epochs, trained with different random initializations or datasets, or evendifferent architectures—all realistic scenarios that researchers often experiment with.
Revised: 1 These may be modelsfrom different training epochs, trained with different random initializations or datasets, or evendifferent architectures—all realistic scenarios that researchers often experiment with.

Original: Let x mi denotea random variable corresponding to the i -th neuron in the m -th model.
Revised: Let x mi denotea random variable corresponding to the i -th neuron in the m -th model.

Original: x mi maps words to theirneuron activations: x mi : t 7! h mt
Revised: x mi maps words to theirneuron activations: x mi : t (cid:55)→ h mt

Original: [ i ] .
Revised: [ i ] .

Original: Similarly, let x m denote a random vector corresponding to theactivations of all neurons in the m -th model: x m : t 7! h mt .
Revised: Similarly, let x m denote a random vector corresponding to theactivations of all neurons in the m -th model: x m : t (cid:55)→ h mt .

Original: We consider four methods for ranking neurons, based on correlations between pairs of models.
Revised: We consider four methods for ranking neurons, based on correlations between pairs of models.

Original: Ourhypothesis is that different NMT models learn similar properties, and therefore similar importantneurons emerge in different models, akin to neural vision models (Li et al., 2016b).
Revised: Ourhypothesis is that different NMT models learn similar properties, and therefore similar importantneurons emerge in different models, akin to neural vision models (Li et al., 2016b).

Original: Our methodscapture different levels of localization/distributivity, as described next.
Revised: Our methodscapture different levels of localization/distributivity, as described next.

Original: See Figure 1 for illustration.
Revised: See Figure 1 for illustration.

Original: 3.1 Unsupervised correlation Methods
Revised: 3.1 Unsupervised correlation MethodsMaximum correlation

Original: [Figure]

Original: Maximum correlation The maximum correlation ( MaxCorr ) of neuron x mi looks for the highestcorrelation with any neuron in all other models:
Revised: The maximum correlation ( MaxCorr ) of neuron x mi looks for the highestcorrelation with any neuron in all other models:

Original: [Equation]
Revised: [Equation]

Original: where ⇢ ( x , y ) is the Pearson correlation coefﬁcient between x and y. We then rank the neurons inmodel m according to their MaxCorr score.
Revised: where ρ ( x , y ) is the Pearson correlation coefﬁcient between x and y. We then rank the neurons inmodel m according to their MaxCorr score.

Original: We repeat this procedure for every model m .
Revised: We repeat this procedure for every model m .

Original: Thisscore looks for neurons that capture properties that emerge strongly in two separate models.
Revised: Thisscore looks for neurons that capture properties that emerge strongly in two separate models.

Original: Minimum correlation The minimum correlation ( MinCorr ) of neuron x mi looks for the neuronsmost correlated with X mi in each of the other models, but selects the one with the lowest correlation: 
Revised: The minimum correlation ( MinCorr ) of neuron x mi looks for the neuronsmost correlated with X mi in each of the other models, but selects the one with the lowest correlation: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Neurons in model m are ranked according to their MinCorr score.
Revised: Neurons in model m are ranked according to their MinCorr score.

Original: This tries to ﬁnd neurons thatare well correlated with many other models, even if they are not the overall most correlated ones.
Revised: This tries to ﬁnd neurons thatare well correlated with many other models, even if they are not the overall most correlated ones.

Original: Regression ranking We perform linear regression ( LinReg ) from the full representation of another model x m 0 to the neuron x mi .
Revised: Regression ranking We perform linear regression ( LinReg ) from the full representation of another model x m (cid:48) to the neuron x mi .

Original: Then we rank neurons by the regression mean squared error.
Revised: Then we rank neurons by the regression mean squared error.

Original: This attempts to ﬁnd neurons whose information might be distributed in other models.
Revised: This attempts to ﬁnd neurons whose information might be distributed in other models.

Original: SVCCA Singular vector canonical correlation analysis ( SVCCA ) is a recent method for analyzingneural networks (Raghu et al., 2017).
Revised: SVCCA Singular vector canonical correlation analysis ( SVCCA ) is a recent method for analyzingneural networks (Raghu et al., 2017).

Original: In our implementation, we perform PCA on each model’srepresentations x m and take enough dimensions to account for 99% of the variance.
Revised: In our implementation, we perform PCA on each model’srepresentations x m and take enough dimensions to account for 99% of the variance.

Original: For each pairof models, we obtain the canonically correlated basis, and rank the basis directions by their CCAcoefﬁcients.
Revised: For each pairof models, we obtain the canonically correlated basis, and rank the basis directions by their CCAcoefﬁcients.

Original: This attempts to capture information that may be distributed in less dimensions thanthe whole representation.
Revised: This attempts to capture information that may be distributed in less dimensions thanthe whole representation.

Original: In this case we get a ranking of directions, rather than individual neurons.
Revised: In this case we get a ranking of directions, rather than individual neurons.

Original: 3.2 Verifying Detected Neurons
Revised: 3.2 Verifying Detected Neurons

Original: [Figure]

Original: We want to verify that neurons ranked highly by the unsupervised methods are indeed important forthe NMT models.
Revised: We want to verify that neurons ranked highly by the unsupervised methods are indeed important forthe NMT models.

Original: We consider quantitative and qualitative techniques for verifying their importance.
Revised: We consider quantitative and qualitative techniques for verifying their importance.

Original: Erasing Neurons We test importance of neurons by erasing some of them during translation.
Revised: Erasing Neurons We test importance of neurons by erasing some of them during translation.

Original: Erasure is a useful technique for analyzing neural networks (Li et al., 2016a).
Revised: Erasure is a useful technique for analyzing neural networks (Li et al., 2016a).

Original: Given a ranked list ofneurons ⇡ , where ⇡ ( i ) is the rank of neuron x i , we zero-out increasingly more neurons according tothe ranking ⇡ , starting from either the top or the bottom of the list.
Revised: Given a ranked list ofneurons π , where π ( i ) is the rank of neuron x i , we zero-out increasingly more neurons according tothe ranking π , starting from either the top or the bottom of the list.

Original: Our hypothesis is that erasingneurons from the top would hurt translation performance more than erasing from the bottom.
Revised: Our hypothesis is that erasingneurons from the top would hurt translation performance more than erasing from the bottom.

Original: Concretely, we ﬁrst run the entire encoder as usual, then zero out speciﬁc neurons from all sourcehidden states { h 1 , . . .
Revised: Concretely, we ﬁrst run the entire encoder as usual, then zero out speciﬁc neurons from all sourcehidden states { h 1 , . . .

Original: , h n } before running the decoder.
Revised: , h n } before running the decoder.

Original: For MaxCorr , MinCorr , and LinReg ,we zero out individual neurons.
Revised: For MaxCorr , MinCorr , and LinReg ,we zero out individual neurons.

Original: To erase k directions found by SVCCA , we instead project theembedding E (corresponding to all activations of a given model over a dataset) onto the spacespanned by the non-erased directions: E [-
-]
Revised: To erase k directions found by SVCCA , we instead project theembedding E (corresponding to all activations of a given model over a dataset) onto the spacespanned by the non-erased directions: E [-
-](cid:48)

Original: = E ( C ( C T C )  1 C T ) , where C is the CCA projectionmatrix with the ﬁrst or last k columns removed.
Revised: = E ( C ( C T C ) − 1 C T ) , where C is the CCA projectionmatrix with the ﬁrst or last k columns removed.

Original: This corresponds to erasing from the top or bottom.
Revised: This corresponds to erasing from the top or bottom.

Original: Supervised Veriﬁcation
Revised: Supervised Veriﬁcation

Original: While our focus is on unsupervised methods for ﬁnding important neurons, we also utilize supervision to verify our results.
Revised: While our focus is on unsupervised methods for ﬁnding important neurons, we also utilize supervision to verify our results.

Original: Since training a supervised classiﬁer on everyneuron is costly, we instead report simple metrics that can be easily computed.
Revised: Since training a supervised classiﬁer on everyneuron is costly, we instead report simple metrics that can be easily computed.

Original: Speciﬁcally, we
Revised: Speciﬁcally, we

Original: [Figure]
Revised: [Figure]

Original: sometimes report the expected conditional variance of neuron activations conditioned on some property.
Revised: sometimes report the expected conditional variance of neuron activations conditioned on some property.

Original: In other cases we found it useful to estimate a Gaussian mixture model (GMM) for predictinga label and measure its prediction quality.
Revised: In other cases we found it useful to estimate a Gaussian mixture model (GMM) for predictinga label and measure its prediction quality.

Original: We obtain linguistic annotations with Spacy: spacy .
Revised: We obtain linguistic annotations with Spacy: spacy .

Original: io .
Revised: io .

Original: Visualization Interpretability of machine learning models remains elusive (Lipton, 2016), but visualizing can be an instructive technique.
Revised: Visualization Interpretability of machine learning models remains elusive (Lipton, 2016), but visualizing can be an instructive technique.

Original: Similar to previous work analyzing neural networks in
Revised: Similar to previous work analyzing neural networks in

Original: NLP (Elman, 1991; Karpathy et al., 2015; K´ad´ar et al., 2016), we visualize activations of neuronsand observe interpretable behavior.
Revised: NLP (Elman, 1991; Karpathy et al., 2015; K´ad´ar et al., 2016), we visualize activations of neuronsand observe interpretable behavior.

Original: We will illustrate this with example heatmaps below.
Revised: We will illustrate this with example heatmaps below.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Data We use the United Nations (UN) parallel corpus (Ziemski et al., 2016) for all experiments.
Revised: Data We use the United Nations (UN) parallel corpus (Ziemski et al., 2016) for all experiments.

Original: We train models from English to 5 languages: Arabic, Chinese, French, Russian, and Spanish, aswell as an English-English auto-encoder.
Revised: We train models from English to 5 languages: Arabic, Chinese, French, Russian, and Spanish, aswell as an English-English auto-encoder.

Original: For each target language, we train 3 models on differentparts of the training set, each with 500K sentences.
Revised: For each target language, we train 3 models on differentparts of the training set, each with 500K sentences.

Original: In total, we have 18 models.
Revised: In total, we have 18 models.

Original: This setting allowsus to compare models trained on the same language pairs but different training data, as well asmodels trained on different language pairs.
Revised: This setting allowsus to compare models trained on the same language pairs but different training data, as well asmodels trained on different language pairs.

Original: We evaluate on the ofﬁcial test set.
Revised: We evaluate on the ofﬁcial test set.

Original: MT training We train 500 dimensional 2-layer LSTM encoder-decoder models with attention Bahdanau et al.
Revised: MT training We train 500 dimensional 2-layer LSTM encoder-decoder models with attention Bahdanau et al.

Original: In order to study both word and sub-word properties, we use a wordrepresentation based on a character convolutional neural network (charCNN) as input to both encoder and decoder, which was shown to learn morphology in language modeling and NMT (Kimet al., 2015; Belinkov et al., 2017).
Revised: In order to study both word and sub-word properties, we use a wordrepresentation based on a character convolutional neural network (charCNN) as input to both encoder and decoder, which was shown to learn morphology in language modeling and NMT (Kimet al., 2015; Belinkov et al., 2017).

Original: 2 While we focus here on recurrent NMT, our approach can beapplied to other models like the Transformer (Vaswani et al., 2017), which we leave for future work.
Revised: 2 While we focus here on recurrent NMT, our approach can beapplied to other models like the Transformer (Vaswani et al., 2017), which we leave for future work.

Original:  Results
Revised:  Results

Original: 5.1 Erasure Experiments
Revised: 5.1 Erasure Experiments

Original: Figure 2 shows erasure results using the methods from Section 3.1, on an English-Spanish model.
Revised: Figure 2 shows erasure results using the methods from Section 3.1, on an English-Spanish model.

Original: For all four methods, erasing from the top hurts performance much more than erasing from thebottom.
Revised: For all four methods, erasing from the top hurts performance much more than erasing from thebottom.

Original: This conﬁrms our hypothesis that neurons ranked higher by our methods have a largerimpact on translation quality.
Revised: This conﬁrms our hypothesis that neurons ranked higher by our methods have a largerimpact on translation quality.

Original: Comparing erasure with different rankings, we ﬁnd similar patternswith MaxCorr , MinCorr , and LinReg : erasing the top ranked 10% (50 neurons) degrades BLEUby 15-20 points, while erasing the bottom 10% neurons only hurts by 2-3 points.
Revised: Comparing erasure with different rankings, we ﬁnd similar patternswith MaxCorr , MinCorr , and LinReg : erasing the top ranked 10% (50 neurons) degrades BLEUby 15-20 points, while erasing the bottom 10% neurons only hurts by 2-3 points.

Original: In contrast, erasing
Revised: In contrast, erasing

Original: SVCCA directions results in rapid degradation – 15 BLEU point drop when erasing 1% (5) of the topdirections, and poor performance when erasing 10% (50).
Revised: SVCCA directions results in rapid degradation – 15 BLEU point drop when erasing 1% (5) of the topdirections, and poor performance when erasing 10% (50).

Original: This indicates that top SVCCA directionscapture very important information in the model.
Revised: This indicates that top SVCCA directionscapture very important information in the model.

Original: We analyze these top neurons and directions in thenext section, ﬁnding that top SVCCA directions focus mostly on identifying speciﬁc words.
Revised: We analyze these top neurons and directions in thenext section, ﬁnding that top SVCCA directions focus mostly on identifying speciﬁc words.

Original: Figure 3 shows the results of MaxCorr when erasing neurons from top and bottom, using modelstrained on three language pairs.
Revised: Figure 3 shows the results of MaxCorr when erasing neurons from top and bottom, using modelstrained on three language pairs.

Original: In all cases, erasing from the top hurts performance more thanerasing from the bottom.
Revised: In all cases, erasing from the top hurts performance more thanerasing from the bottom.

Original: We found similar trends with other language pairs and ranking methods.
Revised: We found similar trends with other language pairs and ranking methods.

Original: 5.2 Evaluating Top Neurons
Revised: 5.2 Evaluating Top Neurons

Original: What kind of information is captured by the neurons ranked highly by each of our ranking methods?
Revised: What kind of information is captured by the neurons ranked highly by each of our ranking methods?

Original: Previous work found speciﬁc neurons in NMT that capture position of words in the sentence (Shi
Revised: Previous work found speciﬁc neurons in NMT that capture position of words in the sentence (Shi

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: et al., 2016a).
Revised: et al., 2016a).

Original: Do our methods capture similar properties?
Revised: Do our methods capture similar properties?

Original: Indeed [Table] we found that many of the topneurons capture position.
Revised: Indeed, we found that many of the topneurons capture position.

Original: For instance, Table 1 shows the top 10 ranked neurons from an EnglishSpanish model according to each of the methods.
Revised: For instance, Table 1 shows the top 10 ranked neurons from an EnglishSpanish model according to each of the methods.

Original: The table shows the percent of variance in neuronactivation that is eliminated by conditioning on position in the sentence, calculated over the test set.
Revised: The table shows the percent of variance in neuronactivation that is eliminated by conditioning on position in the sentence, calculated over the test set.

Original: Similarly, it shows the percent of explained variance by conditioning on the current token identity.
Revised: Similarly, it shows the percent of explained variance by conditioning on the current token identity.

Original: We observe an interesting difference between the ranking methods.
Revised: We observe an interesting difference between the ranking methods.

Original: LinReg and especially SVCCA ,which are both computed by using multiple neurons, tend to ﬁnd information determined by theidentity of the current token.
Revised: LinReg and especially SVCCA ,which are both computed by using multiple neurons, tend to ﬁnd information determined by theidentity of the current token.

Original: MaxCorr and (especially)
Revised: MaxCorr and (especially)

Original: MinCorr tend to ﬁnd position information.
Revised: MinCorr tend to ﬁnd position information.

Original: This suggests that information about the current token is often distributed in multiple neurons, whichcan be explained by the fact that tokens carry multiple kinds of linguistic information.
Revised: This suggests that information about the current token is often distributed in multiple neurons, whichcan be explained by the fact that tokens carry multiple kinds of linguistic information.

Original: In contrast,position is a fairly simple property that the NMT encoder can represent in a small number of neurons.
Revised: In contrast,position is a fairly simple property that the NMT encoder can represent in a small number of neurons.

Original: 5.3 Linguistically Interpretable Neurons
Revised: 5.3 Linguistically Interpretable Neurons

Original: Neurons that activate on speciﬁc tokens or capture position in the sentence are important, as shownin the previous section.
Revised: Neurons that activate on speciﬁc tokens or capture position in the sentence are important, as shownin the previous section.

Original: But they are less interesting from the perspective of capturing languageinformation.
Revised: But they are less interesting from the perspective of capturing languageinformation.

Original: In this section, we investigate several linguistic properties by measuring predictivecapacity and visualizing neuron activations.
Revised: In this section, we investigate several linguistic properties by measuring predictivecapacity and visualizing neuron activations.

Original: Parentheses Table 2 shows top neurons from each model for predicting that tokens are inside/outside of parentheses, quotes, or brackets, estimated by a GMM model.
Revised: Parentheses Table 2 shows top neurons from each model for predicting that tokens are inside/outside of parentheses, quotes, or brackets, estimated by a GMM model.

Original: Often, the parenthesesneuron is unique (low scores for the 2nd best neuron), suggesting that this property tends to be relatively localized.
Revised: Often, the parenthesesneuron is unique (low scores for the 2nd best neuron), suggesting that this property tends to be relatively localized.

Original: Generally, neurons that detect parentheses were ranked highly in most models bythe MaxCorr method, indicating that they capture important patterns in multiple networks.
Revised: Generally, neurons that detect parentheses were ranked highly in most models bythe MaxCorr method, indicating that they capture important patterns in multiple networks.

Original: The next ﬁgure visualizes the most predictive neuron in an English-Spanish model.
Revised: The next ﬁgure visualizes the most predictive neuron in an English-Spanish model.

Original: It activates positively (red) inside parentheses and negatively (blue) outside.
Revised: It activates positively (red) inside parentheses and negatively (blue) outside.

Original: Similar neurons were found in RNNlanguage models (Karpathy et al., 2015).
Revised: Similar neurons were found in RNNlanguage models (Karpathy et al., 2015).

Original: Next we consider more complicated linguistic properties.
Revised: Next we consider more complicated linguistic properties.

Original: Tense We annotated the test data for verb tense (with Spacy) and trained a GMM model to predicttense from neuron activations.
Revised: Tense We annotated the test data for verb tense (with Spacy) and trained a GMM model to predicttense from neuron activations.

Original: The following ﬁgure shows activations of a top-scoring neuron (0.

Revised: The following ﬁgure shows activations of a top-scoring neuron (0.

Original: F 1 ) from the English-Arabic model on the ﬁrst 5 test sentences.
Revised: F 1 ) from the English-Arabic model on the ﬁrst 5 test sentences.

Original: It tends to activate positively (redcolor) on present tense (“recognizes”, “recalls”, “commemorate”) and negatively (blue color) on past
Revised: It tends to activate positively (redcolor) on present tense (“recognizes”, “recalls”, “commemorate”) and negatively (blue color) on past

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: tense (“published”, “disbursed”, “held”).
Revised: tense (“published”, “disbursed”, “held”).

Original: These results are obtained with a charCNN representation,which is sensitive to common sufﬁxes like “-ed”, “-es”.
Revised: These results are obtained with a charCNN representation,which is sensitive to common sufﬁxes like “-ed”, “-es”.

Original: However, this neuron also detects irregularpast tense verbs like “held”, suggesting that it captures context in addition to sub-word information.
Revised: However, this neuron also detects irregularpast tense verbs like “held”, suggesting that it captures context in addition to sub-word information.

Original: The neuron also makes some mistakes by activating weakly positively on nouns ending with “s”(“videos”, “punishments”), presumably because it gets confused with the 3rd person present tense.
Revised: The neuron also makes some mistakes by activating weakly positively on nouns ending with “s”(“videos”, “punishments”), presumably because it gets confused with the 3rd person present tense.

Original: Table 3 shows correlations of neurons most correlated with this tense neuron, according to
Revised: Table 3 shows correlations of neurons most correlated with this tense neuron, according to

Original: MaxCorr .
Revised: MaxCorr .

Original: All these neurons are highly predictive of tense: all are in the top 5 and 9 out of 15(non-auto-encoder) neurons have the highest F 1 score for predicting tense.
Revised: All these neurons are highly predictive of tense: all are in the top 5 and 9 out of 15(non-auto-encoder) neurons have the highest F 1 score for predicting tense.

Original: The auto-encoder English models are an exception, exhibiting much lower correlations with the English-Arabic tenseneuron.
Revised: The auto-encoder English models are an exception, exhibiting much lower correlations with the English-Arabic tenseneuron.

Original: This suggests that tense emerges in a “real” NMT model, but not in an auto-encoder thatonly learns to copy.
Revised: This suggests that tense emerges in a “real” NMT model, but not in an auto-encoder thatonly learns to copy.

Original: Interestingly, English-Chinese models have somewhat lower correlated neuronswith the tense neuron, possibly due to the lack of explicit tense marking in Chinese.
Revised: Interestingly, English-Chinese models have somewhat lower correlated neuronswith the tense neuron, possibly due to the lack of explicit tense marking in Chinese.

Original: The encoderdoes not need to pay as much attention to tense when generating representations for the decoder.
Revised: The encoderdoes not need to pay as much attention to tense when generating representations for the decoder.

Original: Other Properties We found many more linguistic properties by visualizing top neurons rankedby our methods, especially with MaxCorr .
Revised: Other Properties We found many more linguistic properties by visualizing top neurons rankedby our methods, especially with MaxCorr .

Original: We found neurons that activate on numbers, dates,adjectives, plural nouns, auxiliary verbs, prepositions, and more.
Revised: We found neurons that activate on numbers, dates,adjectives, plural nouns, auxiliary verbs, prepositions, and more.

Original: We do not include a detaileddiscussion for lack of space, and instead brieﬂy discuss noun phrase segmentation, a compositionalproperty above the word level.
Revised: We do not include a detaileddiscussion for lack of space, and instead brieﬂy discuss noun phrase segmentation, a compositionalproperty above the word level.

Original: We obtained noun phrase segmentation (using Spacy) and classiﬁedtokens as inside, outside, or beginning of a noun phrase (IOB scheme), and found high-scoringneurons (60-80% accuracy) in every network.
Revised: We obtained noun phrase segmentation (using Spacy) and classiﬁedtokens as inside, outside, or beginning of a noun phrase (IOB scheme), and found high-scoringneurons (60-80% accuracy) in every network.

Original: Many of these neurons were ranked highly by the
Revised: Many of these neurons were ranked highly by the

Original: MaxCorr method.
Revised: MaxCorr method.

Original: In contrast, other methods did not rank such neurons very highly.
Revised: In contrast, other methods did not rank such neurons very highly.

Original: We visualize the top scoring neuron (79%) 
Revised: We visualize the top scoring neuron (79%) an English-Spanish model below

Revised: Notice how the word noun phrases, but negatively is

Original: [Figure]
Revised: [Figure]

Original:  Controlling Translations
Revised:  Controlling Translations

Original: In this section, we explore a potential beneﬁt of ﬁnding important neurons with linguisticallymeaningful properties: controlling the translation output.
Revised: In this section, we explore a potential beneﬁt of ﬁnding important neurons with linguisticallymeaningful properties: controlling the translation output.

Original: This may be important for mitigatingbiases in neural networks.
Revised: This may be important for mitigatingbiases in neural networks.

Original: For instance, gender stereotypes are often reﬂected in automatictranslations, as the following motivating examples from Google Translate demonstrate.
Revised: For instance, gender stereotypes are often reﬂected in automatictranslations, as the following motivating examples from Google Translate demonstrate.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation][Figure] [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: doctor

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The Turkish sentences (1a, 2a) have no gender information—they can refer to either male or female.
Revised: The Turkish sentences (1a, 2a) have no gender information—they can refer to either male or female.

Original: But the MT system is biased to think that doctors are usually men and nurses are usually women,so its generated translations (1b, 2b) represent these biases.
Revised: But the MT system is biased to think that doctors are usually men and nurses are usually women,so its generated translations (1b, 2b) represent these biases.

Revised: If we know the correct gender fromanother source such as metadata, we may want to encourage the system to output a translation withthe correct gender.

Original: We conjecture that if a given neuron matters to the model, then we can control the translation inpredictable ways by modifying its activations.
Revised: We conjecture that if a given neuron matters to the model, then we can control the translation bymodifying its activations.

Original: To do this, we ﬁrst encode the source sentence asusual.
Revised: To do this, we ﬁrst encode the source sentence as usual.

Original: Before decoding, we set the activation of a particular neuron in the encoder state to a value ↵ ,which is a function of the mean activations over a particular property (deﬁned below).
Revised: Before decoding,we set the activation of a particular neuron in the encoder state to a value α (deﬁned below).

Original: To evaluateour ability to control the translation, we design the following protocol:
Revised: Toevaluate our ability to control the translation, we design the following protocol:

Original: Tag the source and target sentences in the development set with a desired property, such as gender(masculine/feminine).
Revised: Tag the source and target sentences in the development set with a desired property, such as gender(masculine/feminine).

Original: We use Spacy for these tags.2.
Revised: We use Spacy for these tags.2.

Original: Obtain word alignments for the development set with using an alignment model trained on 2million sentences of the UN data.
Revised: Obtain word alignments for the development set with using an alignment model trained on million sentences of the UN data.

Original: We use fast align (Dyer et al., 2013) with default settings.3.
Revised: We use fast align (Dyer et al., 2013) with default settings.3.

Original: For every neuron in the encoder, predict the target property on the word aligned to its sourceword activations using a supervised GMM model.
Revised: For every neuron in the encoder, predict the target property on the word aligned to its sourceword activations using a supervised GMM model.

Original: For every word having a desired property, modify the source activations of the top k neuronsfound in step 3, and generate a modiﬁed translation.
Revised: For every word having a desired property, modify the source activations of the top k neuronsfound in step 3, and generate a modiﬁed translation.

Original: The modiﬁcation value is deﬁnedas
Revised: The modiﬁcation value is deﬁned

Original: [Equation]  µ 2 ) , where µ 1 and µ 2 are mean activations of the property we modify from andto, respectively (e g. modifying gender from masculine to feminine), and  is a hyper-parameter.5.
Revised: asα = µ 1 + β ( µ 1 − µ 2 ) , where µ 1 and µ 2 are mean activations of the property we modify from andto, respectively (e.g. modifying gender from masculine to feminine), and β is a hyper-parameter.5.

Original: Tag the output translation and word-align it to the source.
Revised: Tag the output translation and word-align it to the source.

Original: Declare success if the source word wasaligned to a target word with the desired property value (e.g. feminine).
Revised: Declare success if the source word wasaligned to a target word with the desired property value (e.g. feminine).

Original: 6.1 Results
Revised: 6.1 Results

Original: Figure 4 shows translation control results in an English-Spanish model.
Revised: Figure 4 shows translation control results in an English-Spanish model.

Original: We report success rate—thepercentage of cases where the word was aligned to a target word with the desired property–and theeffect on BLEU scores, when varying ↵ .
Revised: We report success rate—thepercentage of cases where the word was aligned to a target word with the desired property–and theeffect on BLEU scores, when varying α .

Original: Our tense control results are the most successful, with upto 67% success rate for changing past-to-present.
Revised: Our tense control results are the most successful, with upto 67% success rate for changing past-to-present.

Original: Modiﬁcations generally degrade BLEU, but theloss at the best success rate is not large (2 BLEU points).
Revised: Modiﬁcations generally degrade BLEU, but theloss at the best success rate is not large (2 BLEU points).

Original: Controlling other properties seems more difﬁcult, with the best success rate for controlling numberat 37%, using the 5 top number neurons.
Revised: Controlling other properties seems more difﬁcult, with the best success rate for controlling numberat 37%, using the 5 top number neurons.

Original: Gender is the most difﬁcult to control, with a 21% successrate using the 5 top neurons.
Revised: Gender is the most difﬁcult to control, with a 21% successrate using the 5 top neurons.

Original: Modifying even more neurons did not help.
Revised: Modifying even more neurons did not help.

Original: We conjecture that theseproperties are more distributed than tense, which makes controlling them more difﬁcult.
Revised: We conjecture that theseproperties are more distributed than tense, which makes controlling them more difﬁcult.

Original: Futurework can explore more sophisticated methods for controlling multiple neurons simultaneously.
Revised: Futurework can explore more sophisticated methods for controlling multiple neurons simultaneously.

Original: This is different our results the previous section we predicted source side property be predictive

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Le Comit´e a appuy´e
Revised: { Ah(cid:152)@b(cid:11) ¤t(cid:152)(cid:0) wh(cid:20)(cid:152)A\ wh(cid:19) } Tn(cid:20)l(cid:152)(cid:0) d§¥(cid:11)w\(cid:14)d§(cid:1)⁄

Original: [Table]
Revised: [Table]

Original: appuie les efforts des autorit´es[Table] El Comit´e apoy´o
Revised: Le Comit´e a appuy´e/appuie les efforts des autorit´es

Original: [Table]
Revised: [Table]

Original: apoyaba [Table] apoya losesfuerzos de las autoridades
Revised: los

Original: [Table]
Revised: [Table]

Original: усилия властей [Table] S @
Revised: las

Original: [Table]
Revised: [Table]

Original: 6.2 Example translations
Revised: 6.2 Example translations

Original: We provide examples of controlling translation of number, gender, and tense.
Revised: We provide examples of controlling translation of number, gender, and tense.

Original: While these are cherrypicked examples, they illustrate that the controlling procedure can work in multiple properties andlanguages.
Revised: While these are cherrypicked, they illustrate that the controlling procedure can work in multiple properties and languages.

Original: We discuss language-speciﬁc patterns below.

Original: Number Table 4a shows translation control results for a number neuron from an English-Spanishmodel, which activates negatively/positively on plural/singular nouns.
Revised: Number Table 4a shows translation control results for a number neuron from an English-Spanishmodel, which activates negatively/positively on plural/singular nouns.

Original: The translation changes fromplural to singular as we increase the modiﬁcation ↵ .
Revised: The translation changes fromplural to singular as we increase the modiﬁcation α .

Original: We notice that using too high ↵ values yieldsnonsense translations, but with correct number: transitioning from the plural adjective particulares(“particular”) to the singular adjective ´util (“useful”), with valid translations in between.
Revised: We notice that using too high α values yieldsnonsense translations, but with correct number: transitioning from the plural adjective particulares(“particular”) to the singular adjective ´util (“useful”), with valid translations in between.

Original: Gender Table 4b shows examples of controlling gender translation for a gender neuron from thesame model, which activates negatively/positively on masculine/feminine nouns.
Revised: Gender Table 4b shows examples of controlling gender translation for a gender neuron from thesame model, which activates negatively/positively on masculine/feminine nouns.

Original: The translationschange from masculine to feminine synonyms as we increase the modiﬁcation ↵ .
Revised: The translationschange from masculine to feminine synonyms as we increase the modiﬁcation α .

Original: Generally, wefound it difﬁcult to control gender, as also suggested by the relatively low success rate.
Revised: Generally, wefound it difﬁcult to control gender, as also suggested by the relatively low success rate.

Original: Tense Table 4c shows examples of controlling tense when translating from English to ﬁve targetlanguages.
Revised: Tense Table 4c shows examples of controlling tense when translating from English to ﬁve targetlanguages.

Original: In all language pairs, we are able to change the translation from past to present bymodifying the activation of the tense neurons from the previous section (Table 3).
Revised: In all language pairs, we are able to change the translation from past to present bymodifying the activation of the tense neurons from the previous section (Table 3).

Original: In Spanish, weﬁnd a transition from past to imperfect to present.
Revised: In Spanish, weﬁnd a transition from past to imperfect to present.

Original: Interestingly, in Chinese, we had to use a fairlylarge ↵ value (in absolute terms), consistent with the fact that tense is not usually marked in Chinese.
Revised: Interestingly, in Chinese, we had to use a fairlylarge α value (in absolute terms), consistent with the fact that tense is not usually marked in Chinese.

Original:  Conclusion
Revised:  Conclusion

Original: We developed unsupervised methods for ﬁnding important neurons in NMT, and evaluated howthese neurons impact translation quality.
Revised: We developed unsupervised methods for ﬁnding important neurons in NMT, and evaluated howthese neurons impact translation quality.

Original: We analyzed several linguistic properties that are capturedby individual neurons using quantitative prediction tasks and qualitative visualizations.
Revised: We analyzed several linguistic properties that are capturedby individual neurons using quantitative prediction tasks and qualitative visualizations.

Original: We alsodesigned a protocol for controlling translations by modifying neurons that capture desired properties.
Revised: We alsodesigned a protocol for controlling translations by modifying neurons that capture desired properties.

Original: Our analysis can be extended to other NMT components (e.g. the decoder) and architectures Gehringet al. (2017); Vaswani et al. (2017), as well as other  tasks.
Revised: Our analysis can be extended to other NMT components (e.g. the decoder) and architectures (Gehring et al., 2017; Vaswani et al., 2017), as well as other datasets from different domains,and even other NLP tasks.

Original: We believe that more work should be doneto analyze the spectrum of localized vs. distributed information in neural language representations.
Revised: We believe that more work should be done to analyze the spectrum of localized vs. distributed information in neural language representations.

Original: We would also like to develop more sophisticated ways to control translation output, for example bymodifying representations in variational NMT architectures (Zhang et al. et al
Revised: We would also like to developmore sophisticated ways to control translation output, for example by modifying representations invariational NMT architectures (Zhang et al., 2016; Su et al., 2018).