Paper:
IDENTIFYING AND CONTROLLING IMPORTANT NEURONS IN NEURAL MACHINE TRANSLATION
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

Review:
The authors propose a number of methods to identify individual important neurons in a machine translation system. The crucial assumption, drawn from the computer vision literature, is that important neurons are going to be correlated across related models (e.g. models that are trained on different subsets of the data). This hypothesis is validated to some extent: erasing the neurons that scored highly on these measures reduced BLEU score substantially. However, it turns out that most of the activation of the important neurons can be explained using sentence position. Supervised classification experiments on the important neurons revealed neurons that tracked properties such as the span of parentheses or word classes (e.g., auxiliary verbs, plural nouns, etc).
Strengths:
* The paper is very well written and provides solid intuitions for the methods proposed.
* The methods seem promising, and the degree of localist representation is striking.
* The methods may be able to address the question of *how* localist the representations are (though no numerical measure of localism is proposed).
* There is a correlation between the neuron importance metrics proposed in the paper and the effect on BLEU score of erasing those neurons from the network (of course, it’s not clear what particular linguistic properties are affected by this erasure - the decrease BLEU may reflect inability to track specific word tokens more than any higher-level linguistic property).
Weaknesses:
* It wasn't clear to me why the neurons that track particular properties (e.g., being inside a parentheses) couldn't be identified using a supervised classifier to begin with, without first identifying "important" neurons using the unsupervised methods proposed in the paper. The unsupervised methods do show their strength in the more exploratory visualization-based analyses -- as the authors point out (bottom of p. 6), the neuron that activates on numbers but only at the beginning of the sentence does not correspond to a plausible a-priori hypothesis. Still, most of the insight in the paper seems to be derived from the supervised experiments.
* The particular linguistic properties that are being investigated in the classification experiments are fairly limited. Are there neurons that track syntactic dependencies, for example?
* I wasn't sure how the GMMs (Gaussian mixture models) for predicting linguistic properties from neuron activations were set up.
* It's nice to see that individual neurons function as knobs that can change the gender or tense of the output (with varying accuracy). At the same time, I was unable to follow the authors' argument that this technique could be used to reduce gender bias in MT.
* I wasn't sure what insight was gained from the SVCCA analyses -- this method seems to be a bit of a distraction given the general focus on localist vs. distributed representation. In general, I didn’t come away with an understanding of the pros and cons of each of the methods.