From: NIPS.cc/2018/Workshop/IRASL/Paper39/Authors
title: Thank you for helpful comments
comment: We would like to thank all three reviewers for their thoughtful reviews and useful feedback. We have uploaded a revised version attempting to address the main comments. Here are specific responses. 

Reviewer 1
- We were referring to the larger localist/distributed debate in artificial intelligence, which includes cognitive scientists and computer scientists. Part of this includes computational modeling of different tasks, such as our case of machine translation. We have modified that sentence to refer to the broad AI field. 
- Your understanding of figure 1 is correct. We’ve added an explanation to the caption. 
- Your understanding of MinCorr is correct. 
- Thank you for the further thoughts and for providing great ideas for next steps. 

Reviewer 2
- Thank you for the suggestion to extend the analysis to more domains and the hypothesis that the low success rate of gender control has to do with the domain. We would like to explore this in the future and added a remark in the conclusion. 
- The difference between different models is that they are trained on different parts of the training data, as noted in Section 4. We will provide more details in an expanded version of this work. 
- Fixed the typo in figure 1 caption. 

Reviewer 3
- We expanded the motivation to translation control experiments in section 6.

From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper39/AnonReviewer2
title: Interpretability for NMT by ranking neurons according to their importance. 
review: Paper aims to analyze the impact and utility of individual neurons in NMT, and a very good fit for the workshop.

Authors propose unsupervised methods for detecting important neurons based on correlations between independently trained models. After detecting such neurons, linguistic properties captured by these neurons are investigated, via supervised classification tasks. Last, by altering the representation of "salient" neurons, translations outputs are controlled, in tense, number and gender categories. 

Study relies on the hypothesis that different NMT models learn similar properties, hence similar important neurons emerge in different models. To capture the similarity, four different correlation methods used (max-corr, min-corr, linear regression and SVCCA) . Further, detected neurons are verified, by masking them during inference, by supervised verification (obtained via linguistic annotations) and visualizations.  

The most interesting experiment, to me, is the gender control, hinting that properties that are complex in nature are distributed within the connectionist system. 

Extending the study to span more architectures, and more importantly, different training set domains would be very interesting. I'm not sure UN corpus is the best domain for the analysis, since it consists of dominantly boiler plate text, and in general, corpus is very low entropy (notice very high bleu scores). Considering the formality of the dataset, low success rate in gender may also be explained by the dataset bias. It would be again interesting to extend the datasets used to cover more diverse set of domains. 

Section for experimental setup, how different models are trained, is very terse. What is different among different models should be clarified, is it only random seed, the order of examples, architectures etc. Would be great to detail them in the appendix perhaps. 

typo. 
- figure 1 caption, methods methods
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper39/AnonReviewer1
title: Interesting, thorough analysis of individual neurons in NMT models
review: This is a well written paper showing interesting properties of individual neurons of NMT models. They do a good job of supporting their claims, and perform a controllability experiment that illustrates their results. 

This paper is a good fit for this workshop--literally presenting a way to understand individual neurons. I appreciate the parallels drawn here between understanding NMT models and understanding NMT models. It encourages folks to try the, little more challenging, task of interpreting NLP models. I don't think any one method in this paper is particularly novel, but they do a through job of presenting the work and there are interesting insights to be drawn from it. 

I'm not entirely convinced that individual neurons are _better_ than looking at looking at "whole vector representations," or PCA directions. To me they're _different_ and show you different things about the model. If this is a claim you want to make, I'd clarify it and support it better. 

Some clarifications I'd appreciate:
* I don't quite see the connection to the localist/distributed debate in neural cognitive science. I understand it from the localist/distributed debate in neural networks, but either missed (somewhere in the paper) or do not understand why this contributes to the cognitive science discussion. I think your findings are interesting enough without this point, and leaving it in the last paragraph of the intro leads me to believe it's more important than perhaps you intend. 
* Figure 1 is not very clear. I spent a good few minutes trying to figure out what the different circles and rectangles were. Could you label it more clearly? I decided that the rectangles were different models and the circles were neurons. If this is incorrect, then this figure needs to be revisited. 
* To find the neuron of minimum correlation you take the max correlated neurons across all models, then choose the one neuron that is the minimum of all those neurons?

Further thoughts:
* This analysis on Transformer will be interesting (as you allude to in a comment on future work). Since position seems to be a salient characteristic that neurons explain well, I wonder if this can lead us to insight on how to improve Transformer's position encoding.  
* I would greatly appreciate a study of random seeds. Instead of looking across models trained on different language pairs, what does comparing a model trained on one language pair with different random seeds tell us about shared neurons and learned representations?
* I would like to see how many PCs were needed to capture 99% of the variance. In particular, the cumsum plot of captured variance vs. PC id would be appreciated. 
* The controllability experiment is really nice. Thanks for doing that.
  - We're probably pretty far from using this in production systems (say for controlling gender issues), but the paper doesn't over-claim and gives promising directions for future work.
* I'd like to see how the next most predictive neurons perform. Both for controllability, and whether or not the next predictive neuron is also predictive of other human-interpretable features. This asks a question of whether a neuron is responsible for one feature or for many. 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper39/AnonReviewer3
title: Official Review: "Identifying and Controlling Important Neurons in Neural Machine Translation"
review: Finding "meaningful" or "salient" neurons is a great way of looking at the inner parts of a model. This has been done in Computer Vision and this work is extending similar ideas to Neural Machine Translation models. Previous works in this area had two limitations, as author(s) mentioned. First, they rely on existing linguistic algorithms/data, and they tend to look vectors of distributed representation rather by looking at individual activations (except for SVCCA method). This work tackles both of them (to some extent) by ranking neurons (at timestep t) based on correlation between pairs of models, different epochs, or different datasets. They also propose a controlling mechanism for the translation models which I would appreciate it if the motivation was better explained specially in the introduction section.

Beside the fact that some of the analysis doesn't translate to immediate increase in the performance, I would say the work is of great value and good contribution, with clear writing and experimentation.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct