From: NeurIPS.cc/2019/Workshop/Neuro_AI/Program_Chairs
title: Paper Decision
decision: Accept (Poster)

From: NeurIPS.cc/2019/Workshop/Neuro_AI/Paper17/AnonReviewer2
title: Improving interpretability by analyzing bulk-behavior of subsets of neuron
importance: 4: Very important
importance_comment: The technique could potentially improve the interpretability of DNN models for feature understanding
rigor_comment: Some rigor is needed in the experimental evaluation. The DeepBind study is not clearly explained and lacks technical rigor or has not been explained clearly. The claim "DeepBind model learns something from raw data" is too vague.
clarity_comment: Technical details were omitted due to space constraints. The main idea and outline of implementation and methodology were explained. Other works considering attribution and importance scores have been cited clearly.
clarity: 3: Average readability
evaluation: 3: Good
intersection_comment: A technique is discussed with application towards analyzing DNN models which are prevalent in machine learning/AI. This technique aims to make models and features learned by DNNs more interpretable, and the principle is inspired by biological hypotheses of functional modularity.
intersection: 4: High
comment: Several questions are unanswered. The experiments demonstrate preliminary investigations, at best. More rigorous experimentation and crossvalidation are needed.
Is the phenomenon observed across different datasets with more diversity?
Is the possibility of memorization/overfitting ruled out? Is that even a concern, or is memorization what functional modularity implicitly refers to?
Does the fact that the network is convolutional help in identifying the biclusters because now the neurons are more structured than fully-connected networks?
The choice of attribution scores etc. is not clearly justified.
technical_rigor: 3: Convincing
category: Neuro->AI

From: NeurIPS.cc/2019/Workshop/Neuro_AI/Paper17/AnonReviewer1
evaluation: 2: Poor
intersection: 2: Low
importance_comment: It is potentially useful to show if functional modularity exists in artificial neural networks. The authors claim they find functional modules by applying biclustering to the neuron-sample matrix but current analyses primarily evaluated on MNIST dataset are not clear and limited to tell whether the existence of functional modules.
clarity: 2: Can get the general idea
technical_rigor: 2: Marginally convincing
intersection_comment: Although the authors claim the usage of neuron attribution and biclustering can be applied to real neurons, there is no explanation of how exactly to apply. Other than that, this paper is mainly focusing on artificial neurons.
rigor_comment: There is no explanation of the selection of tested layer and no comparison between results of other layers. Similarly, the choice of the number of biclusters is also confusing and no explanation is provided. In figure 2c, it looks like multiple biclusters contribute to the same class, then why not change the number of biclusters and evaluate more results.
comment: It could be an interesting direction in terms of looking for functional modules in artificial neural networks via clustering methods but more rigorous analyses and clearer explanations are needed.
importance: 2: Marginally important
title: interesting direction but more rigorous analyses are needed
category: AI->Neuro
clarity_comment: All figures are too small to follow and understand, especially figure 2c which has labels overlapped. The grey tone color of neuron-sample matrix makes figure 1 hard to visualize.

From: NeurIPS.cc/2019/Workshop/Neuro_AI/Paper17/AnonReviewer3
evaluation: 1: Very poor
intersection: 2: Low
importance_comment: If they exist, finding functionally different groups of units in a DNN and using them to generate hypotheses in the brain is an important goal. The authors apply their technique to the second layer of a network trained to recognize digits. No evidence is provided that this network is functionally similar to the brain or that their technique would generalize to more complex ‘sensory networks’. Thus it is difficult to tell from the paper whether their potentially important technique is important.

clarity: 2: Can get the general idea
technical_rigor: 2: Marginally convincing
intersection_comment: They do mention that this method of clustering could be applied to neurons. Does not mention how this might work in practice or what would be gained or why it should be preferred over other clustering methods that have been applied to neurons. 
rigor_comment: Many decisions were not well motivated, justified or described at least in brief. For example: choice of attribution method, choice of network and layer, number of clusters.  The space of parameters and network should have been explored more thoroughly to be convincing. Or at least justification for why their methods did not require more thorough testing.
comment: A potentially interesting and important technique for clustering. More motivation for the decisions in their study of this algorithm. Figures need to be improved in terms of contrast and relative sizes of figure labels. 
importance: 2: Marginally important
title: Novel technique for clustering neurons but not carefully evaluated or motivated.
category: AI->Neuro
clarity_comment: Unclear whether units under study from MNIST were rectified or not. This would have a big difference on attribution depending on how sparse units were.
Figure 1a and c are not well scaled and the contrast is very low. A log scale might have helped and larger labels. 
In line 65 it is not clear what 'the constructed matrix' is (neuron-sample, e, a). 
Similarly figure legends/labels in Figure 2 C are overlapping and tiny. Not vectorized so pixellated.