Paper:
Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCs emerging in response to invariant features, and this finding is robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, increasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep networks (2 hidden layers) which have many LCs lose them when common aspects of deep-NN research are applied (large training data, ReLU activations, early stopping on training accuracy and softmax), suggesting that LCs may not be found in deep-NNs. Switching to more biologically feasible constraints (sigmoidal activation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are not found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by a lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise of LCs be required, this work suggests how to tune a NN so they emerge.

Review:
Paper Overview: 
This paper aims to study when hidden units provide local codes by analyzing the hidden units of trained fully connected classification networks under various architectures and regularizers.  The main text primarily studies networks trained on a dataset where binary inputs are structured to represent 10 classes with each input containing a subset of elements indicative of the class label.  The work also studies fully connected networks trained on the MNIST dataset (with the addition of some pixels indicating each class label).  After enumerating the number of local codes observed under these different settings, the authors conclude the following: (1) "common" properties of deep neural networks & modern datasets seem to decrease the number of local codes (2) specific architectural choices, regularization choices & dataset choices seem to increase the number local codes (i.e. increasing dropout, decreasing dataset size, using sigmoidal activations etc.).   The work then state that these insights may suggest how to train networks to have local codes emerge.  
Review: 
I particularly liked the simple dataset the authors construct in determining whether local codes emerge in hidden units, especially since deep networks and dataset used in practice are overly complex to gain insight for this behavior.  However, I find the overall message to be a bit confusing, especially in regard to using the analysis to construct networks with emergent local codes.  In particular, I feel that the authors could strengthen this work greatly by using their findings to train a deeper neural network for which local codes do emerge on a more realistic dataset.  Furthermore, this work would be significantly more impactful if a network with more local codes does generalize better, but that is unclear as of now (especially since local codes seem to not emerge in practical settings even though these networks are state of the art).  
Criticisms/Questions:
(1) Main:  I'm somewhat confused about the main takeaway from this work in terms of understanding when local codes actually emerge in deep neural networks.  The authors seem to have a number of very specific conditions that are both architecture and dataset dependent, and overall I feel the message would be much stronger if the authors were able to rigorously study perhaps just a few of these conditions across many more settings.  For example, even just studying the impact of activation and providing some conditions/theory or a clearer understanding of which nonlinearities lead to more local codes would be insightful.  The current work seems to be more broad instead of tackling one of these properties in depth. 
(2) I am a bit confused about the thresholds used by the authors in determining whether a hidden unit provides a local code or not. Do you just determine if there is some threshold given by the unit that separates out all points of one class from the rest? 
(3) After several experiments, there are some heavy conjectures trying to rationalize the result of the experiment.  As an example, the authors provide statements like "ReLU is a more powerful activation function than sigmoid."  However, this statement in particular is not exactly correct, since given enough width, networks with either activation function should be able to interpolate the training data.  Another example of this is at the bottom of page 7, when the authors provide 5 possible explanations as to why local codes don't emerge in modern training settings.  It is unclear which of these explanations are true, but it would be great if the authors could actually provide a cleaner rationalization.  
Minor criticisms:
(1) I've seen a number of different conventions for how to refer to the depths of networks, and I believe what you refer to as 3 layer networks would conventionally be referred to as 2-layer networks for theory audiences (as there are 2 weight matrices involved) or 1-hidden layer networks for empirical audiences.  I think adding a figure in the appendix for your architecture would clear up any confusion immediately.  
(2) Some of the formatting is a bit awry: there are references to figures that appear as ?? (see page 8 paragraph 3).  
(3) It would be nice to provide a consistent legend in some of the figures.  For example, Figure 4b has no indication for which settings the colors represent.  
(4) As there seem to be a lot of experiments numbered 1-12, I think it would be much more readable to have different subsections on the different settings and outline the experiments in the subsection more clearly.  Referring back to these numbers on page 3 & 4 constantly makes it less readable.  
(5) I quite liked Figure 8 in the Appendix.  I feel that this would have been a great figure to put towards the front of the paper to provide an example of local codes emerging.