Paper:
According to parallel distributed processing (PDP) theory in psychology, neural networks (NN) learn distributed rather than interpretable localist representations. This view has been held so strongly that few researchers have analysed single units to determine if this assumption is correct. However, recent results from psychology, neuroscience and computer science have shown the occasional existence of local codes emerging in artificial and biological neural networks. In this paper, we undertake the first systematic survey of when local codes emerge in a feed-forward neural network, using generated input and output data with known qualities. We find that the number of local codes that emerge from a NN follows a well-defined distribution across the number of hidden layer neurons, with a peak determined by the size of input data, number of examples presented and the sparsity of input data. Using a 1-hot output code drastically decreases the number of local codes on the hidden layer. The number of emergent local codes increases with the percentage of dropout applied to the hidden layer, suggesting that the localist encoding may offer a resilience to noisy networks. This data suggests that localist coding can emerge from feed-forward PDP networks and suggests some of the conditions that may lead to interpretable localist representations in the cortex. The findings highlight how local codes should not be dismissed out of hand.

Review:
This paper studies the development of localist representations in the hidden layers
of feed-forward neural networks.
The idea is interesting and the findings are intriguing.  Local codes
increase understandability and could be important for better
understanding natural neural networks. Understanding how local codes
form and the factors that increase their likelihood is critically
important.  This is a good start in that direction, but still leaves
open many questions.  The issues raised in the Conclusions section are
also very interesting -- do the local codes increase with networks
that generalize better, or with overtrained networks?
A  weakness in this paper (admitted by the authors in the
Conclusions section) is the dependence of the results on the form of input
representation.  If we consider the Jennifer Aniston cells, they do
not receive as input as well separated inputs as modeled in this
paper.  In fact the input representation used in this study is already
a fairly localist representation as each 1 unit is fairly selectively
on for its own class and mostly off for the other classes.  It will be
very interesting to see the results of hidden layers in deep networks
operating on natural images.
Please give your equation for selectivity.  On Page 2 it is stated "We
use the word ‘selectivity’ as a quantitative measure of the difference
between activations for the two categories, A and not-A, where A is
the class a neuron is selective for (and not-A being all other
classes)."  However you state that neurons were counted as a local
code if the selectivity was above .05.  A difference between
activations for the two categories of .05 does not seem very
selective, so I'm thinking you used something other than the
mathematical difference.
What is the selectivity of units in the input codewords?  With no
perturbation, and S_x=.2, w_R=50, w_P=50, the units in the prototype
blocks have a high selectivity responding with 1 for all patterns in
their class and with 0 for 8/9 of the patterns in the other classes.
Could this explain the much higher selectivity for this case in the
hidden units?  I would like to see the selectivity of the input units
for each of the plots/curves.  This would be especially interesting
for Figure 5.
It is stated that LCs emerge with longer training and that ReLU
neurons may produce more LCs because they train quicker and all
experiments were stopped at 45,000 epochs.  Why not investigate this
by changing learning rates for one of ReLu or sigmoidal units to more
closely match their training speed?  It would be interesting to see if
the difference is simply due to learning rate, or something deeper
about the activation functions.
You found that very few local codes in the HLNs were found when a
1-hot ouput encoding was used and suggest that this means that
emergent local codes are highly unlikely to be found in the
penultimate layer of deep networks.  If your inputs are a local code
(e.g. for low w_R), you found local codes above the layer of local
codes but in this result not below the layer of local codes which
might also imply (as you say in the Conclusions) that more local
coding neurons may be found in the higher layers (though not the
penultimate one as you argue).  Could you analyze how the selectivity
of a hidden layer changes as a function of the selectivity in the
lower and higher layers?
Minor Note -- The Neural Network Design section looks like it still
has draft notes in it.