From: ICLR.cc/2019/Conference/Paper824/Area_Chair1
metareview: This work provides two contributions: 1) Brain-Score, that quantifies how a given network's responses compare to responses from natural systems; 2) CORnet-S, an architecture trained to optimize Brain-Score, that performs well on Imagenet.
As noted by all reviewers, this work is interesting and shows a promising approach to quantifying how brain-like an architecture is, with the limitations inherent to the fact that there is a lot about natural visual processing that we don't fully understand. However, the work here starts from the premise that being more similar to current metrics of brain processes is by itself a good thing -- without a better understanding of what features of brain processing are responsible for good performance and which are mere by-products, this premise is not one that would appeal to most of ICLR audience. In fact, the best performing architectures on imagenet are not the best scoring for Brain-Score. Overall, this work is quite intriguing and well presented, but as pointed out by some reviewers, requires a "leap of faith" in matching signatures of brain processes that most of the ICLR audience is unlikely to be willing to take.
confidence: 4: The area chair is confident but not absolutely certain
recommendation: Reject
title: Interesting take on quantifying similarity of networks to brain visual processing, unclear significance of that result for ICLR audiences

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer1
title: concerns addressed
comment: Thank you for answering my comments, and for running the additional tests.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: additional questions?
comment: Hi,

we were wondering if our new analyses and comments were sufficient or whether you had additional remarks that we could address before the deadline?

From: ICLR.cc/2019/Conference/Paper824/Authors
title: additional questions?
comment: Hi,

we were wondering if our new analyses and comments were sufficient or whether you had additional remarks that we could address before the deadline?

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer2
title: Concerns addressed
comment: Thank you, my concerns were satisfactorily addressed.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: clarifications & generalization analyses
comment: Thank you for your review.

1. Regarding CORnet optimization:
  a. First, we’d like to clarify that the search for a good CORnet architecture was done by hand. We trained a few models with different architectures, evaluated their ImageNet and Brain-Score performance, and tried to improve architectural choices in the next iteration. While we tried to limit our knowledge of Brain-Scores during model building, you are rightfully pointing out that CORnet ought to be evaluated on an independent set.

  b. We therefore collected a new behavioral dataset that used new objects that we’ve not used before in any of our benchmarks. The correlation between the old and the new behavioral rankings was strong (.83).

  c. We also compared rankings on an independent neural dataset collected on the same images as before, and found a very strong correlation too (.93)

  d. To push models further, we also compared the rankings of neural predictivity on a very dissimilar dataset (a subset of MS COCO). The correlation was still robust (.76)

  e. The details of these analyses can be found in Section 4.2 and also in Appendix C. Overall, CORnet-S is among the top models for all these analyses.

2. Regarding quantifying model generalization on other Machine Learning datasets: This is a good point, thus we strived to provide some measure of generalization in the revised manuscript (Fig. 2; Section 4.2). Following Kornblith et al. (2018), we compared model rankings when their classifiers were retrained for CIFAR-100. Here, we observed that Brain-Score is a good predictor of how well models will transfer to CIFAR-100 as fixed feature encoders (r=.69).

3. Regarding the idea to optimize model search not using Brain-Score. This is an interesting idea, however, one that would take many more that just a few weeks to test:
  a. As mentioned before, the search is not automated, so it might take a very long time (in the case of CORnet it took over a year)

  b. Training these models on ImageNet takes a few days and several GPUs at least, so given resource constraints, it would be challenging to perform this search.

  c. Given how correlated Brain-Score and ImageNet performance are, it is not unlikely that other datasets would lead to a similar circuitry. The key difference between optimizing for Brain-Score and optimizing for another dataset is that Brain-Score is not a proxy to what we want. Rather, it is the exact measure for how brain-like a given model is. Adding new recordings to Brain-Score that break existing models will thus further constrain these mechanistic hypotheses of the brain. By optimizing for Brain-Score we are making sure that we are optimizing for our ultimate goal — a model of the human visual system.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: generalization analyses
comment: Thank you for your review!

1. Contribution to ML:

  a. Following your comment, we tested how similar model ranking on Brain-Score is to the ranking on four independent and very different datasets (see Section 4.2 and Figure 3). We found a robust generalization of scores to other brain datasets (neural and behavioral) as well as to CIFAR-100. Using the models from this study as fixed feature encoders, Brain-Score is quite predictive of how well a model will perform on this ML dataset.

  b. Taking a step back, we emphasized in the revised paper that the main purpose of Brain-Score is to help build brain-like models. Those may or may not be also good machine learning models. For instance, human visual system seems to be more robust to various image perturbations, so machine learning models could benefit from that. On the other hand, we explicitly demand models to make mistakes if humans make mistakes. This is typically orthogonal to typical ML goals of reaching maximal performance.

2. Contribution to neuroscience:
  a. In the revised manuscript, we attempted to clarify (Sections 4.2 and 5) that constraining models with more and more brain data will necessarily converge them to the inner workings of the brain. Already today with three limited datasets that currently underlie Brain-Score we observe a very high predictive power of neural responses in the intermediate layers of primate visual system. These models have not been trained in any way to match these internal representations. We think this is a strong indication that the models are already good mechanistic hypotheses of the inner workings of the primate visual system. Adding data to Brain-Score that breaks existing models will help us to further refine these models in making them more brain-like.

  b. Perhaps this comment was meant to question whether, under Brain-Score’s guidance, we would ever find the circuitries that match those in the brain. Having no anatomical constraints in Brain-Score at the moment, it would be hard to expect those circuitries to emerge. As detailed in our CORnet-S circuitry analysis, many different circuitries appear to work sufficiently well at the moment. Perhaps this reflects the lack of constraints in Brain-Score at the moment. On the other hand, it remains unclear to what extent we want to model visual system to begin with. Do we need spikes? Biochemical details? Or should we focus on functional properties only? With Brain-Score, we follow the latter, at least for now, and, given that models can predict neural and behavioral responses very accurately, we think they are already a good functional match which now needs to be further refined as described above.

(continued in separate comment due to max characters)

From: ICLR.cc/2019/Conference/Paper824/Authors
title: clarifications
comment: 
3. The importance of recurrence. We now performed a detailed analysis of what makes CORnet-S work and the amount of recurrence indeed turned out to be the main predictive factor (see Figure 5) of both Brain-Score and ImageNet top-1 performance. Among other factors that played a role, we identified the presence of a skip connection, the size of the bottleneck, and the number of convolutions within each model area (for ImageNet only) as the key contributing factors to CORnet-S’ performance.

4. We found that there was a correlation (.65 for V4 and .87 or IT) which is strong enough to connect neurons to behavior but not sufficient for behavior alone to explain the entire neural population, warranting a composite set of benchmarks.

5. Can the number of neurons explain differences in neural predictivity?
  a. In Figure 6 we show that there does not appear to be a correlation between the number of features and Brain-Score. 

  b. Also note that normally neural predictivity analyses are done by first PCA’ing features down to 1000 components and then building a regression map between neural responses and model responses. This is perhaps another good control showing that at least the number of features that go into regression is always matched across models.

6. Fig. 1 gray dots: We now included an explanation in the caption that these values come from a class of simplified five-layer models that had various hyperparameters manipulated (values captured at different points during model training). The purpose of this test was to see the correlation between ImageNet and Brain-Score in lower performance regimes.

7. Consider for instance two models with IT scores of .580 and .581. Our current approach would simply use these values for the final Brain-Score without re-weighting. We also considered an alternative approach to compute the Brain-Score where models would be ranked on each benchmark separately and then receive a mean “Brain-Rank”. However in that case, the above example with two very close values would lead to the two models receiving different ranks with the same distance as if the values were .580 and .620. This is just to say that we chose to preserve the distance in scores as opposed to a ranking approach. We clarified this in the paper.

8. Comparison against June 2018 arXiv paper. We now included a new section comparing CORnet-S to other recurrent models (see Section 3.2), including the June 2018 one. Briefly, CORnet-S is an attempt to build a high performing model (both on ImageNet and Brain-Score) while keeping it as simple as possible. The June model focuses more on the anatomical necessity of recurrent and feedback connections without constraining the model size. As a result, CORnet-S is substantially simpler (shallower, simpler connectivity, takes much less GPU memory, and is mapped to brain areas) than the June 2018 one.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: more analyses
comment: Thank you again for your review!

1. Comments on using temporal responses:
  This is a good suggestion that we were also interested in. To get a sense whether temporal information adds more discriminatory power on top of the existing benchmarks, we compared model predictivity at two response bins: 90-110 ms (early response) and 190-210 ms (late response), since, following the June 2018 arXiv paper, the main differences in model scores appeared to emerge at later time steps (>170 ms). The resulting model scores can be found in Figure 8. From our analysis, early responses seem to be well-predicted across most models. Differences emerge in the late responses, however they do not appear to be different from the model scores on the mean 70-170 ms data. Similarly to the arXiv paper, we found that a feed-forward model already predicts temporal responses very well, and the recurrent models only out-perform the feed-forward model when the image is explicitly taken off (cf. arXiv paper, Fig. 5C). We hypothesize that the images shown for the current set of recordings might simply not warrant the functional use of recurrent connections. However, we are hopeful that future recordings focusing in on this temporal aspect (such as https://www.biorxiv.org/content/early/2018/06/26/354753) might help to further distinguish between models. So far, we have not yet observed this and thus chose not to include these scores. To contribute to community efforts in this domain, we are releasing temporal (20 ms bins) neural recordings from the data presented in this paper in the Brain-Score platform.

2. Comments on performing more analyses:

  a. This is a great suggestion and we quantified the necessity of each component in the CORnet-S circuity (see new Fig. 5 in the updated paper). We identified that having enough recurrent steps was the single most predictive factor. Having a large enough bottleneck and a skip connection was other important factors, especially for ImageNet. Other factors that contributed mostly to ImageNet performance were the number of convolutions within an area, the total number of areas in the model, and the use of Batch Normalization. Again, an even larger bottleneck that originally considered helped slightly for ImageNet performance, but not for Brain-Score. However, having one less layer on the block circuitry or one more area in the model did not seem to affect Brain-Score significantly.

  b. We also varied the length of the “local” recurrent connection. Note that this recurrence is not exactly local as it goes two convolutional layers back. Reducing the length of this recurrent connection from two layers to one layer reduces model performance, and re-running features through the entire circuitry seems to be most beneficial. Considering that recurrence virtually increases the depth of the network, longer recurrent connections seem intuitive since they create the network with the largest virtual depth.

  c. It may also be interesting to add longer range feedback connections that would go across model areas, e.g., from V4 to V1. This is only meaningful when unrolling in time is done in a biological manner so that feedback can be properly integrated with feed-forward inputs. We tested several CORnet variants with biological time unrolling and feedback from V4 to V1 (but simpler block structure to fit the model into memory) but observed no real change in adding a feedback connection. Our interpretation is that feedback connections do not automatically help or hurt model’s performance but a wider search might pinpoint to specific types of connections that may help. However, this would be a topic for another study.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: Thanks!
comment: Great, thank you for the quick clarification! All of these are great suggestions, and we are currently running analyses to quantify what parts are critical to the model.

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer3
title: clarifying "type of weight"
comment: The question is generally asking about what specifically about the CORNET-S architecture makes it perform well 
on Brain-Score (and which for ImageNet) compared to other similar networks.    The types of weights refers to feedback vs recurrent vs convolutional filters vs pooling (and connections between which layers are most critical).
I was just asking for some form of ablation study or analysis similar to Figure 4 in https://arxiv.org/pdf/1807.00053v2.pdf  to show what parts of your architecture you feel are critical for a good 
model of visual processing in the brain.

From: ICLR.cc/2019/Conference/Paper824/Authors
title: can you please clarify "type of weight"?
comment: Thank you for your review. Could you please clarify what you mean by the "type of weight in the network"?

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer3
title: Not enough analysis
review: Please consider this rubric when writing your review:
1. Briefly establish your personal expertise in the field of the paper.
2. Concisely summarize the contributions of the paper.
3. Evaluate the quality and composition of the work.
4. Place the work in context of prior work, and evaluate this work's novelty.
5. Provide critique of each theorem or experiment that is relevant to your judgment of the paper's novelty and quality.
6. Provide a summary judgment if the work is significant and of interest to the community.

1. I am a researcher working at the intersection of machine learning and
biological vision.  I have experience with neural network models and
visual neurophysiology.

2. This paper makes two contributions: 1) It develops Brain-Score - a
dataset and error metric for animal visual single-cell recordings.  2)
It develops (and brain-scores) a new shallow(ish) recurrent network
that performs well on ImageNet and scores highly on brain-score. 

3. The development of Brain-Score is a useful invention for the field.  A nice
aspect of Brain-Score is that responses in both V4 and IT as well as behavioral
responses are provided.   I think it could be more useful if the temporal dynamics (instead of the
mean number of spikes) was included.    This would allow to compare temporal
responses in order to compare "brain-like" matches.

4. This general idea is somewhat similar to a June 2018 Arxiv paper
(Task-Driven Convolutional Recurrent Models of the Visual System)
https://arxiv.org/abs/1807.00053
but this is a novel contribution as it is uses the Brain-Score dataset.

One limitation of this approach relative to the June 2018 ArXiv paper
is that the Brain-Score method is just representing the mean neural
response to each image - The Arxiv paper shows that different models
can have different temporal responses that can also be used to decide
which is a closer match to the brain.

5. More analysis of why CORNET-S is best among compact models would greatly
strengthen this paper.  What do the receptive fields look like?  How do they compare
to the other models.  What about the other high performing networks (e.g. DenseNet-169)?
How sensitive are the results to each type of weight in the network?   What about feedback connections
(instead of local recurrent connections)? 

6. This paper makes a significant contribution, in part due to the
development and open-sourcing of Brain-Score.  The significance of the
contribution of the CORnet-S architecture is limited by the
lack of analysis into what aspects make it better than other models.




rating: 5: Marginally below acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer2
title: Interesting attempt to bridge the gap between ML and neuroscience
review: In this interesting study, the authors propose a score (BrainScore) to (1) compare neural representations of an ANN trained on imagenet with primate neural activity in V4 and IT, and (2) test whether ANN and primate make the same mistakes on image classification.  They also create a shallow recurrent neural network (Cornet) that performs well according to their score and also reasonably well on imagenet classification task given its shallow architecture.

The analyses are rigorous and the idea of such a score as a tool for guiding neuroscientists building models of the visual system is novel and interesting.

Major drawbacks:

1. Uncertain contribution to ML: it remains unclear whether architectures guided by the brain score will indeed generalize better to other tasks, as the authors suggest.

2. Uncertain contribution to neuroscience: it remains unclear whether finding the ANN resembling the real visual system most among a collection of models will inform us about the inner working of the brain.


The article would also benefit from the following clarifications:

3. Are the recurrent connections helping performance of Cornet on imagenet and/or on BrainScore?

4. Did you find a correlation between the neural predictivity score and behavioral predictivity score across networks tested? If yes, it would be interesting to mention.

5. When comparing neural predictivity score across models, is a model with more neurons artificially advantaged by the simple fact that there is more likely a linear combination of neurons that map to primate neural activations? Is cross-validation enough to control for this potential bias?

6. Fig1: what are the gray dots?

7. “but it also does not make any assumptions about significant differences in the scores, which would be present in ranking. “
What does this mean?

8. How does Cornet compare to this other recent work: https://arxiv.org/abs/1807.00053 (June 20 2018) ?

Conclusion:
This study presents an interesting attempt at bridging the gap between machine learning and neuroscience. Although the impact that this score will have in both ML and Neuroscience fields remains uncertain, the work is sufficiently novel and interesting to be published at ICLR. I am fairly confident in my evaluation as I work at the intersection of deep learning and neuroscience.

rating: 7: Good paper, accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2019/Conference/Paper824/AnonReviewer1
title: promising work, further tests needed
review: This is an interesting paper in which the authors propose a shallow neural network, the architecture of which was optimized to maximize brain score, and show that it outperforms other shallow networks at imagenet classification. The goal of this paper — allowing brain data to improve our neural nets — is great. The paper is well written (except for a comment on clarity right below), and presents an interesting take on solving this problem. 

It is a little unclear how the authors made CORnet optimize brain score: “However, note that CORnet-S was developed using Brain-Score as a guiding benchmark and although it was never directly used in model search or optimization, testing CORnet-S on Brain-Score is not a completely independent test.” Making these steps clearer is crucial for evaluating better what the model means. In the discussion “We have tested hundreds of architectures before finding CORnet-S circuitry and thus it is possible that the proposed circuits could have a strong relation to biological implementations.” implies that the authors trained models with different architectures until the brain score was maximized after training. A hundred(s) times of training on 2760 + 2400 datapoints are probably plenty to overfit the brainscore datasets. The brain score is probably compromised after this, and it would be hard to make claims about the results on the brain modeling side. The author acknowledge this limitation, but perhaps a better thing is to add an additional dataset, perhaps one from different animal recordings, or from human fMRI? 

Arguably, the goal of the paper is to obtain a model that overpowers other simpler models, and not necessarily to make claims about the brain. The interesting part of the paper is that the shallow model does work better than other shallow models. The authors mention that brain score helps CORnet be better at generalizing to other datasets. Including these results would definitely strengthen the claim since both brain score and imagenet have been trained on hundreds of times so far. 

Another way to show that brain score helps is to show it generalizes above or differently from optimizing other models. What would have happened if the authors stuck to a simple, shallow model and instead of optimizing brain score optimized performance (hundreds of times) on some selected image dataset (this selection dataset is separate from imagenet, but the actual training is done on imagenet) and then tested performance on imagenet? Is the effect due to the brain or to the independent testing on another dataset?

rating: 7: Good paper, accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct