From: ICLR.cc/2019/Conference/Paper276/Authors
title: Response to the "COMMENTS RELATED TO REVISION" part
comment: First, thank you for considering the additional analyses as positive. Here we would like to comment on your concerns.

- it's a stretch to generalize this to conclusions concerning functional importance in neuroscience.

Response: we stated the 2nd argument by regarding DNN as a model of the visual cortex in terms of the hierarchical feature representations (as we said in the introduction part). Although DNN and the visual cortex are of course not the same, the feasibility of this modelling is supported by the following points:
1. Both DNN and the visual cortex can recognize objects (of course, the performance is different, but the essential function is basically similar)
2. DNN and the visual cortex have similar hierarchical representations at µm~mm resolution (Yamins et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Cadieu et al., 2014; Guclu & van Gerven, 2015; Horikawa & Kamitani, 2017).


- it would be surprising if there were vestigial features in the network that are learned but play no important role - especially among early layers. 

Response: we agree with you that those features that are learned but are not important are interesting per se. However, our main interest here is on how DNN can recognize objects (e.g., what feature representations are necessary for the high performance?). To answer this question scientifically, we should investigate the functionally important features, rather than the unimportant features. Again, this study is the first one that quantitatively showed that orientation selectivity is needed in DNN.

From: ICLR.cc/2019/Conference/Paper276/Area_Chair1
metareview: The authors conduct experiments to study orientation selectivity in neural networks. 

The reviewers generally agreed that the paper was clearly written and easy to follow. Further, the experimental analysis demonstrates that contrary to what was claimed in some previous work, the learned orientation selectivity can be useful for generalization. 

However, the reviewers also raised a number of concerns: 1) that the conclusions are drawn on the basis of a couple of neural network architectures; the authors attempted to add results using a Resnet50 model, but this analysis was ultimately removed when the authors discovered a bug; 2) in the context of the contributions in neuroscience it was not clear that the limited results on the two artificial networks are sufficient to help draw such conclusions, and that 3) since the network is trained to recognize objects, it would seem natural that the model would learn neurons that are sensitive to orientation and that it is not clear how the author’s observations might lead to better trained models. 
While the reviewers were not completely unanimous in their scores, the AC agrees with a majority of the reviewers that the work while interesting could be strengthened by additional experiments on other architectures. 

confidence: 3: The area chair is somewhat confident
recommendation: Reject
title: Clearly written paper, but unclear impact

From: ICLR.cc/2019/Conference/Paper276/AnonReviewer1
title: We're getting there
comment: Thanks for running the ResNet control – that's reassuring. 

With respect to claims, I think you could phrase abstract, contributions (end of intro) and discussion a bit more carefully. For instance, instead of claiming that "orientation selectivity causally contributes to the performance of object recognition," a more accurate phrasing would be that "orientation selectivity causally contributes to object recognition in a number of architectures we explored, suggesting that it may be a crucial component of any object recognition system." The first part is a solid finding you have; the second part is your hypothesis/speculation.

I'm still not 100% sold on the paper, but I'm willing to up my score given your efforts, in particular if you tone down the claims a bit.

From: ICLR.cc/2019/Conference/Paper276/Authors
title: Response to AnonReviewer3
comment: We appreciate the overall positive comments, for considering this paper to be “tackling a very interesting problem that seems to have not received yet enough attention.” Below, we address the concerns of AnonReviewer3.

- The only concern I would have is about the causal claims in Section 3.4…

Response: As suggested, we have toned down the claim in the last sentence in Section 3.4 (“These results prove that…” to “These results suggest that…”).

We agree that there are several ways to claim the causality between a cause and the outcome. However, we used ablation in this paper for the following two reasons:
1) Ablation is the simplest manipulation to investigate the causality (as suggested).
2) Ablation (or inactivation) experiments have long been the most popular way to investigate the causality, especially in biology [1] and neuroscience (e.g. [2-4]), where researchers tackle complex black-box systems (gene networks, brains, etc.) like the DNNs. 


[1] For example, if we investigate whether a cause of a disease is a specific gene, the most convincing way to prove the causality between the gene and the disease is to knockout the gene from animals and see whether the animals display the disease phenotype.
[2] Leor N. Katz, Jacob L. Yates, Jonathan W. Pillow, and Alexander C. Huk. Dissociated functional significance of decision-related activity in the primate dorsal stream. Nature, 535(7611):285–288, 2016.
[3] Srivatsun Sadagopan, Wilbert Zarco, and Winrich A. Freiwald. A causal relationship between face-patch activity and face-detection behavior. eLife, 6:1–14, 2017.
[4] Michael M. Yartsev, Timothy D. Hanks, Alice Misun Yoon, and Carlos D. Brody. Causal contribution and dynamical encoding in the striatum during evidence accumulation. eLife, 7:1–24, 2018.

From: ICLR.cc/2019/Conference/Paper276/Authors
title: Response to AnonReviewer2
comment: We thank the reviewer for the overall positive comments, and describing this paper as “well written and interesting” and “This style of analysis is sorely lacking and fits the theme of learning representations.” We also appreciate the very important suggestions, which we address below.

- The only concern I have (and it is somewhat significant), is the notion of "generalization". This is a rather loaded term, and it is not clear on cursory inspection where this generalization comes from. Is it a function of invariance among higher layers to scene geometry? What are the fundamental underpinnings of these observations outside of correlation to orientation selectivity.
- It seems almost a tautology that removing orientation selectivity would impair performance. I would be more satisfied, and render a higher rating if I felt there was a "smoking gun" with respect to evidence. The conclusion is convincing, but the reasoning comes across as somewhat vague. With that said, it is also understood that this is a non-trivial matter to address and perhaps this paper is an important first step.

Response: First, we would like to reconfirm our new findings (these are inserted in the first paragraph of the discussion section):
1) Orientation selectivity is not just a superficial byproduct of object recognition, but is causally indispensable for object recognition.
2) Orientation selectivity is not necessary for the memorization (see also Fig. 4c), but is important for the generalization of the image-class relationships, at least by introducing shift-invariance of the fully connected layer.

We totally agree with the reviewer in that “orientation selectivity in the lower layers” and “the overall generalization” are distant phenomena. We thus performed an additional analysis to connect these two concepts more closely in the new section 3.5, which is based on the reviewer’s question on the invariance in the higher layers. In summary, we compared the shift-invariance of the higher layers between an ordinary network and a network where orientation-selective units of Conv 2 or Conv 3 were ablated, finding that “units in the fully connected layer of the ablated network had significantly higher (shift-)variances than those of the vanilla network” (Fig. 6). Thus, we concluded that orientation-selective units in the lower layers introduce a part of shift-invariance of the fully connected layers, thereby contributing to the generalization of object recognition.

From: ICLR.cc/2019/Conference/Paper276/Authors
title: Response to AnonReviewer1
comment: We highly appreciate the thorough comments. Here we provide the responses to all the comments.

- On the other hand, I'm not really sure what we learn from the paper.

We would like to reconfirm our new findings (these are inserted in the first paragraph of the discussion section).
1) Orientation selectivity is not just a superficial byproduct of object recognition, but is causally indispensable for object recognition.
2) Orientation selectivity is not necessary for memorization (see also Fig. 4c), but is important for the generalization of the image-class relationships, at least by introducing shift-invariance of the fully connected layer.

- The result seems trivial. How should a network be able to recognize objects without detecting edges of certain orientation (which implies orientation selectivity)?
 - Related to the previous point, pretty much every single (supervised or unsupervised) learning objective investigated so far has produced orientation selectivity, so it seems pretty well established that orientation selectivity is somehow useful. The interesting question is what needs to be done on top of it in order to get a representation useful for object recognition, but in this respect the paper does not contribute anything.

Response: Indeed, some papers have already revealed the existence of Gabor feature representations in the early layers of DNNs [1-2]. However, ours is the first study that quantitatively compares the orientation selectivity among different layers and shows that orientation selectivity is a correlate and a cause of the generalization performance, which, we believe, elicited positive comments like “This paper presents interesting analysis” (reviewer #2) and “The authors tackle a very interesting problem that seems to have not received yet enough attention” (reviewer #3) from other reviewers.

In addition, we are confident that the framework we used (investigation of the correlation between the unit selectivity and generalization and the causality by ablation experiments) will generally be useful for analyzing the higher feature representations in future studies. 

- The results are mostly on CIFAR-10 and only one network (VGG-16) trained on ImageNet is considered. Given the generality of the claims ("orientation selectivity [plays] a causally important role in object recognition" – abstract), the authors would have to show that their results also hold for other high-performing networks on ImageNet, and not just VGG-16 (sort of, see next point). Otherwise, an appropriate conclusion would be that orientation selectivity plays a causal role in the functioning of VGG-16 and some networks trained on CIFAR-10.

Response: In this study we analyzed networks that were trained with different initializations or different training datasets, which, on the other hand, reviewer #3 appreciated as “reproducibility.” Considering other empirical papers in ICLR (e.g. [3-4]), comparing three networks with different initializations or different training datasets to check the generality of the findings is sufficient.

- The analysis meant to establish causality (section 3.4) produces pretty mixed results on VGG-16 (Fig. A6b), where ablating the top 50% orientation-selective units in some layers has a *smaller* effect than ablating the rest. How do the authors explain this result?

Response: In section 3.4, we showed that ablating orientation-selective units in the several early layers of VGG-16 (block1_conv2, block1_pool, and block2_conv1) significantly dropped the performance, which is consistent with the CIFAR-10 cases. In higher layers of VGG-16 (e.g., block2_conv2) where ablating units with the top 50% gOSI had a smaller impact than ablating units with the bottom 50% gOSI, the represented features critical for the generalization should not be orientations but something else. Thus, when we ablate units with the bottom 50% gOSI in these layers, we might disrupt those features, causing performance impairment.


[1] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical report, University of Montreal, pp. 1–13, 2009.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, pp. 1097–1105, 2012.
[3] Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations (ICLR), 2018.
[4] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017.

From: ICLR.cc/2019/Conference/Paper276/Authors
title: General response to the reviewers’ comments
comment: We would like to thank the three reviewers for the thorough and constructive comments despite their busy schedules. In response to the comments, we have revised the paper by providing additional data and by polishing the writing, which, we believe, has significantly improved the quality of the paper.

Key modifications include:
1) In the new section 3.5, we performed an additional analysis to answer how the orientation-selective units in the lower layers contribute to the overall generalization.
2) We added a new paragraph in the first part of the discussion section to clarify our new findings.
3) We moved the figure showing “All grating images presented to the 7-layer CNNs trained on CIFAR-10” from Fig. 1 to the appendix Fig. A1 in order to conform to the page limit. Therefore, all figure numbers have changed.

Finally, we would like to note that our paper has also substantial contributions 1) to the debate on the harmfulness/importance of selectively activated units in DNNs [1-3] by presenting concrete examples where selectivity is important for generalization, and 2) to the neuroscience community, where, although orientation selectivity has been extensively analyzed for these 60 years since [4], its functional importance in a natural environment has remained unanswered.


[1] Kairen Liu, Rana Ali Amjad, and Bernhard C. Geiger. Understanding Individual Neuron Importance Using Information Theory. arXiv:1804.06679, 2018.
[2] Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations (ICLR), 2018.
[3] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv:1806.02891, 2018.
[4] David H. Hubel and Torsten N. Wiesel. Receptive fields of single neurones in the cat’s striate cortex. Journal of Physiology, 148(12):574–591, 1959.

From: ICLR.cc/2019/Conference/Paper276/AnonReviewer3
title: Interesting and well-written paper
review: The paper presents a study on orientation selectivity in DNNs for image classification, arguing that this type of selectivity in the lower layers is crucial for generalization. This hypothesis is tested through an ablation study, which the authors interpret as a suggestion of the existence of a causal relation.

The authors tackle a very interesting problem that seems to have not received yet enough attention. The paper is quite well-written and clear, making it understandable also to non-experts. The only concern I would have is about the causal claims in Section 3.4. I’m not completely sure the ablation experiments are the correct way to “prove” causality, as opposed to somehow trying to intervene on the orientation selectivity directly. On the other hand, this seems complicated to prove and the approach proposed in the paper seems a pragmatic solution. I would possibly hedge slightly the causal claims.

From an outsider point of view, I think the paper provides an interesting contribution to the discussion on orientation selectivity. I particularly appreciated its clarity and reproducibility.

rating: 7: Good paper, accept
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper

From: ICLR.cc/2019/Conference/Paper276/AnonReviewer2
title: Review of Causal importance of orientation selectivity for generalization in image recognition
review: COMMENTS RELATED TO REVISION:
The new analysis that has been added takes a step towards getting at the relationship to invariance. This is a positive. In general comments, the authors state as important contributions: 

"1) to the debate on the harmfulness/importance of selectively activated units in DNNs [1-3] by presenting concrete examples where selectivity is important for generalization, and 2) to the neuroscience community, where, although orientation selectivity has been extensively analyzed for these 60 years since [4], its functional importance in a natural environment has remained unanswered."

On point 2, while this provides an example of an artificial network where orientation plays an important role, it's a stretch to generalize this to conclusions concerning functional importance in neuroscience.

One point 1, I agree that this paper takes steps in the right direction, but ultimately the overall conclusions still feel as though they are a natural and implied consequence of limiting orientation selectivity. It is noted that  orientation selectivity is "not just a superficial byproduct of object recognition, but is causally indispensable for object recognition". The idea that selectivity for oriented edges is indispensible for object recognition again is a conclusion that feels as though it would be shocking if this were not true. The notion of it being a superficial byproduct of object recognition presupposes that the purpose of the system is to recognize objects. Again, it would be surprising if there were vestigial features in the network that are learned, but play no important role - especially among early layers. 

I think the paper might be strengthened by re-working this second point to more strongly establish the causality the authors claim. 

ORIGINAL COMMENTS:
This paper presents interesting analysis and an ablation study on orientation selectivity in neural networks. This is analyzed with respect to generalization performance in decisions made. Overall, the paper is well written and interesting. However, I have a number of comments / concerns as described below:

Positives:
- The paper presents an in depth analysis of the role of orientation selectivity in neural hierarchies. This style of analysis is sorely lacking and fits the theme of learning representations
- The paper itself is well written and quite polished
- The authors have taken great care to rule out any possible confounds through experiments that are not identical to the main claims or objective (specifically the study of section 3.3)

To address:
- The only concern I have (and it is somewhat significant), is the notion of "generalization". This is a rather loaded term, and it is not clear on cursory inspection where this generalization comes from. Is it a function of invariance among higher layers to scene geometry? What are the fundamental underpinnings of these observations outside of correlation to orientation selectivity.
- It seems almost a tautology that removing orientation selectivity would impair performance. I would be more satisfied, and render a higher rating if I felt there was a "smoking gun" with respect to evidence. The conclusion is convincing, but the reasoning comes across as somewhat vague. With that said, it is also understood that this is a non-trivial matter to address and perhaps this paper is an important first step.
rating: 5: Marginally below acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2019/Conference/Paper276/AnonReviewer1
title: Straightforward paper, but unclear what we learn from it
review: *Update after discussion period*
I remain unconvinced. The authors failed to address my clearly articulated request for a more thorough analysis of additional networks trained on ImageNet (e.g. ResNet), which I don't think is asking for too much given a discussion period of three weeks.


Summary:
The authors find that (1) DNNs exhibit orientation selectivity in many of their hidden layers' units, (2) in the intermediate layers this selectivity emerges during training, concurrently with the network's ability to generalize, and (3) ablating orientation-selective units in the early layers impairs a network's generalization performance. 

Strengths:
+ Very straightforward and easy to follow 
+ Technically sound

Weaknesses:
- Feels trivial
- The claims seem to be too general

Conclusion:
I'm torn on the paper. On the one hand, it reports some potentially interesting observations (e.g. trajectory of emergence of orientation selectivity over training). On the other hand, I'm not really sure what we learn from the paper.


Specific comments:

- The result seems trivial. How should a network be able to recognize objects without detecting edges of certain orientation (which implies orientation selectivity)?

- Related to the previous point, pretty much every single (supervised or unsupervised) learning objective investigated so far has produced orientation selectivity, so it seems pretty well established that orientation selectivity is somehow useful. The interesting question is what needs to be done on top of it in order to get a representation useful for object recognition, but in this respect the paper does not contribute anything.

- The results are mostly on CIFAR-10 and only one network (VGG-16) trained on ImageNet is considered. Given the generality of the claims ("orientation selectivity [plays] a causally important role in object recognition" – abstract), the authors would have to show that their results also hold for other high-performing networks on ImageNet, and not just VGG-16 (sort of, see next point). Otherwise, an appropriate conclusion would be that orientation selectivity plays a causal role in the functioning of VGG-16 and some networks trained on CIFAR-10.

- The analysis meant to establish causality (section 3.4) produces pretty mixed results on VGG-16 (Fig. A6b), where ablating the top 50% orientation-selective units in some layers has a *smaller* effect than ablating the rest. How do the authors explain this result?

rating: 4: Ok but not good enough - rejection
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct