Paper:
Task-Driven Convolutional Recurrent Models of the Visual System
Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.

Review:
This paper introduces and evaluates neuroscience-inspired refinements of classic deep feedforward convolutional neural networks for object classification. Specifically, the authors experiment with new types of processing units which implement mechanisms of recurrence and top-down feedback within the network -- processes thought to be important functional mechanisms in biological vision. The paper is well-conceived and well-argued, and the model development and evaluation is extremely comprehensive and technically accomplished. Importantly, the authors demonstrate how their recurrent models achieve comparable performance to much deeper feedforward networks, while at the same time providing a better fit to neurophysiological data. Moreover, this paper is also an excellent demonstration of how processing principles inspired by neurocognitive theory can be investigated using machine learning and neural network modelling, leading to sophisticated, quantitative evaluation of hypotheses about the functional organisation of the brain (following, e.g., Cox, Seidenberg, & Rogers, 2015; Kriegeskorte, 2015).  Detailed comments  A clear strength of this paper is its comprehensive modelling rigor. For example, I particularly liked the use of carefully matched control networks (lines 119-128). Another example is the attempt to control for the trainability of different architectures through comprehensive hyperparameter tuning (lines 159-171). It is also great that the code for the models is being publically released.  Reciprocal Gated Cells (and their parameterised generalization), introduced in this paper, would seem to have the potential to make an important fundamental contribution to computer vision engineering (in addition to yielding more neurocognitively plausible models). I think the authors could expand a little on the design choices leading to the Reciprocal Gated Cell. For example, couldn’t the standard LSTM cell be adapted to include bypassing? Is there an intuitive reason why the mutual, symmetric, gating by the h and c layers of each others’ inputs can be regarded, a priori, as a desirable property?  My main critical comment is with the ethological relevance of an object labelling task (this is an issue for other similar work as well, not just this paper). As the authors note, primate vision has the goal of transforming the visual input into representations that “readily guide behaviour”. Is object labelling really the most suitable machine learning task for approximating successful behaviour? Why should accurate labelling of objects be the objective function of a model of biological visual systems? This seems particularly problematic for non-human primates, who do not use labels for objects at all, but nevertheless activate adaptation-optimised, behaviour-relevant high-level representations of objects. When a monkey sees either a lion or a leopard, presumably they activate a high-level representation we might gloss as “it’s dangerous; it’s fast; I need to get away right now”. This would be very different to the representation activated for, say, a banana. However, a labelling task will lead the network to learn representations that are best for visually discriminating between all objects (e.g. LION, LEOPARD, BANANA), not representations that reflect meaningful, behaviour-relevant similarities and differences between objects (representations in which LION and LEOPARD are presumably more similar to each other than to BANANA). It seems to me that training neural networks to decode categorical similarity or property knowledge rather than object labels (Devereux, Clarke, & Tyler, 2018; Jozwik, Kriegeskorte, Storrs, & Mur, 2017; Rabovsky & McRae, 2014; Rogers et al., 2004) may reflect more suitable objectives for networks intended to model meaningful object processing. Classic connectionist literature on concept learning (e.g. McClelland and others) may also be relevant here. However, I consider this criticism to be relatively minor, given the other significant contributions of the paper.  Given the stated motivation to model biological visual systems, I would have liked to have seen the models evaluated against a wider range of neuroscience data, including fMRI (perhaps using Representational Similarity Analysis (Kriegeskorte, Goebel, & Bandettini, 2006), following, for example, the analyses of (Khaligh-Razavi & Kriegeskorte, 2014)). Comparison to EEG, ECoG, or MEG data would also have been interesting (e.g. Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016), given the modelling focus on plausible dynamics (lines 101-105).   Is it particularly surprising that vanilla RNNs and LSTMs do not increase performance on object labelling, since these architectures are designed for sequence modelling (which is not relevant to the object labelling task)? I would like to see a slightly more detailed discussion and rationale for why the authors considered these worth testing in the first place (in terms of how they might relate to known functional properties of the primate visual system). Perhaps some of the ideas in lines 143-158 belong earlier, in the introduction and methods sections.  Final comments  From a computational cognitive neuroscience perspective, it is very exciting that the recurrent models presented in this paper do as well as neuro-cognitively implausible ultra-deep feedforward models. This paper is a rarity in that it simultaneously makes fundamental contributions to both computer vision engineering and high-level computational neuroscience, and I therefore heartily recommend its acceptance.   References   Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016). Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, 6, 27755. https://doi.org/10.1038/srep27755  Cox, C. R., Seidenberg, M. S., & Rogers, T. T. (2015). Connecting functional brain imaging and Parallel Distributed Processing. Language, Cognition and Neuroscience, 30(4), 380–394. https://doi.org/10.1080/23273798.2014.994010  Devereux, B. J., Clarke, A., & Tyler, L. K. (2018). Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway. Scientific Reports, 8(1), 10636. https://doi.org/10.1038/s41598-018-28865-1  Jozwik, K. M., Kriegeskorte, N., Storrs, K. R., & Mur, M. (2017). Deep Convolutional Neural Networks Outperform Feature-Based But Not Categorical Models in Explaining Object Similarity Judgments. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01726  Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014). Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLoS Comput Biol, 10(11), e1003915. https://doi.org/10.1371/journal.pcbi.1003915  Kriegeskorte, N. (2015). Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing. Annual Review of Vision Science, 1(1), 417–446. https://doi.org/10.1146/annurev-vision-082114-035447  Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, 103(10), 3863–3868. https://doi.org/10.1073/pnas.0600244103  Rabovsky, M., & McRae, K. (2014). Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning. Cognition, 132(1), 68–89. https://doi.org/10.1016/j.cognition.2014.03.010  Rogers, T. T., Lambon Ralph, M. A., Garrard, P., Bozeat, S., McClelland, J. L., Hodges, J. R., & Patterson, K. (2004). Structure and Deterioration of Semantic Memory: A Neuropsychological and Computational Investigation. Psychological Review, 111(1), 205–235. https://doi.org/10.1037/0033-295X.111.1.205