Paper:
Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream
Humans recognize visually-presented objects rapidly and accurately. To understand this ability, we seek to construct models of the ventral stream, the series of cortical areas thought to subserve object recognition. One tool to assess the quality of a model of the ventral stream is the Representational Dissimilarity Matrix (RDM), which uses a set of visual stimuli and measures the distances produced in either the brain (i.e. fMRI voxel responses, neural firing rates) or in models (features). Previous work has shown that all known models of the ventral stream fail to capture the RDM pattern observed in either IT cortex, the highest ventral area, or in the human ventral stream. In this work, we construct models of the ventral stream using a novel optimization procedure for category-level object recognition problems, and produce RDMs resembling both macaque IT and human ventral stream. The model, while novel in the optimization procedure, further develops a long-standing functional hypothesis that the ventral visual stream is a hierarchically arranged series of processing stages optimized for visual object recognition.

Review:
Summary: This paper presents an algorithm for 
      category-level object recognition that searches a large space of 
      heterogeneous hierarchical neural network architectures using a boosting 
      mechanism. The resulting network found via this algorithm is used to 
      predict multiunit neural activity in monkey IT, as well as the similarity 
      structure in monkey neural IT and human fMRI IT representations. This is 
      the first reasonably successful prediction of neural activity in IT in 
      response to complex naturalistic images of objects. Pros: 
      Predicts neural activity in IT in response to complex naturalistic 
      images of objects for the first time and with reasonable success. This is 
      a significant achievement, and will be exciting to many at NIPS. 
      Introduces a new algorithm for category-level object recognition, 
      which optimizes a very complex, heterogenous neural network architecture 
      using a boosting methodology. This will be of interest also to 
      non-neuroscientists interested in deep learning and object classification. 
      The comparisons to neural data are done to a high standard, with 
      results replicated on three independently collected datasets (two monkey 
      multiunit, one human fMRI). The paper is clearly written in 
      general. Cons: The Krizhevsky et al. object recognition 
      network should be included in the evaluation, since this model was shown 
      to be promising in closely related prior work (e.g., [2]). This is a very 
      natural baseline model to compare to. The Krizhevsky network was shown to 
      achieve classification performance superior to that of Monkey IT, on the 
      exact Neural Representation Benchmark dataset used in this paper. 
      The paper describes running the proposed HMO algorithm to obtain a 
      network N_{HMO} used in the comparisons to experimental data. It would be 
      very interesting to learn more about the properties of this optimal 
      network. What is its topology? Is the permitted heterogeneity in the space 
      of networks used, or does this procedure end up using networks of some 
      standard depth? Since intuition into what makes a good architecture is 
      generally hard to come by, it would be useful to see this example. It 
      would also be useful to describe other properties of the specific HMO 
      implementation used in this paper, namely, how many networks were 
      evaluated in the optimization of single stack networks, etc. As it 
      stands it is not possible to determine whether the proposed algorithm is 
      necessary to model IT well, or whether other state-of-the-art object 
      recognition methods would perform comparably. This has bearing on the view 
      advanced at the end of the discussion that the paper's results argue for 
      one relatively fixed, unmodifiable "core recognition" network which feeds 
      into more plastic "downstream specialists" responsible for learning 
      different tasks. The Krizhevsky network, by contrast, would be plastic 
      throughout all levels, with no obvious "two-step" arrangement. If it 
      performs comparably in predicting IT responses, it would weaken the 
      paper's argument in support of their dual system view. On the other hand, 
      if the Krizhevsky network underperformed the proposed method, this would 
      be very informative. As it stands, arguing in support of the two-step 
      arrangement seems premature given the other tenable but unexamined models 
      in the literature. Minor: The description of the HMO 
      algorithm is hard to follow. The score equation in line 143, Sum N(F(s)), 
      makes sense to me as E(N(s)). Also it appears that F is never defined. The 
      reweighted score function of line 156 again makes use of the undefined F, 
      F_1, etc, which presumably again is N. 229: "Matricx" -> 
      "Matrix" Fig. 3A: include Y axis label