Paper:
Stimulus domain transfer in recurrent models for large scale cortical population prediction on video
To better understand the representations in visual cortex, we need to generate better predictions of neural activity in awake animals presented with their ecological input: natural video. Despite recent advances in models for static images, models for predicting responses to natural video are scarce and standard linear-nonlinear models perform poorly. We developed a new deep recurrent network architecture that predicts inferred spiking activity of thousands of mouse V1 neurons simultaneously recorded with two-photon microscopy, while accounting for confounding factors such as the animal's gaze position and brain state changes related to running state and pupil dilation. Powerful system identification models provide an opportunity to gain insight into cortical functions through in silico experiments that can subsequently be tested in the brain. However, in many cases this approach requires that the model is able to generalize to stimulus statistics that it was not trained on, such as band-limited noise and other parameterized stimuli. We investigated these domain transfer properties in our model and find that our model trained on natural images is able to correctly predict the orientation tuning of neurons in responses to artificial noise stimuli. Finally, we show that we can fully generalize from movies to noise and maintain high predictive performance on both stimulus domains by fine-tuning only the final layer's weights on a network otherwise trained on natural movies. The converse, however, is not true.

Review:
##### I have read over the author's rebuttal, and I am satisfied by their clarification and response. ##### In this submission, the authors use modern deep learning techniques to develop a predictive model of mouse V1 cortex. Most notably, this model seeks to fill a gap in the literature for simultaneously predicting large populations of neurons responding to both natural video and noise stimuli, whereas previous models had mainly focused on static image stimuli in small cortical populations.   The model developed by the authors receives these video stimuli as inputs, and consists of well-established computer vision architectures — here a feed-forward convolutional network feeding into a convolutional GRU — to extract stimulus feature representations common to all neurons. These are then read-off and modulated by a spatial-transformer network to ensure receptive-field stability of the regressed neural outputs. A novel structural contribution has been added to account for non-stimulus driven experimental noise which might be present in the data. Specifically, the readouts are additionally modified by “shift” and “modulator” networks which factor in these sources of noise such as pupil movement (correction to this is needed for proper receptive-field mapping), pupil diameter, and measured running speed of the animal (which is known to be correlated with V1 activity in rodents).  They show that this model outperforms a non-recurrent linear-nonlinear control model (fig. 2a) across both datasets for all three mice, and assess the relative contributions of the novel shift and modulator component networks (fig. 2c). In “domain transfer” experiments, they show both qualitative and quantitative evidence for models generalizing outside their training set to produce receptive fields and tuning curves similar to mouse V1 activity, even though the mice have been shown different noise stimuli. Lastly, the authors study the quantitative amount of transfer between stimulus domains and find that models trained on movies serve as reasonable predictors of white-noise induced activity, and find that the converse is not true (fig. 4b). However, a network which combined domain knowledge from both stimuli sets provides a sufficient basis space for explaining activity presented in either domain (fig. 4c).  Overall, I am impressed by the ambition of this work. I believe that the field could benefit from more of this style of large-scale modeling, and further consideration of generalization to novel stimuli as a complex yet intriguing phenomenon. The application to large-scale populations responding to complex stimuli, instead of only a small subset of V1 responding to simpler stimuli is also noteworthy. I found the introduction of the shifter and modulator networks to be a novel idea, encoding well-thought-out structural priors into the model.  That being said, although I have no complaints concerning the high-level scope of analysis presented in this submission, I do have some concerns as to the strength of results for several reasons:  (1). I am willing to believe the qualitative comparison between the “full” model to a “standard” linear-nonlinear system in fig 2a. Based on the lack of recurrence alone, the result seems intuitive. However, I am concerned with the quantitative estimate between these two models due to not controlling for numbers of learnable parameters. From what I understand, the authors balance the number of parameters in the ConvNet betweeen models — three 12-feature layers in the full model and one 36-feature layer in the LN mode — and readout structures. However, in the absence of the GRU in the LN model, there is an imbalance in these numbers. Generally speaking, you can interpret this in two ways: either (i). the extra units and nonlinearities can artificially inflate the score of the full model if no hyperparameter optimization has been performed or (ii) the authors had performed a hyper parameter sweep (number of units) across both models and settled on the best performing settings. Given that no mention towards the latter has been discussed in the submission, I am inclined to believe that the former is true.  (2). I am concerned with a statement that was made inside the “Network Implementation” subsection which stated that all readouts biases were initialized to the mean firing rate of that neuron. It seems to me that this choice is also reasonable, however it sounds like that is a strong prior for what these values should be. Could the authors discuss the impact of this initialization, and how it relates to why the full model might only be performing ~50-70% of the oracle estimator?  (3). Similar to the above in (1), even if I am intrigued by the application of the shift and modulator networks, I am concerned that for parameter balancing reasons and the lack of a hyperparameter sweep, the results in 2c might be artificially inflated. Additionally, I’m concerned that the novelty of such an approach might be detracting from what might be a more standard way of incorporating these stimulus-independent noise sources — namely, by having them as generic features that are concatenated as inputs to the readout MLP, rather than these specific priors. I would be just as impressed by seeing a larger/deeper readout network with a more generic structure perform at the level (or better) than something with a “clever” implementation.   (4). Almost unsurprisingly, the authors find that models trained on movies are well-correlated with noise-induced activity, although the converse is not true, and that a model trained on both can in fact predict both. I would have liked a similar comparison between natural movies and the rendered videos as well, because it is unclear whether the statistics between even these two domains are imminently transferable. What I did find interesting here is that although the movie-optimized models are well correlated with noise responses, they show high variance in predicting the low-levels details such as orientation and direction tuning. The paper ends on a bit of a cliff-hanger in the sense that they do not propose a better method of a generalized model than what amounts to just training on all stimulus contingencies.   In summary, I am inclined to like this work. The results — although not necessarily as strong as they could be — feel intuitive. If text length were not a factor, I personally feel like I would have benefitted from an extended discussion/interpretation section of results, and more detailed analysis of those presented in the submission (specifically as far as the domain transfer section is concerned). The approach and aims of this work alone are a worthwhile contribution to the neuroscience community as a whole.