Paper:
Automatic Neuron Detection in Calcium Imaging Data Using Convolutional Networks
Calcium imaging is an important technique for monitoring the activity of thousands of neurons simultaneously. As calcium imaging datasets grow in size, automated detection of individual neurons is becoming important. Here we apply a supervised learning approach to this problem and show that convolutional networks can achieve near-human accuracy and superhuman speed. Accuracy is superior to the popular PCA/ICA method based on precision and recall relative to ground truth annotation by a human expert. These results suggest that convolutional networks are an efficient and flexible tool for the analysis of large-scale calcium imaging data.
in shape, size, spacing, and resting-level fluorescence. Additionally, different neuroscience studies may require differing ROI selection criteria. Some may require only cell bodies [5,11], while others involve dendrites [6]. Some may require only active cells, while others necessitate both active and inactive cells [10]. Some neuroscientists may wish to reject slightly out-of-focus neurons. For all of these reasons, a neuroscientist may spend hours or days tuning the parameters of nominally automated methods, or may never succeed in finding a set of parameters that produces satisfactory results.
As a way of dealing with these difficulties, we focus here on a supervised learning approach to automated ROI detection. An automated ROI detector could be used to replace manual ROI detection by a human expert in the traditional workflow, or could be used to make the basis learning algorithms more reliable by providing good initial conditions for basis images. However, the usability of an automated algorithm strongly depends on it attaining high accuracy. A supervised learning method can adapt to different ROI selection criteria and generalize them to new datasets. Supervised learning has become the dominant approach for attaining high accuracy in many computer vision problems [12].
We assemble ground truth datasets consisting of calcium imaging videos along with ROIs drawn by human experts and employ a precision-recall formalism for quantifying accuracy. We train a sliding window convolutional network (ConvNet) to take a calcium video as input and output a 2D image that matches the human-drawn ROIs as well as possible. The ConvNet achieves near-human accuracy and exceeds that of PCA/ICA [6].
The prior work most similar to ours used supervised learning based on boosting with hand-designed features [13]. Other previous attempts to automate ROI detection did not employ supervised machine learning. For example, hand-designed filtering operations [14] and normalized cuts [15] were applied to image pixel correlations.
The major cost of supervised learning is the human effort required to create the training set. As a rough guide, our results suggest that on the order of 10 hours of effort or 1000 annotated cells are sufficient to yield a ConvNet with usable accuracy. This initial time investment, however, is more than repaid by the speed of a ConvNet at classifying new data. Furthermore, the marginal effort required to create a training set is essentially zero for those neuroscientists who already have annotated data. Neuroscientists can also agree to use the same trained ConvNets for uniformity of ROI selection across labs.
From the deep learning perspective, an interesting aspect of our work is that a ConvNet that processes a spatiotemporal (2+1)D image is trained using only spatial (2D) annotations. Full spatiotemporal annotations (spatial locations and times of activation) would have been more laborious to collect. The use of purely spatial annotations is possible because the neurons in our videos are stationary (apart from motion artifacts). This makes our task simpler than other applications of ConvNets to video processing [16].

Review:
This paper presents the first fully supervised approach to cell detection of two-photon imaging dat, based on a fully convolutional framework (ZNN). The results are compelling and seems not far from the performance of human annotators. Results are compared to a common PCA/ICA approach but not to the state of the art in unsupervised cell detection (e.g., CMNF). The output of this scheme is limited to the spatial domain (unlike other approach which in one complex step determine also spike activity). Performance for awake behaving mice primary visual cortex and medial entorhinal cortex.    
NOTE Following the rebuttal and reading of the other reviews I lowered the "Technical Quality" which is now 2 (originally 3).  The authors of Ref1 describe cases where a human annotator could fail to reliably detect and demix cell boundaries. Therefore to claim superiority over such unsupervised methods the current paper needs to compare performance against the performance of such approaches (see Ref1, e.g. using). It is generally required to compare against CMNF in order to claim state of the art accuracy.   It is unclear whether the paper used re-evaluation in order to correct the seemingly erroneous ground truth used for annotation. (see specific comment 8 below).    It would be generally impressive if the authors can replace or extend the large temporal smoothing and downsampling with network’s kernel that achieve the same result in a supervised manner, e.g., using 3D kernels. If the authors already experienced with 3D training and the results ere not convincing this must be included in the paper. I see this point quite critical in the evaluation of the paper as the temporal smoothing is widely not justified (as I explain in my specific comments below).    Specific Comments:   1. 27-28. The authors suggest that CNMF and PCA/ICA are inadequate for some datasets. This point is very weak and incomplete. What are these datasets? And in particular how the current paper is superior over such methods (e.g., CNMF) on these unspecified datasets? 2. 28-29. The authors say that unsupervised methods have not fully replaced the traditional approach. However approaches like CNMF were published in 2016 and other laboratories are likely to share their code in open source pipelines this and next year. Therefore the claim that the traditional approach is still in practice is unjust.  3. 34-39. The text suggests that different experimental setups and scientific goals would require different cell detection outcomes and hence unsupervised approach will depend on parameter tuning. While this may be a true case, it is unclear how much of this burden is actually eliminated by preparation of human ground truth — it seems that all parameter tuning are parametrization of the model, and in the end will require less human effort than ground truth preparation. Moreover, once a laboratory fixed the parameters for several setups the tuning on new datasets (marginal effort) is likely to be greatly reduced. Therefore, I recommend the authors to suppress this paragraph.  4. 39-40. The authors point that current automatic methods require proofreading. However accuracy is what matters — hence also the suggested supervised method as well end up requiring proofreading. Please correct my impression or delete this point.   5. 51. The authors compare to a common method (PCA/ICA) that is not state of the art. First, it should be noted by the authors. Second, it will strengthen the paper if the authors compare to Ref1. 6. 71-77 can be suppressed.  7. 99-101. I find this explanation too heuristic. First at this point the author is unaware of the temporal sampling rate and cannot appreciate the smoothing effect. Second, the usefulness of making the dataset more manageable in size should be marginal where sophisticated methods could perform great (fast and accurate) without data reduction. Also, the procedure used (rectangular filter with max-pooling operation) is nonstandard and require better reasoning. Finally, 132 is read as a drawback of this “Severe reduction.” And then comes 133 which simply goes against the paper’s own method. This should be fixed by introducing better reasoning.  8. 110-115 is confusing. First the V1 dataset is presented. Then a sentence about human annotation time. Then the MEX dataset is presented. Following sentence is about both datasets. And then another seemingly general sentence about human annotation time (I suppose for the MEC dataset since numbers contradict the ones for V1; or could be different estimates of the same human time?).   9. 127 “an an” typo.  10. 132. The choice to heavily down sample in time was the authors’ and hence the explanation in 132 should be revised and made more convincing (the authors can say that other applications were inferior in performance, or come up with a alternative reasoning — perhaps related to the non-isotropic nature of the problem and the different noise distributions in space vs time. 11. 134. Space typo.  12. 136. The large difference of depth between the two datasets should be succinctly explained.  13. 177-179. I strongly disagree with the claim that the reevaluation and the margins reported translate to “nearing human performance.” Instead I suggest the authors repeat this scheme with another human expert and then let the two expert reconcile their disagreements into a mutual call (as done in other fields). It is unfair to let the human be convinced by the algorithm’s output and this is certainly an invalid evaluation metric. Also it is unclear that the reported margin shows near human performance even it this F score was obtained in a valid way. Finally the F1 score of the export w.r.t his own judgment being 0.89 does not represent a standard value for the F score that should be attained.   14. 194-197. Given the above reservations it will contribute if the authors can repeat the “human interrogation” procedure with the PCA/ICA result and see if naive humans are convinced to change their opinion.    References:  Ref1 ([9] in the submitted paper) Simultaneous Denoising, Deconvolution, and Demixing of Calcium Imaging Data (Neuron 2016)