Paper:
Combinatorial Energy Learning for Image Segmentation
We introduce a new machine learning approach for image segmentation that uses a neural network to model the conditional energy of a segmentation given an image. Our approach, combinatorial energy learning for image segmentation (CELIS) places a particular emphasis on modeling the inherent combinatorial nature of dense image segmentation problems. We propose efficient algorithms for learning deep neural networks to model the energy function, and for local optimization of this energy in the space of supervoxel agglomerations. We extensively evaluate our method on a publicly available 3-D microscopy dataset with 25 billion voxels of ground truth data. On an 11 billion voxel test set, we find that our method improves volumetric reconstruction accuracy by more than 20% as compared to two state-of-the-art baseline methods: graph-based segmentation of the output of a 3-D convolutional neural network trained to predict boundaries, as well as a random forest classifier trained to agglomerate supervoxels that were generated by a 3-D convolutional neural network.

Review:
This paper proposed a novel method to handle the large-scale segmentation problem in mapping neuroanatomy. The proposed method uses deep neural network to evaluate local energy given a segmentation configuration, as well as an energy minimization approach to greedily guide the (super)voxel agglomeration. To handle the inherent large scale of the targeted problem, the authors also consider the issue of pruning from several aspects.
The paper is well-written, with lots of details provided. The studied problem is definitely of great impact and would significantly benefit the community of neuroscience. The reviewer noticed the large size of the problem and appreciates the considerable amount of works achieved, as described by the paper.  On the other hand, I also want to point out some associated weakness:  1. To my understanding, two aspects which are the keys to the segmentation performance are: (1) The local DNN evaluation of shape descriptors in terms of energy, and (2) The back-end guidance of (super)voxel agglomeration. Although experiment showed gains of the proposed method over GALA, it is yet not clear enough which part is the major contributor of such gain. The original paper of GALA used methods different from this paper (3D-CNN) to generate edge probability maps. Is the edge map extraction framework in this paper + GALA a fair enough baseline? It would be great if such edge map can also be visualized.  2. The proposed method to some extent is not that novel. The idea of generating or evaluating segmentation masks with DNN has been well studied by the vision community in many general image segmentation tasks. In addition, the idea of greedily guiding the agglomeration of (super)voxels by evaluating energies pretty much resembles many bottom-up graph-theoretic merging in early segmentation methods. The authors, however, failed to mention and compare with many related works from the general vision community. Although the problem general image segmentation somewhat differs from the task of this paper, showing certain results on general image segmentation datasets (like the GALA paper) and comparing with state-of-the-art general segmentation methods may give a better view of the performance of the proposed method.  3. Certain parts did not provide enough explanation, or are flooded by too much details and fail to give the big picture clearly enough. For example in Appendix B Definition 2, when explaining the relationship between shape descriptor and connectivity region, some graphical illustrations would have helped the readers understanding the ideas in the paper much easier and better.  ----------------------------------------------------------------------------- Additional Comments after Discussion  Upon carefully reading the rebuttal and further reviewing some of the related literature, I decide to down-grade scores on novelty and impact. Here are some of the reasons:  1. I understand this paper targets a problem which somewhat differs from general segmentation problems. And I do very much appreciate its potential benefit to the neuroscience community. This is indeed a plus for the paper. However, an important question is how much this paper can really improve over the existing solutions. Therefore, to demonstrate that the algorithm is able to correctly find closed contours, and really show stronger robustness against weak boundaries (This is especially important for bottom up methods), the authors do need to refer to more recent trends in the vision community.  2. I noticed one reference "Maximin affinity learning of image segmentation, NIPS 2009" cited in this paper. The paper proposed a very elegant solution to affinity learning. The core ideas proposed in this paper, such as greedy merging, Rand Index like energy function show strong connections to the cited paper. The maximin affinity is basically the weakest edge along a minimum spanning tree (MST), and we know greedy region merging is also based on cutting weakest edges on a MST. The slight difference is the author proposed the energy term at a lot of positions and scales but the previous paper has a single energy term for the global image. In addition cited paper also addressed a similar problem in the experiment. However, the authors not only did not include the citation as an experimental baseline, but also failed to provide detailed discussions on the relation between the two works.  3. The authors argued for greedy strategy, claiming this is better than factorizable energies. "By sacrificing factorization, we are able to achieve the rich combinatorial modeling provided by the proposed 3d shape descriptors." I guess what the authors mean is they are putting more emphasis on local predictions. But this statement is not solidly justified by the paper. In addition, although to some extent I could understand this argument (local prediction indeed seems important because the size of cells vs volume are much smaller than segments vs whole image in general segmentation), there are significant advances on making pretty strong local predictions from the general vision community using deep learning, which the authors fail to mention and compare.  I think overall the paper addressed an interesting problem and indeed showed solid research works. However it will better if the authors could better address contemporary segmentation literature and further improve the experiments.