Paper:
Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
Convolutional Neural Networks (CNNs) can be shifted across 2D images or 3D videos to segment them. They have a fixed input size and typically perceive only small local contexts of the pixels to be classified as foreground or background. In contrast, Multi-Dimensional Recurrent NNs (MD-RNNs) can perceive the entire spatio-temporal context of each pixel in a few sweeps through all pixels, especially when the RNN is a Long Short-Term Memory (LSTM). Despite these theoretical advantages, however, unlike CNNs, previous MD-LSTM variants were hard to parallelise on GPUs. Here we re-arrange the traditional cuboid order of computations in MD-LSTM in pyramidal fashion. The resulting PyraMiD-LSTM is easy to parallelise, especially for 3D data such as stacks of brain slice images. PyraMiD-LSTM achieved best known pixel-wise brain image segmentation results on MRBrainS13 (and competitive results on EM-ISBI12).

Review:
The authors set out to solve a problem that the highly promising architecture of MDLSTM-RNNs have, which is the ability to parallelise computations. A novel sequential flow of information is proposed, which arguably makes parallelisation much easier. Major issues ===i) MDRNNs are not treated fairly. Fig 1 a) and Fig 2 left are dangerously misleading: they make the impression that it takes four MDRNNs to scan a whole 2d plane. But this is not true, the scanning can continue from the center point to the lower right corner, and the prediction can be made there. Consequently, a single sweep of a MDRNN sees *all* the pixels, a theoretical benefit of MDRNNs over Pyramid RNNs, for which this property is not shown.ii) The authors make clear that their computational flow, that of a Pyramid, introduces some independencies which make parallelization easier. What they do not is to elaborate on the consequences of this from a statistical view--i.e. what assumptions are made then.E.g. if we look at Fig 1 c), the pixels which are neighbours along the columns are treated as independent. This seems as a big restriction to me and I think should be discussed in the text--e.g. is it possible to undo this dependency? (see next point)iii) The C-LSTM layer is not justified, the authors do not give reasons why they do it. I can imagine that it helps to make up for the dependencies that the Pyramid layer is lacking--but the authors should make clear why this is done.iv) It is not explained very well how the segmentation is actually done. The paper makes the impression that a single pass over a volume forms a single prediction--the output of the softmax. But what if many pixels are to be segmented? Do the authors propose to sweepover the volume once per segmentation label? Further, why are the authors chosing the MSE as a training criterion, opposed to the negative log-likelihood of the categorical distribution induced by the softmax?v) The experiments are very domain specific, while the method is not. I don't see why no results on more standard benchmarks are presented, e.g. Pascal VOC. This would add a lot of value to the paper, as it eases the reader's effort to evaluate the method for applications in different contexts. Minor issues ===i) I think that Lucas Theis' work on generative modelling with spatial LSTMs [2] might (!) be of interest for the related work section.ii) The LSTM explanation is poor. The x symbol is overloaded, it is not clear what dimensionality the different quantities have, \text{} is not used in the subscripts, also the reader has to guess what x is. It should be easy to improve this to take cognitive burden from the reader.iii) Introduction, "neighbouring" -> "predecessing".iv) The correct cite for dropout on the non-recurrent connections of RNNs is [1], who did that earlier than Google. [1] Pham, Vu, et al. "Dropout improves recurrent neural networks for handwriting recognition." Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on. IEEE, 2014.[2] @unpublished{Theis2015c, author = "L. Theis and M. Bethge", title = "Generative Image Modeling Using Spatial LSTMs", year = 2015, journal = "arXiv", month = "Dec", keywords = "deep learning, generative modeling, natural image statistics, lstm, mcgsm", url = "http://arxiv.org/abs/1506.03478/" }