Paper:
Decision forests (DF), in particular random forests and gradient boosting trees, have demonstrated state-of-the-art accuracy compared to other methods in many supervised learning scenarios. In particular, DFs dominate other methods in tabular data, that is, when the feature space is unstructured, so that the signal is invariant to permuting feature indices. However, in structured data lying on a manifold-such as images, text, and speech-neural nets (NN) tend to outperform DFs. We conjecture that at least part of the reason for this is that the input to NN is not simply the feature magnitudes, but also their indices (for example, the convolution operation uses "feature locality"). In contrast, naïve DF implementations fail to explicitly consider feature indices. A recently proposed DF approach demonstrates that DFs, for each node, implicitly sample a random matrix from some specific distribution. Here, we build on that to show that one can choose distributions in a manifold aware fashion. For example, for image classification, rather than randomly selecting pixels, one can randomly select contiguous patches. We demonstrate the empirical performance of data living on three different manifolds: images, time-series, and a torus. In all three cases, our Manifold Forest (MO R F) algorithm empirically dominates other state-of-the-art approaches that ignore feature space structure, achieving a lower classification error on all sample sizes. This dominance extends to the MNIST data set as well. Moreover, both training and test time is significantly faster for manifold forests as compared to deep nets. This approach, therefore, has promise to enable DFs and other machine learning methods to close the gap with deep nets on manifold-valued data.

Review:
====== Post Rebuttal ======
No rebuttal was provided and all reviewers have raised issues. Therefore, I will maintain the original rating.
====== Summary====== 
The paper puts forward a potential issue with the standard decision tree and tries to remedy it. The issue is that standard decision trees, by independently picking the split feature(s), virtually discard the structure of the input features (if any). In this work, structure essentially refers to ordered input features such as sequences, signals, and images. The work is motivated by the fact that convolutional networks (ConvNet) exploit this structure thanks to its local convolution operation. The idea of this work is to bridge this gap by constraining the split feature selection to follow the local structures of the input where each split feature would correspond to a local bounding box of the input feature. Applying this idea to random forests, it demonstrates significant improvements over unstructured split features, on a few synthesized datasets as well as MNIST dataset. The performance is also compared to one ConvNet architecture which demonstrates similar performance on synthesized datasets while being considerably slower than the proposed random forests.
====== Strengths and Weaknesses ====== 
+ The motivation to make random forests respect the input structure similar to ConvNets is well-grounded and important since random forests are still being used in certain applications where computational complexity and/or interpretability are crucial factors.
+ It proposes a simple technique to add locality to the split criterion which brings significant improvements over the standard random forest.
- I believe the paper’s title should be closing the gap to *convolutional* neural networks since fully-connected networks are not local in the first place. 
- continuing on the previous point, the proposed method pushes the decision forests closer to “locally-connected” networks where, although the features are local, they are not shared across different locations in the input. The additional “sharing” property which takes locally-connected networks to convolutional networks is an important property of ConvNets since it brings translation equivariance for the representations. The proposed random forest method is not translation-equivariant by design and is only local. 
- regarding the previous point, a larger difference between ConvNet and the proposed method is more imminent if one goes to datasets with “non-aligned” observations. This already becomes more evident in the MNIST (which contains mostly aligned digits) where ConvNet clearly outperform MORF but would likely become more significant when going to real-world datasets, e.g. CIFAR.
- the proposed method is very similar to patch-based random forest image recognition methods. For instance, several variants exist that are used for object or part detection in a given image where a patch is selected from the image for the split criterion. This patch will respect locality in a similar way to the proposed MORF’s bounding boxes. For instance, see “tomography scans” example of Criminisi et al. 2012 (section 4.5).
- The paper is missing to provide many important details
- what are the actual hyperparameter (hp) values used for the different methods in figure 1,2, and 3. This includes the hp relevant for the random forests including the number of decision trees, the stopping criterion, the random data partitioning method, number of random projections, as well as h_min, h_max, w_min, and w_max. It also does not discuss the hyper parameters of the baseline methods including k for KNN, distance measure for KNN, penalty cost for SVM, variance for the RBF kernel, ConvNet architecture, etc.
- more importantly, it is not mentioned how these hp are optimized for the different baselines as well as the proposed method. What algorithm has been used (e.g. grid search)? How much hp optimization budget is used for different baselines? Is there a validation set put aside for hp optimization?
- from the description in the start of page 4, it seems that the atoms for the proposed MORF are vectors of binary dimensions while for the general SPORF each atom’s element can be -1 as well (page 3). Why is this choice made despite the fact that it reduces the capacity of the model?
- the bounding box sampling seems biased as presented. That is, bounding boxes closer (than h_max and/or w_max) to the right and/or lower borders are more likely since the number of valid boxes will be lower. 
====== Final Decision ====== 
I think it will be very interesting to bridge the gap between ConvNets and random forests since the latter comes with attractive properties. While I find all the concerns that are listed above important, my rating is mainly due to the novelty of this work compared to the prior patch-based random forest techniques for image analysis.
====== Points of improvements ====== 
I believe it’s important to disentangle the two main properties of convolution operation in ConvNets being shared and local parameters. Then, accordingly propose strategies to bring these properties to a decision tree.