Paper:
Convolutional Neural Fabrics
Despite the success of CNNs, selecting the optimal architecture for a given task remains an open problem. Instead of aiming to select a single optimal architecture, we propose a "fabric" that embeds an exponentially large number of architectures. The fabric consists of a 3D trellis that connects response maps at different layers, scales, and channels with a sparse homogeneous local connectivity pattern. The only hyper-parameters of a fabric are the number of channels and layers. While individual architectures can be recovered as paths, the fabric can in addition ensemble all embedded architectures together, sharing their weights where their paths overlap. Parameters can be learned using standard methods based on backpropagation, at a cost that scales linearly in the fabric size. We present benchmark results competitive with the state of the art for image classification on MNIST and CIFAR10, and for semantic segmentation on the Part Labels dataset.

Review:
The authors propose a convolutional neural fabric model, which addresses the issue of selecting an optimal CNN architecture. The proposed "fabric" model consists of 3D trellis, which are essentially the response maps that capture scale, layer, and channel information of the traditional CNN architectures. Each node in the trellis is connected to a 3x3 scale-channel neighborhood from the previous layer. The authors show that the traditional CNN architectures can be embedded and implemented via neural fabric models. Finally, the authors present results on MNIST, CIFAR and "Part Labels" dataset and show competitive results. 
I think the authors present an interesting and a novel idea that addresses an important problem of how to select an optimal CNN architecture. I think the major strength of this paper is the novelty in the technical approach.  I had several comments in regards to the experiments. As is standard for many deep learning architecture papers, the authors conduct experiments on MNIST dataset, CIFAR, and in this case "Part Labels" dataset. In my opinion, the tasks related to these datasets (especially MNIST) are not very challenging, and the results are extremely saturated (over >95% accuracy). I would like to see some of the experiments on more challenging tasks such as semantic segmentation on PASCAL, etc. Having said this, I think the key contribution of this paper is the novelty of the method, and I would be willing to overlook the experiments on these "less challenging" tasks, especially since it seems like this is a common practice in this field (a practice, which is not good in my opinion).  Additionally, I also didn't find some of the experiments that convincing. I thought that the results for the Parts Labeling task were solid. However, the disparities between the error rates on the MNIST dataset seem pretty large (considering how saturated the performance is). Especially if we look at the top 2 baseline methods, we observe that without using data augmentation , these methods achieve better results with significantly less number of parameters (>3M) than CNF-dense. The same trend occurs on the CIFAR dataset, where we have top 2 methods significantly outperform CNF-dense method with almost 30M less parameters than CNF-dense. Therefore, based on these experiments, I am not convinced that the method would work very well on some of the more challenging tasks (such as semantic segmentation).   In summary, even though I didn't find many of these experiments convincing, I would still like to see this paper published. There has been a lot of incremental work on CNNs recently, that do not contribute much to the field despite very good performance in certain tasks. In comparison,  even though this paper may not have the most impressive empirical results, it presents a novel alternative to CNNs, which addresses an important CNN architecture selection problem.  ---POST REBUTTAL---  I read the rebuttal. I agree with AR5 that the authors should include more experiments clearly outlining theoretical and practical costs of training compared to the traditional CNNs. As I mentioned previously, I am not convinced that the proposed approach is practical at its current state. It doesn't seem to outperform other techniques on the more challenging datasets, even if it uses a larger number of parameters than other methods. However, I like the idea of the paper, and I think that it could have a solid impact on deep learning community, as the CNN architecture selection problem is an important one.