Paper:
End-to-End Kernel Learning with Supervised Convolutional Kernel Networks
In this paper, we introduce a new image representation based on a multilayer kernel machine. Unlike traditional kernel methods where data representation is decoupled from the prediction task, we learn how to shape the kernel with supervision. We proceed by first proposing improvements of the recently-introduced convolutional kernel networks (CKNs) in the context of unsupervised learning; then, we derive backpropagation rules to take advantage of labeled training data. The resulting model is a new type of convolutional neural network, where optimizing the filters at each layer is equivalent to learning a linear subspace in a reproducing kernel Hilbert space (RKHS). We show that our method achieves reasonably competitive performance for image classification on some standard "deep learning" datasets such as CIFAR-10 and SVHN, and also for image super-resolution, demonstrating the applicability of our approach to a large variety of image-related tasks.

Review:
This paper provides a supervised manner to leran a multilayer kernel machine, which is an extension of convolutional kernel networks (without supervision). Its major purpose is to couple learning and data representation together within the context of kernel machines. 
This paper proposes an original idea and theoretically appealing solutions to solve it.   Quality:  Its first part (Section 1, 2) is excellent, but its latter part may be a little weak.   Section 3 is a little dense with numerous details and heuristics. A pseudo-code showing the overall framework may be helpful for readers.  Section 4 is a little short to validate the potential effectiveness of the proposed method. A direct comparison between CKN and CNN with same architecture may be more informative for readers.  Originality:  The purpose of this paper is appealing and its solutions are novel. It is worth an effort to explore the bridge between kernel machines and deep neural networks. But experimental results may not be enough to express its effectiveness fairly. Several direct comparisons will make this paper a more influential one.  Potential impact:  Its motivation is important, but more well-designed experiments will definitely make the effectiveness of this paper easier to accept.  Clarity:  I think Section 3 a little dense, a pseudo-code summarizing details will be welcome.  Detailed Comments:  (1) In the context of supervised learning, this paper seems to put less efforts to explain why the proposed solution would be superior to a standard CNN. I am not sure its natural regularization function (the norm in the RKHS) will be the key to make itself a competitive alternative to CNN.  (2) Since the superiority may not be easily evaluated in an analytical form, it will be more informative to validate them empirically. Otherwise, the discussion may, by chance, mislead readers falling into the assumption that a CKN is doomed to be better than a CNN.  (3) The discussion in the image classification experiments (Section 4.1) may not be sufficient. For example, why a 9-layer network is chosen? Will a shallower CKN be more suitable for CIFAR-10?  Moreover, it will be more informative to provide the performance of CNNs with same architecture. It can be understood if dropout and batch-normalization are removed in the training course of CNN, but it is important to show the superiority of supervised CKN in the same architecture.  Also, it will be beneficial to show the performance gain compared to its unsupervised counterpart (learnt by a k-means algorithm in all layers). Besides, the k-means algorithm is sensitive to initialization, whether the supervised CKN will still achieve competitive performance when the result of k-means is not good. Repeating the experiment several times or tuning the network from a random initialization will give more insights to readers.  At last, I have several personal curiosities about the method. a) How many patches is required for the k-means algorithm to output a good initialization? b) Whether transferring weights from a pre-trained network will boost its performance. c) Whether fine-tuning the 2-layer CKN-GM or CKN-PM in [18] will show improvements. I personally think this is the key to show the effectiveness of the proposed solution.  (4) The image super-resolution experiment may have the same problem. Why is a 9-layer network preferred? Since [8] claimed that a deeper network is hard to output a better result compared to a shallower one, it will be beneficial to discuss the reason that the supervised CKN can tackle this problem. I also wonder whether the performance of a 3-layer CKN will outperform its counterpart CNN in [8].  Moreover, in the context of super-resolution, inference time is also important to evaluate a method. I think the 9-layer CKN with 128 filters for each layer may be too computationally heavy for this task. It is helpful to provide a visualization of the trade-off between accuracy and inference time for different methods like Figure 1 in [26].  Summary:  Overall, it is a promising paper with potentially huge influence. Due to the limited space, a tradeoff between details and experiments will surely make itself a shining one in the NIPS.