Paper:
The quest for increased visual recognition performance has led to the development of highly complex neural networks with very deep topologies. To avoid high computing resource requirements of such complex networks and to enable operation on devices with limited resources, this paper introduces adaptive kernels for convolutional layers. Motivated by the non-linear perception response in human visual cells, the input image is used to define the weights of a dynamic kernel called Adaptive kernel. This new adaptive kernel is used to perform a second convolution of the input image generating the output pixel. Adaptive kernels enable accurate recognition with lower memory requirements; This is accomplished through reducing the number of kernels and the number of layers needed in the typical CNN configuration, in addition to reducing the memory used, increasing 2X the training speed and the number of activation function evaluations. Our experiments show a reduction of 66X in the memory used for MNIST, maintaining 99% accuracy and 16X memory reduction for CIFAR10 with 92.5% accuracy.

Review:
The paper introduces adaptive kernels (that adapts its weights as a function of image content) to the framework of CNN. The benefit of adaptive kernels is the reduction of memory usage (at training and at the inference time) as well as training speedups (up to 2x). The kernels are evaluated on two datasets MNIST and CIFAR10
I like the idea of building models that are memory efficient at training and at evaluation time. However, the evaluation of the proposed adaptive kernels is rather limited. In order to improve the paper, the authors could take into consideration the following points:
1. Why there is still a need to combine adaptive convolutions with regular convolutions? What would the model performance be for a model with only adaptive kernels?
2. I might have missed it, but I couldn't find any motivation on why tanh is used as nonlinearity. Would the method work with relu?
3. Traditional convolutional kernels together with max pooling operations ensures some degree of translation invariance. How big is the generalization gap for the tested models when adaptive kernel is used?
4. How sensitive are the results to the number of adaptive kernels in the layers.
5. Adaptive kernels have only been tested in the first convolutional layer, would the adaptive kernels work well also in different layers?
6. On CIFAR10 the results seem to be worse that other methods. However, it is important to note that the Adaptive Kernels CNN has way less parameters. It would be interesting to see how the performance of adaptive kernels based CNNs scales with the number of parameters.
7. The evaluation on two datasets seems to be rather limited, additional comparisons should be included.
8. The authors acknowledge the similarities (and some differences) with Brabandere et al (2016). It might be beneficial to include comparison to this approach in the experimental section. Moreover, given the similarities, it might be good to discuss the differences in the approaches in the introduction section.
9. The ideas presented in the paper seems related to general concept of hypernetworks, where one network learns (or helps to learn) paramenters of the other network. It would be nice to position the ideas from the paper w.r.t. this line of research too.
10. Another related paper seems to be Spatial Transformer Networks (Jaderberg et al.).
I like the drawings, however, the font on the drawings is too small - making it hard to read.
Some typos:
1. the difficult to train the network
2. table 2: Dynamic -> Adaptive?
Overall, the paper presents interesting ideas with some degree of originality. I'd encourage the authors to extend the intro and position the ideas w.r.t. existing works and extend the evaluation.