Paper:
Conventionally, convolutional neural networks (CNNs) process different images with the same set of filters. However, the variations in images pose a challenge to this fashion. In this paper, we propose to generate sample-specific filters for convolutional layers in the forward pass. Since the filters are generated on-thefly, the model becomes more flexible and can better fit the training data compared to traditional CNNs. In order to obtain sample-specific features, we extract the intermediate feature maps from an autoencoder. As filters are usually high dimensional, we propose to learn a set of coefficients instead of a set of filters. These coefficients are used to linearly combine the base filters from a filter repository to generate the final filters for a CNN. The proposed method is evaluated on MNIST, MTFL and CIFAR10 datasets. Experiment results demonstrate that the classification accuracy of the baseline model can be improved by using the proposed filter generation method.

Review:
The authors propose an approach to dynamically generating filters in a CNN based on the input image. The filters are generated as linear combinations of a basis set of filters, based on features extracted by an auto-encoder. The authors test the approach on recognition tasks on three datasets: MNIST, MTFL (facial landmarks) and CIFAR10, and show a small improvement over baselines without dynamic filters.
Pros:
1) I have not seen this exact approach proposed before.
2) There method is evaluated on three datasets and two tasks: classification and facial landmark detection.
Cons:
1) The authors are not the first to propose dynamically generating filters, and they clearly mention that the work of De Brabandere et al. is closely related. Yet, there is no comparison to other methods for dynamic weight generation. 
2) Related to that, there is no ablation study, so it is unclear if the authors’ contributions are useful. I appreciate the analysis in Tables 1 and 2, but this is not sufficient. Why the need for the autoencoder - why can’t the whole network be trained end-to-end on the goal task? Why generate filters as linear combination - is this just for computational reasons, or also accuracy? This should be analyzed empirically.
3) The experiments are somewhat substandard:
- On MNIST the authors use a tiny poorly-performance network, and it is no surprise that one can beat it with a bigger dynamic filter network.
- The MTFL experiments look most convincing (although this might be because I am not familiar with SoTA on the dataset), but still there is no control for the number of parameters, and the performance improvements are not huge
- On CIFAR10 - there is a marginal improvement in performance, which, as the authors admit, can also be reached by using a deeper model. The baseline models are far from SoTA - the authors should look at more modern architecture such as AllCNN (not particularly new or good, but very simple), ResNet, wide ResNet, DenseNet, etc.
As a comment, I don’t think classification is a good task for showcasing such an architecture - classification is already working extremely well. Many other tasks - for instance, detection, tracking, few-shot learning - seem much more promising.
To conclude, the authors propose a new approach to learning convolutional networks with dynamic input-conditioned filters. Unfortunately, the authors fail to demonstrate the value of the proposed method. I therefore recommend rejection.