Paper:
Discriminative Unsupervised Feature Learning with Convolutional Neural Networks
Current methods for training convolutional neural networks depend on large amounts of labeled samples for supervised training. In this paper we present an approach for training a convolutional neural network using only unlabeled data. We train the network to discriminate between a set of surrogate classes. Each surrogate class is formed by applying a variety of transformations to a randomly sampled 'seed' image patch. We find that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition. The feature representation learned by our algorithm achieves classification results matching or outperforming the current state-of-the-art for unsupervised learning on several popular datasets (STL-10, CIFAR-10, Caltech-101).

Review:
This paper presents a new approach to discriminative unsupervised feature learning. Given an unlabelled set of images, 'seed' patches are extracted which act as surrogate classes for training. For each of these seed images, a variety of transformations is applied to generate training examples within the seed classes. A convolutional network is trained on top of the patches to predict the seed classes. After training, feature are extracted by forward propagating a new image, extracting the feature maps at each layer and pooling over them. Experimentation is performed on several standard image classification benchmarks, with additional experimentation to determine the effectiveness of transformations and invariance properties of the learned features.The proposed approach is simple and demonstrates strong performance on each of the classification tasks. The additional analysis, particularly the effects of removing transformations, was most welcome. The paper is clearly written for the most part with enough information given for the reader to reproduce the results.Detailed comments / questions:- Can you go into more detail about patch extraction? Are you just sampling a large number of patches, computing the gradients for each one, and choosing the top-N?- You should mention that the STL-10 dataset uses 96x96 images earlier in the paper, in order to give context for the choice of 32x32 patch size.- Was the 32x32 patch size arbitrary chosen? Do you have any intuition on how to select patch sizes as a function of the image sizes?- section 4.1: A picture or two, perhaps one illustrating the convnet during surrogate training and another illustrating feature extraction would be really helpful. The description is fairly clear for a reader familiar with this line of work but may be challenging to understand for those that are not.- From a practical point of view, given a dataset with a small amount of labels, one of the simplest things to try is feeding the images through a pre-trained convolutional net on ImageNet and training a linear classifer on top of these features. I think it would beneficial to include these results in table 1, even through they fall under a different category of algorithms (you could include this in a separate block of the table). It would be interesting to see how your results compare (at least on Caltech 101, your results are on par to DeCAF)- How much does validation on the surrogate task act as a proxy for validation on the classification task? Does the network with the best validation performance on the surrogate task also have the best validation performance on the classification task? Figure 3 hints that this might be true. Table 1 in the supplementary material presents classification accuracies with several networks. I would highly recommended including (maybe in a separate table) both the surrogate validation scores as well as the classification validation scores for each of these tasks, to see how well they correspond.- Supplementary, table 2: How come the diagonal is different than the results in table 1 with the same network?- To my best knowledge, the best published result on STL-10 is 70.1% +- 0.6% from "Multi-Task Bayesian Optimization" (Swersky et al, NIPS 2013). They achieved their results by first extracting k-means feature maps and training a convolutional network on top of these. By fixing the k-means features, the network is much less prone to overfitting. A strong baseline that I'd recommend the authors try, if time permits, is to use the same approach they do but include each of the transformations you use in the paper. This give a relatively straightforward way of running convnets on small labelled datasets which I suspect would be very competitive with your approach.