Paper:
Learning from Small Sample Sets by Combining Unsupervised Meta-Training with CNNs
This work explores CNNs for the recognition of novel categories from few examples. Inspired by the transferability properties of CNNs, we introduce an additional unsupervised meta-training stage that exposes multiple top layer units to a large amount of unlabeled real-world images. By encouraging these units to learn diverse sets of low-density separators across the unlabeled data, we capture a more generic, richer description of the visual world, which decouples these units from ties to a specific set of categories. We propose an unsupervised margin maximization that jointly estimates compact high-density regions and infers low-density separators. The low-density separator (LDS) modules can be plugged into any or all of the top layers of a standard CNN architecture. The resulting CNNs significantly improve the performance in scene classification, fine-grained recognition, and action recognition with small training samples.

Review:
The paper describes an method for improving the generalizability of features from pre-trained convolutional neural networks. It introduces a new low-density separator layer that is trained in an unsupervised manner to discriminate between quasi-classes on a very large dataset of unlabeled images. The quasi-classes are selected from the data by first selecting a small number of examples using the k-means++ heuristic, and then expanding these iteratively using linear support vector machines. Given the quasi-classes, which are assumed to correspond to high density regions in the feature space, low density separators are then fit by optimizing an objective designed to produce separators that separate regions with small intra-class distances and large inter-class distances. The experiments show that the resulting features give impressive results on several datasets when there are few labeled training examples, outperforming features from a vanilla pre-trained CNN and other state-of-the-art (DAG-CNN).
The problem of exploiting large datasets of unlabeled images to improve representations is widely acknowledged as being extremely important, and this work makes an important step in this direction. The approach in the paper still requires a CNN that has been pre-trained on a large set of labeled images, but shows that more generalizable representations can be extracted from such a network using a post-hoc unsupervised training process on a larger sets of unlabeled images.  The idea of automatically identifying and using pseudo-classes for unsupervised training on a very large dataset comes from the authors previous ECCV submission [17] given in the supplementary material, with the main delta being that the method presented here is more general, and it allows for fine tuning the full network with backprop when sufficient examples are available. Personally, I think the delta is sufficient, and the results on standard datasets when there are few labeled examples clearly demonstrate the effectiveness of the approach.  In general, I think that the problem being addressed is important, that the idea is novel, the paper is well written and clear, and the results are impressive. The supplementary material is a valuable addition.  Issues:  It wasn't clear to me from the paper how exactly recursive feature elimination was done: are the most active features selected using the YFCC100M dataset or the target dataset? Also, I would expect that PCA would work better than random projections: was this attempted too?  The paper uses the term "unsupervised pre-training" to refer to the process of identifying the low dimensional separators. I found this to be a little confusing, as the term is often used to refer to methods like stacked autoencoders and RBMs for initializing weights when training deep nets. Here, the pre-training actually happens after the network has undergone supervised training on a labeled dataset. Furthermore, the "pre-training" step is not always followed by fine-tuning (i.e. when only a very small number of examples are available, and the features are used off the shelf), which makes is even more misleading.  Including some information (perhaps in the supplementary material) on how long the training took would be useful.  In Figure 4 you give the classification accuracies for a single scale CNN.  Since the single scale LDS+CNN replaces the last CNN layer, it would be good to see the performance of the single scale CNN for features from both the last and penultimate layers in these figures (i.e. fc6 and fc7).  Corrections: - Line 100: "An unit" -> "A unit" - Line 207: "SVM's" -> "SVMs" - Line 286: "significant" -> "significantly" - Figure 3: "categary" -> "category"