Paper:
We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) We suggest that the common approach of bootstrapping an image representation using the available labels introduces an unwanted bias, which we avoid by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) We use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; Finally, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled, simultaneously. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.

Review:
This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task. 
  Overall, my current vote for this paper is a weak reject. The main reason is that the paper really combines a set of known methods (self-supervised learning, consistency-based regularization, and an ad-hoc training regimen. On the other hand, the paper is well-written and provides nice rigorous experiments showing clear improvements over state of art by porting these known techniques from different domains (self/semi-supervised learning). However, if satisfactory answers to questions below are given, I am willing to change my rating.
Main Argument
Strengths
  - Overall the paper tackles an interesting problem, and does so in a way that achieves good results beyond state of art. 
  - An ablation study is provided which shows the contribution of different parts of the method. 
  - The paper is very well-motivated, written, and methods are described nicely and succinctly.
Weaknesses
  - Clearly the methods employed are, by themselves, not novel and have been used for a range of other ML problem formulations. What is the clear contribution/novelty of the work?
  - While the paper raises an interesting motivation about not biasing feature learning by using self-supervised learning, it's not clear to me that this claim is justified. What is the evidence for this, besides better performance? While the ablation w/o self-supervised learning performs more poorly than everything combined, the other parts of the ablation (with self-supervised learning) also perform poorly. Clearly, there is some interaction between the different aspects of the method, but I am not sure what that is. Why is the full combination so much better than if any one thing is removed?
  - The method is very similar to KCL (e.g. loss (3) is the same and justified via a graphical model formulation in that paper). The paper does not really read like it is building off of that though, which seems a bit misleading. Do the authors believe there is additional novelty, or is it a matter of adding the three contributions to KCL? What is the major reason that self-supervision with KCL still doesn't do as well? As far as I can tell, the only difference is self-supervised learning, the fact that you have a manual 3-stage curriculum, and the robust ranking method/consistency loss (i.e. the three contributions).
  - You assume that the number of clusters is known; this is one of the advantages of all of the prior work in that they can estimate this. How well does the method work if the number of clusters is not known? 
Additional comments not related to final vote:
  - The paper seems to start out in a way that implies the problem is new; citations should be provided to the three main compared works in the intro to make it clear that this is a follow-on to an existing problem. Further, as mentioned above MCL/KCL (especially the latter) are very similar in nature and introduced this problem (including [1] which also used unsupervised feature learning in one condition). These should be included as "work most related to ours" given that they came before Han et al.
  [1] Deep Image Category Discovery using a Transferred Similarity Function, https://arxiv.org/abs/1612.01253.