Paper:
Query K-means Clustering and the Double Dixie Cup Problem
We consider the problem of approximate K-means clustering with outliers and side information provided by same-cluster queries and possibly noisy answers. Our solution shows that, under some mild assumptions on the smallest cluster size, one can obtain an (1 + )-approximation for the optimal potential with probability at least 1 − δ, where > 0 and δ ∈ (0, 1), using an expected number of O(
noiseless same-cluster queries and comparison-based clustering of complexity O(ndK + K 3 δ ); here, n denotes the number of points and d the dimension of space. Compared to a handful of other known approaches that perform importance sampling to account for small cluster sizes, the proposed query technique reduces the number of queries by a factor of roughly O(
3 ), at the cost of possibly missing very small clusters. We extend this settings to the case where some queries to the oracle produce erroneous information, and where certain points, termed outliers, do not belong to any clusters. Our proof techniques differ from previous methods used for K-means clustering analysis, as they rely on estimating the sizes of the clusters and the number of points needed for accurate centroid estimation and subsequent nontrivial generalizations of the double Dixie cup problem. We illustrate the performance of the proposed algorithm both on synthetic and real datasets, including MNIST and CIFAR 10.

Review:
This paper investigates the problem of active-semi-supervised clustering, by considering both noiseless (perfect oracle) and noisy (imperfect oracle) query responses. The authors provide probabilistic guarantees for low approximation errors to the true optimal k-means objective. The corresponding query complexities are substantially lower than in the existing literature. Importantly, as noted by the authors, their query complexity is independent of the size of the dataset.  The main strength of the paper lies in the considerable technical rigour with which the subject has been handled. Numerous results are presented which are either very novel, or, where similar to existing results, provide substantial improvements in terms of the resulting complexities. The experiments while not extensive, are adequate to show considerable improvement over the apparently only available relevant pre-existing method. The paper discusses very technical material, but the authors have done well to elucidate the important points within the text.  My major concern lies in the practical relevance of the theory. If I understand correctly the assumption is that the oracle has perfect knowledge of the optimal k-means solution, and not of the true solution. However, this seems quite implausible in most situations. Indeed, as far as I can tell in the experiments the queries are answered based on whether or not the points are in the same true cluster, and not the optimal k-means assignment. Certainly this seems to be a much more realistic implementation, especially in the crowdsourcing scenario mentioned by the authors. That said, the performance on the real datasets is clearly good. It still does not take much of a stretch to imagine very simple examples where the true clusters' means are markedly different from the k-means solutions'.  Given the relative infancy of this area, it would be interesting to compare with existing (non active) semi-supervised clustering or semi-supervised classification algorithms (the latter both active and non active). For example, how many true labels are needed for semi-supervised classification to do as well as your approach? This comparison seems relevant if weighing up the relative cost of label queries against pairwise assignment queries.  I recorded only one significant typo: in the description of Algorithm 2 there is repetition of "without replacement".  ########################################  In the authors' response the justification of the assumptions underlying their theoretical contributions were revisited. Of course I completely recognise that it is necessary to have a well defined mathematical or probabilistic description of the target solution. I also recognise that in many real clustering problems the true cluster labels may not be consistent with any intuitive mathematical formulation of a clustering objective. I would still argue that the assumption of the availability of the optimal k-means solution to an oracle (or crowd) is very unrealistic. I would find it far more satisfying if (even relatively strong) assumptions on the true clusters were made, e.g., that the data arise from a GMM with perhaps an upper bound on the overlap/misclassification rate w.r.t. the true model. Even if this is equally unrealistic in practice, it at least does not create a disconnect between the assumption (that optimal k-means solution is known) and implementation (queries are answered w.r.t. the true clusters and not w.r.t the clusters in the optimal kmeans solution). That said, I am satisfied that this issue is more relevant to the entire literature on this problem and not the paper specifically. Also that there is more than sufficient precedent for these assumptions in high quality publications.  I also accept that given the page limit it is not possible to include results comparing active-semi-supervised clustering with semi-supervised classification. The authors' applications in bioinformatics sound extremely interesting and relevant given the importance of same cluster queries being more realistic than label queries. I would find it interesting as a reader to have an example described in the paper where label queries are not relevant. This does not necessitate inclusion of experiments on such a problem, just a description perhaps in the introduction, motivating the relevance of the problem the authors are studying.  Given the above, and reading the other reviewers' comments, I more strongly recommend acceptance.