Paper:
Active Nearest-Neighbor Learning in Metric Spaces
We propose a pool-based non-parametric active learning algorithm for general metric spaces, called MArgin Regularized Metric Active Nearest Neighbor (MARMANN), which outputs a nearest-neighbor classifier. We give prediction error guarantees that depend on the noisy-margin properties of the input sample, and are competitive with those obtained by previously proposed passive learners. We prove that the label complexity of MARMANN is significantly lower than that of any passive learner with similar error guarantees. Our algorithm is based on a generalized sample compression scheme and a new label-efficient active model-selection procedure.

Review:
This paper studies active learning for nearest neighbor classifiers in metric spaces. Built upon a series of work on nearest neighbor classifiers (e.g. Gottlieb and Kontorovich,  2015), the paper provides an active learning algorithm for nearest neighbors, dubbed MARMANN, that has nontrivial label complexity guarantees. Specifically, an example is provided to show that the proposed algorithm has superior label complexity over its passive version.  The key technical trick is to select an appropriate "scale", t, such that the margin error term epsilon(t), and the sample compression generalization bound phi(t), are almost of the same order. After selecting the scale, a 1-NN classifier is output.
Technical quality: the proofs in this paper are quite sound.   For practical implementation, I wonder if the labels queried for different scales t can be reused. Perhaps we can organize the data in some tree structure, e.g. cover trees (Beygelzimer, Kakade, Langford, 2004), and walk down the cover tree in some way.  In addition, I wonder if the idea of "active comparison of pairs of classifiers" in (Balcan, Hanneke, Vaughan, 2010) can be used in the model selection (scale selection) procedure, since the term O(1/\hat{G}) term is a bit unsatisfying to me.   The fact that the label complexity bound is exponential in the doubling dimension of the metric space is a bit concerning, although I think this type of dependency may be difficult to get rid of for nonparametric classifiers.  Novelty: although the general idea of using sample compression bounds for condensed nearest neighbor classification has appeared in e.g. (Gottlieb, Kontorovich, Nisnevich, 2014), the idea here using random sampling in each Voronoi cell and noisy binary search the "correct" scale to minimize the error bound is novel.  Potential Impact: There has been relative little literature on active learning with non-parametric classifiers, e.g. nearest neighbor classifier. I think this is a nice starting point and will spur future work.  Clarity and Presentation: The main result, Theorem 3.1 is a bit difficult to appreciate on first sight, since it is very different from PAC learning with a fixed/parameterized hypothesis class.   In addition, the paper may need some explanation on how close is the error of the classifier to the Bayes optimal, in additive sense.  I find Theorem 5.4 and 5.5 difficult to follow, but I understand the high level idea as follows -- we would like to find two neighbor scales t_0 < t_0' such that \hat{\epsilon}(t_0) < \phi(t_0) \hat{\epsilon}(t_0') > \phi(t_0') If this is true then the scale t_0 is optimal up to constants in terms of error bound achieved. Is that right?   *** UPDATE AFTER READING AUTHOR FEEDBACK AND DISCUSSION *** I think overall this is a nice paper that provides active learning algorithm working in general metric space and providing nontrivial label complexity guarantees. I vote for acceptance.   In the author feedback, the author says that in realizable case, Algorithm 1 of BHV10 is not directly applicable; I can roughly see that since the number of scales to be considered is O(m^2). But under some standard regularity assumptions (density for each class does not change too fast), can one use a "doubling trick", i.e. set scales t_L = t_1 \leq t_2 \leq .. \leq t_n-1 \leq t_n = t_U  such that for any i,  nu(t_i+1) \leq 2 * nu(t_i)  phi(t_i-1) \leq 2 * phi(t_i) (*)   and we make sure that  nu(t_U) \leq 1/m  phi(t_L) \leq 1/m (*)   Now consider running Algorithm 1 of BHV10 with a label budget L. for each scale t, we call GenerateNNSet(t, I, delta) with label budget L/n. (If during the execution of the procedure it runs out of label budget then it simply returns a trivial classifier.) Then active comparison is performed to output a final classifier. This can make sure that the algorithm adapts to the "correct" scale with a additional multiplicative factor of O(n log n/\delta) label complexity.   Admittedly, my idea is a crude one, since finding scales that satisfies conditions (*) apriori is a bit difficult without knowing anything about the data distribution.  (Reviewer 1 agrees that something like this might work, but some technical details need to be worked out.)