Paper:
Predicting Useful Neighborhoods for Lazy Local Learning
Lazy local learning methods train a classifier "on the fly" at test time, using only a subset of the training instances that are most relevant to the novel test example. The goal is to tailor the classifier to the properties of the data surrounding the test example. Existing methods assume that the instances most useful for building the local model are strictly those closest to the test example. However, this fails to account for the fact that the success of the resulting classifier depends on the full distribution of selected training instances. Rather than simply gathering the test example's nearest neighbors, we propose to predict the subset of training data that is jointly relevant to training its local model. We develop an approach to discover patterns between queries and their "good" neighborhoods using large-scale multilabel classification with compressed sensing. Given a novel test point, we estimate both the composition and size of the training subset likely to yield an accurate local model. We demonstrate the approach on image classification tasks on SUN and aPascal and show its advantages over traditional global and local approaches.

Review:
This paper proposes a method to predict, given a novel test instance, a useful subset of examples to train a local classifier. Existing methods in the space of lazy local learning typically fix the size of the selected subset and select relevant training instances based on the proximity (nearest neighbors) to the test example. By contrast, the proposed method estimates the required size of the relevant subset while allowing for deviation from a strict nearest neighbor ranking. The method consists of three main steps. The first (offline) is to generate a training set of instance-neighborhood pairs (obtained by distance-weighted sampling of neighbordhoods of varying sizes and for each training intance, heurestically selecting the best performing such pairs). The second (also offline) is to rely on the multi-label classification framework of [19] to learn a mapping between an instance and its useful neighborhood (reperesented as an indicator vector of confidences over the entire training set). The third (online) applies the learned mapping to a novel test intance to predict an indicator vector of confidences: from this vector, candidate neighbordhoods, of varying sizes, are generated via confidence-weighted sampling and used to train a number of local classifiers. Finally the prediction of the most confident local classifier is retained.This paper is extremely well written. The ideas are presented very clearly. The authors constantly situate the paper and the proposed methods well within the existing body of works. The result is a paper that reads well and highligths the important differences with respect to existing works.My main concern at this juncture is the fairness of the experimental evaluation. In general, the reviewer finds it surprising that the local baselines perform so poorly (almost consistently at chance) on the SUN dataset given that it is indeed perfectly well suited for lazy learning. However, beyond this, in the current setup it is unclear wether the gains with respect to the local baselines are due to the learning of relevant neighborhoods (steps 1 and step 2) or whether they are simply due to choosing the most confident local classifier out of a presumably large (how large is it?) set of trained classifiers (the final step in step 3). In other words, the reviewer believes a key and important baseline is missing here. This baseline would consist of applying step 3 on a confidence indicator vector (y^_q) whose entries are proportional to the distance (eucledian or learned) to the test query. In this manner, we can evaluate whether the reported gains are due to the learned y^_q or whether they are due to the trial and error tests of step 3, in which case the contribution of the paper is very small.In addition, the reviewer does not believe that using the auto-selected k from the proposed method to evaluate the local baseline is fair. The current reported local baselines should have their own auto-selected k values. If this is the case, please  clarify.Also, again with regards to experiments, it would be very important (even critical) to make explicit the exact number of positive training instances per data set and per attribute as well as the distribution of k per attribute so that we may get an understanding of the proportion of positives that are selected.Finally, the reviewer would like a motivation for the computation of k* in line 175 via an averaging of probabilities (in general probabilities cannot be averaged) for step 1. Also, how is the Platt calibration performed exactly? Finally, are the local classifiers in step 3 calibrated? If so, how? How many such local classifiers are used?