Paper:
Active Learning from Imperfect Labelers
We study active learning where the labeler can not only return incorrect labels but also abstain from labeling. We consider different noise and abstention conditions of the labeler. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under fairly natural assumptions on the noise and abstention rate of the labeler. This algorithm is adaptive in a sense that it can automatically request less queries with a more informed or less noisy labeler. We couple our algorithm with lower bounds to show that under some technical conditions, it achieves nearly optimal query complexity.

Review:
The paper studies active learning where the labelers may make errors and abstain from providing a label. The paper shows lower bounds and upper bounds on the number of queries needed in the one dimensional case and d dimensional case.  One possible motivation for this work is the results of Shah and Zhou “Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing” that shows that the only incentive compatible mechanism (under some assumptions) for crowed sourcing requires that workers abstain when they are uncertain.
 • You are making the assumption that the hypothesis is the epigraph of holder smooth function. How realistic is this assumption? For example, can the decision boundaries of neural network, decision trees and kernel models have these properties? • Line 120: you are making here the assumption that the distribution is uniform over the unit cube. This should be made explicitly. How sensitive are the results to this assumptions? For example, Freund, Seung, Shamir and Tishby in their work on the query by committee algorithm used an assumption that the distribution is within a constant from uniform. • Assumption 3 seems very restrictive since it assumes equality.  • Line 162 – can you show how does the assumption result in the decision boundary being a single point? • The technique uses membership queries and furthermore assume that the labeler is not consistent in the sense that when queried multiple times on the same point it may return different answers. Both assumptions are questionable. • Theorem 2: since the algorithm performs the significance test at every round, don’t you need to use the union bound to make sure that the significance test is true for all queries? In that case, the significance bound for the test should be updated to reflect the number of times it is being used. • Lines 232-234: you claim that errors are more harmful than missing labels. Is it a universal truth or specific to the current set of assumptions? • Theorem 9: the number of queries made is exponential in the dimension whereas standard PAC models are typically linear in the dimension. In what cases is it better to use your model?   In their response, the authors address many of my concerns. The constraints are not "as bad" as I thought they are. Nevertheless, there are still many constraints in this setting and no demonstration of a case in which these constraints hold. I think this paper can be much improved and would create bigger impact after such improvements are implemented.