Paper:
On-the-Job Learning with Bayesian Decision Theory
Our goal is to deploy a high-accuracy system starting with zero training examples. We consider an on-the-job setting, where as inputs arrive, we use real-time crowdsourcing to resolve uncertainty where needed and output our prediction when confident. As the model improves over time, the reliance on crowdsourcing queries decreases. We cast our setting as a stochastic game based on Bayesian decision theory, which allows us to balance latency, cost, and accuracy objectives in a principled way. Computing the optimal policy is intractable, so we develop an approximation based on Monte Carlo Tree Search. We tested our approach on three datasets-named-entity recognition, sentiment classification, and image classification. On the NER task we obtained more than an order of magnitude reduction in cost compared to full human annotation, while boosting performance relative to the expert provided labels. We also achieve a 8% F 1 improvement over having a single human label the whole set, and a 28% F 1 improvement over online learning.
"Poor is the pupil who does not surpass his master." -Leonardo da Vinci

Review:
This paper presents a hybrid approach for using both     crowdsourced labels and an incrementally (online) trained model     to address prediction problems; the core idea is to lean heavily     on the crowd as the system is ramping up, learn from the labels     thus acquired, and then use the crowd less and less often as the     model becomes more confident. This is done via a sophisticated     framing of the problem as a stochastic game based on a CRF     prediction model in which the system and the crowd are both     players. The system can issue one or more queries q for tokens x     (with true label y) which elicit responses r, where there is a     utility U(q,r) for each outcome; the system thus attempts to     pick the actions that will maximize the expected     utility. Furthermore, the queries are not issued all at once,     but at times s (with response times t); utility is maximized     with respect to a t_deadline by which an answer needs to be     computed (this thus determines how many queries are sent out, at     what rate, etc.)Computing this expected utility requires using     the simulation dynamics model P(y,r,t|x,q,s) in order to compute     the utilities as in (4). Given the utility values, the optimal     action could be chosen; however, the introduction of continuous     time makes this intractable to optimize and as such an     approximation is used based on Monte Carlo Tree Search and TD     learning (Algorithm 1). Experiments were conducted on a     simplified named entity recognition (NER) task, a sentiment     recognition task, and a face identification task, using four     methods: the majority vote of n human judges (1,3,5), online     learning (using the true labels), the "threshold" baseline (the     authors' model but without continuous time, in which m queries     are sent out at each step until the model prediction's     uncertainty is reduced below a threshold), and finally the     authors' full model (LENSE). In terms of precision/recall/F1,     the full model outperforms all but the 5-vote crowdsourced     version, though the "threshold" baseline does nearly as well     (and with lower latency for NER; Table 2). The authors also show     how the model requires fewer and fewer crowdsourced labels over     time given a sufficiently strong model to train (see Figure 3),     and that compared to online learning the accuracy is high from     the first example (Figure 4), since the system can leverage     crowd expertise more heavily while the model uncertainty is     still high.      This was a really interesting paper, and one that I expect could     generate both a lot of discussion and further work as well as     adoption in practice. There have been a variety of heuristic     approaches to learning from the crowd while training a system,     but this is the first complete and principled proposal I have     seen. The results against reasonable baselines are     impressive. As such, I feel the work is highly original and     significant, and in my mind deserves to be included in the     conference. That said, it does not fare nearly as well on the     clarity front, which unfortunately could detract from the     paper's potential impact. Many of these issues are correctible     (even for the camera-ready) if the authors are willing to put in     the time and effort and perhaps somewhat adjust the tone with     respect to the "threshold" baseline vs. the full continuous time     version.      The primary issue with clarity comes from the introduction of     continuous time, beginning partway through Section 3. The     motivation is reasonable, i.e., it might be necessary to operate     on a time (as well as a cost) budget for each classification,     but experimentally it seems to have very little benefit, while     the cost is that the formulation and discussion (as well as the     optimization procedure) become substantially more complex.In     fact, in Table 2, it appears that the "threshold" baseline     achieves significantly lower latency on the NER task while still     getting roughly equivalent performance (marginally worse); on     the face ID task it actually performs better (though still     roughly equivalent). The authors argue (l. 314) that the     baseline does better on the second task because it's a single     label prediction task and doesn't have significant interaction     between labels - however, the NER task, which has such     interaction in spades, only seems to benefit marginally from     this.For many practical tasks, the "threshold" baseline will     be good enough, and is already such a signficant gain (in terms     of being a principled hybrid learning system that outperforms     active and online learning).The paper would likely have     greater impact if the authors made this clear in the     development, i.e., the core method could be developed fully     (without the introduction of time, and as such would be easier     to understand and implement), and the addition of continuous     time could be shown as an extension.In the same vein, when     discussing the results, the authors could be more clear that the     performance of the "threshold" method is quite strong, even in     cases where there is a sequence dependence between tasks. This     is a suggestion and not strictly necessarily, as I feel the     paper is strong enough for inclusion as it is, but I do feel it     would improve the paper and the chances for the technique being     widely understood and adopted.      There are a number of smaller issues with respect to the results     and how they are reported. First, the introduction states that     the model "beats human performance" - but this is not true in     the 5-vote case for NER; strangely the 5-vote case is missing     for the Face-ID and Sentiment tasks in multiple tables and     figures (the right half of Tables 2, Table 3 and Figure 4) - it     really should be included. Likewise, in Table 2, the results for     the 5-vote case should be bolded as they represent the best     performance. The latency value for Threshold in NER should also     be bolded for the same reason (and the LENSE value for Face     ID). More importantly, the "Threshold" baseline is missing for     the sentiment experiment, and as such doesn't appear in Table 3     or Figure 4 - again, it should really be included as well.            A few minor comments with respect to the method description:            -While there is a discussion of t_deadline in l.183-187,     it is not clear what if any t_deadline was used in the     experiments, and what criteria was used to determine when the     answer was ready to report an answer - was it an entropic     threshold as with the baseline or was it reachingsome t_deadline?            -In equation (2), it seems the distribution needs to be given     the variable s as well (i.e., p(y,r,t|x,q,s).      -In equation 5, the use of q in F(q) is *very* confusing - I     assumed this referred to a query q, but in fact it just     represents a nominal distribution function - please choose some other     letter, as there are plenty to choose from.      -In the description of the Threshold baseline (l.246-252), I     assume the the expression should be (1 - p_theta(y_i|x))*0.3^m     <= (1-0.88) (note the RHS) as opposed to >= 0.88, as increasing     m will reduce uncertainty, not increase it.