Paper:
Scoring Workers in Crowdsourcing: How Many Control Questions are Enough?
We study the problem of estimating continuous quantities, such as prices, probabilities, and point spreads, using a crowdsourcing approach. A challenging aspect of combining the crowd's answers is that workers' reliabilities and biases are usually unknown and highly diverse. Control items with known answers can be used to evaluate workers' performance, and hence improve the combined results on the target items with unknown answers. This raises the problem of how many control items to use when the total number of items each workers can answer is limited: more control items evaluates the workers better, but leaves fewer resources for the target items that are of direct interest, and vice versa. We give theoretical results for this problem under different scenarios, and provide a simple rule of thumb for crowdsourcing practitioners. As a byproduct, we also provide theoretical analysis of the accuracy of different consensus methods.

Review:
This paper examines the problem of determining 
      what fraction of control questions (with known answers) vs. target 
      answers (unknown) to use when using crowdsourcing to estimate 
      continuous quantities (i.e., not categorical judgments). The authors 
      describe two models for using control questions to estimate worker 
      parameters (bias and variance from true answers); a two-stage 
      estimator which estimates worker parameters from the control items 
      alone, and a joint estimator which comes up with an ML estimate using 
      both control and target items. The authors derive expressions for the 
      optimal number of target items (k) for each case, beginning with a 
      clear statement of the results and then going through detailed but 
      clear derivations. They then show their results empirically on both 
      synthetic and real data, showing how the estimates align with the true 
      optimal k in cases where model is a perfect match to the data vs. 
      misspecified (for synthetic) and then show how the practical effects 
      of misspecification when dealing with real data. They close with 
      valuable recommendations for practitioners in terms of choosing 
      between the models. First of all, this is a very important and 
      highly practical setting - as someone who has run many crowdsourced 
      tasks as well as read/heard many accounts from others, using control 
      questions is a tried and true method of estimating bias and variance 
      in judges; much of the past theoretical work has ignored this and 
      assumed no such control questions are available. While control 
      questions could be used in these other methods in principle, I 
      know of no previous paper that has examined the *value* of control 
      items and how many should be used in a given setting. I have often 
      wondered about this question in my own experiments, and have 
      considered working on the problem myself; as such I was delighted to 
      read this thorough treatment by the authors. This paper is excellent 
      in so many ways: it is unusually clear in its writing, from the 
      motivation and setup to the explanation of their the strategy and 
      purpose of their approach before diving into the derivations, to the 
      setup and explanation of the experiments and their implications. The 
      past literature is well-covered, the figures are clear, the notation 
      and development are easy to follow. The estimation algorithms and 
      the optimal k are clear, and the discussion of the effects of 
      model mismatch and recommendations for real settings/applications 
      are insightful and valuable. I would recommend this paper not only to 
      my colleagues who work on the analysis of crowdsourcing, but also to 
      many others who are users of crowdsourcing: an excllent paper all 
      around, that I expect will be well-regarded at the conference and 
      well-cited in future years.