Approximating the crowd
The problem of approximating the crowd is that of estimating the crowds majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an algorithm, CrowdSense, that works in an online fashion where items come one at a time. CrowdSense dynamically samples subsets of the crowd based on an exploration/exploitation criterion. The algorithm produces a weighted combination of the subsets votes that approximates the crowds opinion. We then introduce two variations of CrowdSense that make various distributional approximations to handle distinct crowd characteristics. In particular, the first algorithm makes a statistical independence approximation of the labelers for large crowds, whereas the second algorithm finds a lower bound on how often the current subcrowd agrees with the crowds majority vote. Our experiments on CrowdSense and several baselines demonstrate that we can reliably approximate the entire crowds vote by collecting opinions from a representative subset of the crowd.
approximating the crowd crowdsourcing wisdom of crowds labeler quality estimation aggregating opinions