  Hybrid crowd-machine classifiers can achieve superior performance by
combining the cost-effectiveness of automatic classification with the accuracy
of human judgment. This paper shows how crowd and machines can support each
other in tackling classification problems. Specifically, we propose an
architecture that orchestrates active learning and crowd classification and
combines them in a virtuous cycle. We show that when the pool of items to
classify is finite we face learning vs. exploitation trade-off in hybrid
classification, as we need to balance crowd tasks optimized for creating a
training dataset with tasks optimized for classifying items in the pool. We
define the problem, propose a set of heuristics and evaluate the approach on
three real-world datasets with different characteristics in terms of machine
and crowd classification performance, showing that our active hybrid approach
significantly outperforms baselines.
