  Samples with ground truth labels may not always be available in numerous
domains. While learning from crowdsourcing labels has been explored, existing
models can still fail in the presence of sparse, unreliable, or diverging
annotations. Co-teaching methods have shown promising improvements for computer
vision problems with noisy labels by employing two classifiers trained on each
others' confident samples in each batch. Inspired by the idea of separating
confident and uncertain samples during the training process, we extend it for
the crowdsourcing problem. Our model, CrowdTeacher, uses the idea that
perturbation in the input space model can improve the robustness of the
classifier for noisy labels. Treating crowdsourcing annotations as a source of
noisy labeling, we perturb samples based on the certainty from the aggregated
annotations. The perturbed samples are fed to a Co-teaching algorithm tuned to
also accommodate smaller tabular data. We showcase the boost in predictive
power attained using CrowdTeacher for both synthetic and real datasets across
various label density settings. Our experiments reveal that our proposed
approach beats baselines modeling individual annotations and then combining
them, methods simultaneously learning a classifier and inferring truth labels,
and the Co-teaching algorithm with aggregated labels through common truth
inference methods.
